Answering to a Higher PowerFour principles define software testing in a regulated environmentSandy Weinberg, Ph.D.
A few years ago, a packager of kosher hot dogs launched an advertising campaign around the slogan “We Answer to a Higher Power.” In a U.S. Food and Drug Administration (FDA)-regulated environment, software also must be tested to assure that it meets a higher “kosher” standard. Every competent programmer thoroughly tests newly developed software before stamping it as ready-to-go. Generally, approval indicates that the program functions successfully and effectively in its intended application. But, if that environment is heavily regulated by the FDA (as are clinical, manufacturing, laboratory and shipping systems in the pharmaceutical, biologics, medical device and some cosmetic, veterinary and foods industries), additional control standards apply.
These standards are defined in the federal regulation 21 CFR Part 11, and call for system validation and testing in addition to norms of functionality. Concerned about management control of programs used in just about all areas of the biomedical industries,1 the FDA requires specialized testing to assure not only that a system is properly performing, but that the regulated company is in full control of that performance. Four principles define the parameters of this specialized testing:
• requirements tests
• border case tests
• pathway testing
• use of high-medium-low risk levels
Requirements testing
The FDA requires that all software must be tested against documented requirements, established by the user’s management, for that program. Only by knowing management’s exact purposes for the program can the FDA visitor determine whether or not the software has been properly tested. To assure compliance, user management should adopt, sign and date (prior to testing) a formal set of requirements. A trace matrix, tracking those requirements against the specific test scripts applied to each (and, when possible, to the modules or program sections applying to each function) may be used to demonstrate the correspondence of tests to requirements.
Border cases
Software testing should focus on cases or data sets that examine the border conditions of the function. For example, if a database is designed to handle 20 values for a variable, the program should be tested to make certain that it accepts value number 20, and that value number 21 is rejected (and does not overwrite value number 1). While all functions described in the requirements document referenced above should be tested, the selection of test cases should focus on border conditions.
Pathway testing
Over the years, the FDA has struggled to define how many cases should be tested before arriving at the (logical and appropriate) conclusion that the critical criteria isn’t the number of cases, but rather the number of pathways or functions. Repeatedly testing the same pathway over and over adds no additional information, but assuring that all important pathways are tested provides the greatest quality control.
For example, consider testing a simple four function calculator.2 Testing the addition function by adding random numbers over and over again does not provide additional information with each new test, and it tells nothing about the division function of the device. To prove functionality, each of the four functions (addition, subtraction, multiplication and division) should be tested with a normal case (add three plus four) and one or more atypical cases (divide by zero; multiply by one, etcetera). Repeating a specific test over and over is unnecessary redundancy, since it’s the pathways that matter, not the datasets.
Risk levels
While these FDA concerns may seem to be creating significant additional work over normal functionality testing, they may not be required in all cases. The FDA encourages the use of a risk assessment to mitigate the effort necessary to demonstrate compliance. In simple terms
• the requirements for pathway testing, testing to requirements, and extensive testing of border cases are only enforced in high-risk situations
• in medium risk environments, a more cursory demonstration of testing to requirements is sufficient
• in low-risk situations, basic function testing is likely to be sufficient
The key is the prior establishment of varying standards according to risk level, and the prior (to testing) assessment of risk.
In the most basic of terms, risk refers to threats to human health and safety.
• If a software failure or error can result in death or serious injury with some reasonable probability of occurrence, it is defined as high-risk.
• If a failure is likely to result in minor injury or significant loss of quality of life, it is considered a medium risk.
• If no injury or significant quality of life danger exists, it is a low-risk environment.
With a carefully written standard and analysis, risk can be used to limit the software testing effort required.
Summary
Software intended for an FDA-regulated setting must answer to a higher standard. The “kosher laws” require that, in addition to standard functional testing, the code must be tested
• against pre-approved, formalized, documented requirements
• using border cases
• through all significant pathways
But, just as “kosher laws” do not apply to everyone, the FDA offers a dispensation based upon the results of a risk analysis, permitting little or no additional testing for software that offers no reasonable threat to human health and safety.
References
1. About the only systems exempted are financial and human resources (HR), and even HR systems may fall within the regulatory umbrella if they include training records and professional credentials.
2. No, calculators do not have to be tested!
Sandy Weinberg is an Associate Professor of Health Care Management at Clayton State University and a Senior Consultant at Tunnell Consulting. He may be reached at editor@ScientificComputing.com.