Validation Update: State of the Art 2009
Lack of documentation, system validation reflect all too common uncertainty
Sandy Weinberg, Ph.D. and Ron Fuqua, Ph.D.
The United States Food and Drug Administration has promulgated standards for the testing and management control of computer systems used in clinical testing, manufacturing execution systems (MES), laboratory information management systems (LIMS), computerized medical devices and other systems since 1985. This year is the fifth anniversary of the most recent formalization of those requirements, found in 21 CFR Part 11. With more than a 20-year history of struggling to validate systems and a five-year requirement to do so, it seems appropriate to assess the current state-of-the-art. We’ve reviewed files from independent audits of U.S. and European systems and self-reports from system developers and users to try to arrive at a subjective summary.
In 2009, spurred by Part 11 and by FDA investigation questions, most pharmaceutical, biologics and device development and manufacturing organizations are reporting that
• all pre-existing systems have been validated
• new systems are validated as a part of the acceptance/installation process
• revalidation occurs approximately every three to four years
The exception to self-declaration of up-to-date validation of systems is in drug and biologics discovery organizations, often universities or university spin-off companies with little or no FDA involvement. Few, if any, of these organizations have included system validation in their quality agendas.
These conclusions are far from scientific: the data is derived from five audits conducted in the first half of 2009, supplemented by conversations with nine systems administrators or senior users describing the validation efforts of their organizations. The sample is not random, and not sufficient to accurately represent the wide range of U.S. and European pharmaceutical MES, LIMS and other systems. But if the goal, as it is here, is to provide a summary of the general impression of the system validation state-of-the-art, those subjective impressions may be of some value.
In examining the actual validation results of self-proclaimed validation-compliant organizations, it is noted that, in most cases, the purchase or employment of a system that a manufacturer has claimed is validated is accepted as prima fascia evidence with little or no testing or review, even if the system in question has been customized or modified. The vendor support for claims of validation varies significantly: most often, the term “validated” is used interchangeably with “tested,” and does not meet 21 CFR Part 11 validation standards.
Other validation weaknesses fall into three primary areas:
• inadequate testing
• incomplete documentation
• inadequate change control
Most organizations demonstrated two or more of these weaknesses. Inadequate testing includes a failure to test border cases, the stress situations that can detect likely failures as the system is utilized at the extremes of its intended purpose. Documentation problems seem to most commonly reflect delays or gaps between system upgrades and document revisions: often SOPs are antiquated, documentation refers to previous versions; and/or requirements documents have been superseded by events and new needs. Change control seems to follow one of two extremes: some organizations seem to respond to user whims without adequate re-testing and documentations; while others seem to resist any changes regardless of their value. The balance of evaluating, implementing, testing and documenting changes in response to encountered errors or changing needs is apparently a rarity.
So, this subjective, informal summary of the state-of-the-art is a mixed report. Most regulated organizations are aware of the need, specified in 21 CFR Part 11, to validate computer systems. Definitions of exactly what constitutes “validation” differ widely, apparently often resulting in actions inadequate to truly assure system accuracy and managerial control. And the major weaknesses in those validation efforts — inadequate testing, incomplete documentation and inadequate change control — reflect a common uncertainty.
The industry recognizes the need to control and manage computer systems in order to rely upon their outputs, and it is moving slowly to implement those controls and managerial constraints. However, it still has a long road to travel.
Sandy Weinberg is an associate professor of health care management and Ron Fuqua is an assistant professor of health care management at Clayton State University. They may be reached at editor@ScientificComputing.com.