Deborah A. Payne , Graciela Russomando , Mark W. Linder , Katarina Baluchova , Tester Ashavaid , Werner Steimer , Parviz Ahmad-Nejad and the IFCC Committee for Molecular Diagnostics (C-MD)

External quality assessment (EQA) and alternative assessment procedures (AAPs) in molecular diagnostics: findings of an international survey

De Gruyter | Published online: May 8, 2020

Abstract

Objectives

Quality management for clinical laboratories requires the establishment of internal procedures including standard operating procedures (SOPs), internal quality control (QC), validation of test results and quality assessment. External quality assessment (EQA) and alternativeassessment procedures (AAPs) are part of the quality hierarchy required for diagnostic testing. The International Organization for Standardization (ISO) document with requirements for conformance ISO 15189 and the Clinical and Laboratory Standards Institute document (CLSI) QMS24 require participation in EQA schemes and AAPs where applicable. The purpose of this study was to perform a global survey of EQA and AAPs for key procedures in molecular diagnostic laboratories.

Methods

The Committee for Molecular Diagnostics of the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC C-MD) conducted a survey of international molecular laboratories that covered specific topics of molecular diagnostic services as well as methods for EQA and AAPs. The survey addressed the following aspects: (1) usage of laboratory-developed test (LDT), (2) participation in EQA schemes and (3) performance of AAPs.

Results

A total of 93 responses from laboratories located in Asia, Europe, the Middle East, North America and South America were received. The majority of the participating laboratories (65.9%) use LDTs and 81.3% stated that it is mandatory for them to participate in EQA programs, while 22% of the laboratories reported not performing AAPs. Thirty-one percent of the laboratories use EQAs for fewer than 50.0% of their reported parameters/analytes.

Conclusions

While the majority of laboratories perform EQA and AAPs to improve their quality in molecular diagnostics, the amount of AAPs as quality procedures differs within the laboratories. Further surveys are necessary to clarify the existing needs in additional EQAs and standardized AAPs. The survey will also guide future efforts of the IFCC C-MD for identifying quality practices in need to improve harmonization and standardization within molecular diagnostics.

Introduction

Molecular diagnostics is one among the fastest developing disciplines in the modern laboratory medicine. Molecular genetic diagnostics has some characteristics which are generally referred to as genetic exceptionalism and which must be taken into account in quality assurance [1], [2]. Additionally and in contrast to other disciplines, molecular diagnostics offers an extremely wide range of detection options for a particular parameter such as identification of sequence variation or the detection of infectious pathogens. For example, a whole range of different molecular genetic genotyping methods can be used to detect a particular sequence variation like FV-Leiden, and these methods can be based on laboratory-developed tests (LDTs) [3].

These different methods require different strategies and procedures for quality control (QC) and must be incorporated routinely into a common quality hierarchy system for the laboratory. This includes a quality management system, standard operating procedures (SOPs), alternative assessment procedures (AAPs), internal quality assurance, validation of test results and external quality assessment (EQA).

EQA, also known as proficiency testing (PT), is a key strategy for the comparison of analytical performance between laboratories [3], [4], [5], [6], [7], [8], [9]. Participation in EQAs is a prerequisite for accreditation according to the International Organization for Standardization (ISO) guidance documents 15189 and 17025 and the Clinical and Laboratory Standards Institute (CLSI) QMS24 documents. Additionally EQAs are mandatory for the performance of human genetic tests in some countries [10], [11], [12]. When EQA schemes are not available to the laboratory, AAPs may be used to provide additional quality assurance for the laboratory and are required in this instance by some accrediting organizations. AAPs may also enhance or augment the quality management program [10], [11], [12].

In order to clarify the standardization for molecular diagnostic tests at an international level and in practice, the Committee for Molecular Diagnostics of the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC C-MD) published previously the results of an online survey addressing the post-analytical phase in molecular laboratories to extend its efforts to provide harmonization and standardization for molecular diagnostic testing [13], [14]. As a follow-up to these efforts, we conducted here a survey among voluntary molecular diagnostic laboratories to identify the quality assurance methods used. The generated data provide basic information on current practices in molecular diagnostic laboratories in more than 30 countries. The results of this study intend to stimulate a broad discussion in laboratories offering molecular genetic testing on current quality procedures and practices in molecular diagnostics and serve as a basis for future queries.

Materials and methods

The survey was designed to capture data on practices in quality assurance for molecular diagnostics. Questions were proposed, discussed and approved by the members of the IFCC C-MD and were deemed important.

The online survey addressed the following sections: (1) laboratory demographics, (2) LDTs, (3) EQA and (4) AAPs. Additionally, participants were asked to designate areas of molecular reporting that are in need of standardization. The survey consisted of 30 questions and had both multiple choice and open answer formats. The survey took approximately 25 min to complete and the data generated were collected using Surveymonkey.com [15]. To increase the number of participating laboratories, we offered a Spanish and an English version of the survey. More than 500 laboratories were invited to participate in a call distributed through the national societies of the IFCC member states. The survey started in November 2018 and was opened for participation for a duration of 8 weeks.

Results

Laboratory demographics

A total of 93 laboratories participated in the survey. Sixty-eight laboratories answered the English version of the survey, and 25 participants used the Spanish survey version for reporting. Twenty-eight of 93 (30.1%) laboratories were private laboratories, and 12.9% were university affiliated. The majority was medical center affiliated (37/93; 39.8%) and 7.5% of participating laboratories were large commercial laboratories performing more than 1000 molecular genetic tests per month (Table 1). The majority of the participating laboratories were located in Europe and Latin America. Eleven were located in the US with five additional laboratories each from the Middle East and Asia. The exact origin of the participating laboratories is listed in the Supplementary Table 1.

Table 1:

Description of the participating laboratories.

% n
Private 30.1 28
University affiliated 12.9 12
Medical center affiliated 39.8 37
Large commercial (processes more than 1000 molecular tests a month) 7.5 7
Other (please specify) 9.7 9
Total 93

Regarding the area(s) of molecular diagnostic reporting performed by the participating laboratories, hereditary diseases have been reported as the most common response. Microbiology (infectious disease) and pharmacogenomics were performed by 48.4% and 42.9%, respectively. For this question, the laboratories were technically able to select multiple areas. The exact distribution of the determined responses is shown in Table 2.

Table 2:

Subspecialties of molecular diagnostic laboratories participating in the survey.

% n
Inherited diseases 65.9 60
Microbiology (infectious diseases) 48.4 44
Pharmacogenomics 42.9 39
Personalized/precision diagnostics 30.8 28
Noninvasive pregnancy testing 8.8 8
Oncology 35.2 32
Transplantation 15.4 14
Other (for instance, infectious, hemostasis, hematology, clinical trial inclusion) 40.0 10
Answered 91
Skipped 2

    The percentages of laboratories reporting specific areas of specialization are listed for each category. Ninety-one of 93 laboratories responded.

Participants were asked to indicate the extent to which commercial assay kits versus LDTs were utilized in their laboratories. The majority of laboratories reported using commercial assays, 84.6% (77/91), with 34.1% using only commercial assays, and 65.9% (58/88) reported using LDTs.

Participation in external quality assessments (EQAs)

An EQA is an interlaboratory comparison that may extend throughout all phases of a testing cycle including interpretation of results. The majority (93.9%) of the laboratories stated that it is mandatory for them to participate in EQAs or PT if they are available. However, there is variability in the utilization of EQAs within laboratories where 24.4% (22/90) of the participants reported that the percentage of parameters covered by EQAs/PT is between 0% and 25%, and 51.1% (46/90) of the participants stated that 75%–100% of their parameters are externally controlled using EQAs/PT. The exact distribution of the determined responses is displayed in Table 3.

Table 3:

EQA coverage of molecular parameters.

% n
0%–25% 24.4 22
26%–50% 6.7 6
51%–75% 17.8 16
76%–100% 51.1 46
Answered 90
Skipped 3

    Ninety of 93 laboratories responded. Percentage of nucleic acid tests that are covered by EQAs/PTs by the participating laboratories is given.

Among the laboratories that reported participating in EQA, 20 different EQA providers were named. Among the named providers we found that the Reference Institute for Bioanalytics (RfB), the College of American Pathologists (CAP) and Instand e.V. were most frequently named by the laboratories (Supplementary data, Table 2).

Performing alternative assessment procedures (AAPs)

When PT materials are not available, the laboratory is responsible for establishing an AAP for verifying the acceptability of test performance [16]. AAPs vary and are dependent on the parameters analyzed. Examples of AAP include: (1) internal and/or external split samples, (2) audit sample review, (3) analysis of calibrator control material, (4) analysis of interlaboratory QC results, (5) re-evaluation of interpretative results, (6) clinical correlations, (7) examination of surrogate organisms or materials and (8) government or university interlaboratory comparison programs. Guidelines and workflows for AAP are available in section 3.2 of the CLSI QMS24 document [12]. Among the laboratories included in the survey, 50% (n=46/91) stated that AAP is performed if there is no EQA available for the parameter offered in routine diagnostics. A total of 20.9% (n=19/91) of the laboratories stated that they perform an AAP for every parameter determined in the laboratory while 22% (n=20/91) reported not to perform AAPs at all (Table 4).

Table 4:

When proficiency testing materials are not available, the laboratory is responsible for establishing an alternative assessment procedure (AAP) for verifying the acceptability of test performance.

% n
We perform alternative assessment procedures (AAPs) for every analyte in our laboratory even if an EQA scheme is available 20.9 19
We only perform alternative assessment procedures (AAPs) if there is no EQA available 50.6 46
We only perform alternative assessment procedures (AAP) for representative analytes using specific detection techniques 6.6 6
We do not perform alternative assessment (AAP) procedures 22 20
Total 91

    The number and percentages of participating laboratories performing AAPs are summarized. Ninety-one of 93 laboratories responded.

Concerning the frequency of AAPs performed, 17.2% of the laboratories reported that they perform AAP once after installation of a new diagnostic assay. A total of 17.2% and 32.18% stated that they perform AAPs once or twice a year per parameter while 12.64% of the laboratories do not have a defined frequency of conducting AAPs (Table 5).

Table 5:

Frequency of AAP performance by the laboratories.

% n
We perform AAP once after installation of a new diagnostic assay 17.24 15
We perform AAP once a year per parameter 17.24 15
We perform AAP twice a year per parameter 32.18 28
We do not perform AAP at all 20.69 18
We do not have a defined frequency for performing AAP 12.64 11

The sample size of the AAP was also addressed by the survey. Eighty-one percent of the laboratories include up to 10 samples in their AAPs while 11.8% check their quality based on 11–20 samples. One laboratory stated to include more than 50 samples in an AAP. The majority of AAPs get approved by the head of department (41.4%; n=36/87). A total of 13.8% of the AAPs are approved by the technical supervisor while 12.6% of the laboratories reported that no one is approving the AAP (Tables 6 and 7).

Table 6:

Number of samples that are usually included in AAPs.

% n
<10 81.2 69
11–20 11.8 10
21–50 5.9 5
>51 1.2 1
Answered 85
Skipped 8

    Eighty-five of 93 laboratories responded.

Table 7:

Who is responsible for approving the results/interpretation of the AAP?

% n
Head of the department 41.4 36
Clinical or anatomical pathologist 12.6 11
Technical supervisor 13.8 12
No one 12.6 11
Other (please specify) 19.5 17
Answered 87
Skipped 5

    Among “other” the following answers were given: molecular biologist (n=1), head of genetic laboratory (n=1), clinical pharmacologist (n=1), lab manager (n=1), quality manager (n=1), head of laboratory/European Specialist in Laboratory Medicine (n=1), project leader (n=1), clinical chemist (n=1), no answer (n=2). Eighty-seven of 93 laboratories responded.

Looking at the way the AAPs are performed, most of the participants stated that they retest already characterized material and re-evaluate the interpreted results. Additionally, 48.8% of laboratories take characterized material from other laboratories and include this in their AAP. Only 6% reported to use surrogate organisms carrying characterized genetic material. Splitting of control material used is done in 14.3% of the laboratories (Table 8).

Table 8:

Description of how an AAP is performed in the participating laboratory.

% n
We take characterized material from other laboratories and include this in our AAP 48.8 41
We split manufacturer control material and generate characterized material for the subsequent AAP procedure 14.2 12
We perform retesting of already characterized material and re-evaluation of interpreted results 57.1 48
We use surrogate organisms that carry characterized genetic sequences 6 5
Other 16.7 14
Answered 84
Skipped 9

    The laboratories were asked what kind of AAP is performed in their laboratory. One may select more than one answer. Eighty-four of 93 laboratories responded.

Discussion

QC in a laboratory requires the establishment of a quality management system that includes various procedures including SOPs, AAPs, internal quality assurance, validation of test results and external quality evaluation. In order to investigate which procedures are implemented in the laboratories on an international level, the C-MD of the IFCC has prepared a survey highlighting relevant topics.

The available data show current practices in molecular diagnostic laboratories in more than 30 countries. In addition, the data from the study identified several aspects of quality assurance such as AAP and EQA. For the majority of the laboratories, more than 75% of the parameters/analytes offered are available from EQAs, but 24% of the participating laboratories state that they only perform EQA on 25% of the parameters/analytes performed in their routine diagnostics. There are plenty of opportunities for improvement and may be simply due to non-participation in EQAs or due to the fact that EQA schemes or EQA providers are not available for defined parameters/analytes. In order to mitigate one reason for non-participation, the C-MD has installed a listing of well-known EQA providers on its webpage and specified their offered analytes [17]. Other reasons for omitting EQAs is that EQA providers do not supply to corresponding countries due to logistical issues and/or the lack of adequate financial compensation for schemes. The cost burden to participating laboratories may also be too high for regular participation in the EQA qualifying interlaboratory comparisons.

While participation in the EQAs is now widely accepted in the laboratories, our data show a more heterogeneous picture for participation in AAPs. This may be because, depending on the parameter, AAPs are more complex in their implementation and there is a lack of clear recommendations regarding which kind of AAPs should be employed for the wide variety of molecular testing methods and diagnostic assays currently being offered by molecular laboratories. AAPs as a quality assurance measure are carried out differently by the participating laboratories. While 50.6% of the laboratories perform AAPs only if no EQA is available, 20.6% of the participants always do so and even if an EQA is available for the parameter. Interestingly, 22% of laboratories perform no AAPs.

Although best practice guidelines exist, a lack of clear recommendations for implementation of AAPs may lead to a quite different extent of samples which are included in AAPs. While 17% perform AAPs once after installation of a new diagnostic assay, 32% of the laboratories implemented AAPs twice a year per parameter. The majority of laboratories include less than 10 samples in an AAP, and 18.8% of the laboratories include more than 10 samples. The responsibility for the review and approval of the AAPs is obviously organized differently in laboratories as well: While in 41.4% of the laboratories, the head of department and in 12.6% the clinical pathologist is approving the results of the quality measurement, we found that in 13.8% the technical supervisor is checking the results of the procedure. This demonstrates that AAPs are not clearly defined and need characterization and standardization.

The limitation of this study is certainly the fact that the majority of the participating laboratories are from Europe. There is also an imbalance within Europe, as eight answers came from German laboratories and only one from the UK or Italy. The participation for some countries (i.e. Italy) was lower than in a previous study [18]. A similar imbalance was evident in the results from South America: 10 laboratories answered the questions from Argentina and only one from Brazil, probably due to the languages offered in this survey. In order to increase the number of participating laboratories, consideration should be given to re-inviting them to participate or extending the length of the survey.

Another limitation of the study is that substantial parts of the results may also depend on economic and health policy conditions within the countries of the participating laboratories. Having in mind that participation in EQAs is only mandatory in some countries it is understandable that some parts of the results of the survey can be traced back to this. Additionally, the study does not take into account how financially costly comparable quality assurance measures exist for individual laboratories although these will also have an impact on the results of the survey.

Furthermore, the listing of proficiency test providers is influenced by the origin of the participating laboratories. The C-MD wants to investigate further the topics of quality assurance through AAP and EQAs. It will be interesting to find out how for instances quantitative assays are established in AAP within the laboratories, which reference materials are used therefore or which guidelines are followed in establishing the assays. To this end, the upcoming surveys will also be placed for other countries in order to define the need for quality-improving measures more concretely and to address the current status quo in low-income countries as well.

Acknowledgments

The IFCC C-MD would like to acknowledge the guidance and support of the IFCC’s Scientific Division, specifically Dr. Philippe Gillery as well as the IFCC staff, specifically Paola Bramati.

    Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.

    Research funding: None declared.

    Employment or leadership: None declared.

    Honorarium: None declared.

    Competing interests: Authors state no conflict of interest.

References

1. Mattocks CJ, Morris MA, Matthijs G, Swinnen E, Corveleyn A, Dequeker E, et al. A standardized framework for the validation and verification of clinical molecular genetic tests. Eur J Hum Genet 2010;18:1276–88. Search in Google Scholar

2. McGuire AL, Fisher R, Cusenza P, Hudson K, Rothstein MA, McGraw D, et al. Confidentiality, privacy, and security of genetic and genomic test information in electronic health records: points to consider. Genet Med 2008;10:495–9. Search in Google Scholar

3. Haselmann V, Geilenkeuser WJ, Helfert S, Eichner R, Hetjens S, Neumaier M, et al. Thirteen years of an international external quality assessment scheme for genotyping: results and recommendations. Clin Chem 2016;62:1084–95. Search in Google Scholar

4. Ahmad-Nejad P, Dorn-Beineke A, Pfeiffer U, Brade J, Geilenkeuser WJ, Ramsden S, et al. Methodologic European external quality assurance for DNA sequencing: the EQUALseq program. Clin Chem 2006;52:716–27. Search in Google Scholar

5. Dorn-Beineke A, Ahmad-Nejad P, Pfeiffer U, Ramsden S, Pazzagli M, Neumaier M. Improvement of technical and analytical performance in DNA sequencing by external quality assessment-based molecular training. Clin Chem 2006;52:2072–8. Search in Google Scholar

6. Kalman LV, Lubin IM, Barker S, du Sart D, Elles R, Grody WW, et al. Current landscape and new paradigms of proficiency testing and external quality assessment for molecular genetics. Arch Pathol Lab Med 2013;137:983–8. Search in Google Scholar

7. Keppens C, Dequeker EM, Patton SJ, Normanno N, Fenizia F, Butler R, et al. International pilot external quality assessment scheme for analysis and reporting of circulating tumour DNA. BMC Cancer 2018;18:804. Search in Google Scholar

8. Maly FE, Fried R, Spannagl M. Participation and performance in INSTAND multi-analyte molecular genetics external quality assessment schemes from 2006 to 2012. Clin Lab 2014;60:333–8. Search in Google Scholar

9. Touitou I, Rittore C, Philibert L, Yague J, Shinar Y, Aksentijevich I. An international external quality assessment for molecular diagnosis of hereditary recurrent fevers: a 3-year scheme demonstrates the need for improvement. Eur J Hum Genet 2009;17:890–6. Search in Google Scholar

10. Gentechnikgesetzes BfdRÖ-Äd. Bundesgesetzblatt für de Republik Österreich – Änderung des Gentechnikgesetzes. 2005. Search in Google Scholar

11. Bundesministerium f. Gesundheit. Gendiagnostikgesetz. http://www.gesetze-im-internet.de/gendg/index.html, Accessed August 2019. http://www.gesetze-im-internet.de/gendg/index.html, Accesed: 2014. Search in Google Scholar

12. CLSI. Clinical and Laboratory Standards Institute (CLSI) guideline QMS24 3rd ed. Using proficiency testing and alternative assessment to improve medical laboratory quality. 2016: 132. Search in Google Scholar

13. Payne DA, Baluchova K, Peoc’h KH, van Schaik RH, Chan KC, Maekawa M, et al. Pre-examination factors affecting molecular diagnostic test results and interpretation: a case-based approach. Clin Chim Acta 2017;467:59–69. Search in Google Scholar

14. Payne DA, Baluchova K, Russomando G, Ahmad-Nejad P, Mamotte C, Rousseau F, et al. Toward harmonization of clinical molecular diagnostic reports: findings of an international survey. Clin Chem Lab Med 2018;57:78–88. Search in Google Scholar

15. SurveyMonkey.com. https://www.surveymonkey.com/. Search in Google Scholar

16. Richards CS, Grody WW. Alternative approaches to proficiency testing in molecular genetics. Clin Chem 2003;49:717–8. Search in Google Scholar

17. Committee for Molecular Diagnostics (C-MD) of the IFCC . IFCC list of EQAs in molecular diagnostics. https://www.ifcc.org/ifcc-scientific-division/sd-committees/c-md/externalqualityassessment-proficiencytestinginmoleculardiagnostics/. Search in Google Scholar

18. Capoluongo E, La Verde N, Barberis M, Bella MA, Buttitta F, Carrera P, et al. BRCA1/2 molecular assay for ovarian cancer patients: a survey through Italian departments of oncology and molecular and genomic diagnostic laboratories. Diagnostics (Basel) 2019;9:2–10. Search in Google Scholar

Supplementary Material

The online version of this article offers supplementary material (https://doi.org/10.1515/cclm-2020-0101).

Received: 2020-01-29
Accepted: 2020-04-12
Published Online: 2020-05-08
Published in Print: 2021-02-23

©2020 Walter de Gruyter GmbH, Berlin/Boston