## Abstract

**Background:** Limit of detection (LOD) is an important performance characteristic of clinical laboratory tests. Verification, as recommended by the CLSI EP17-A2 guideline, is done by testing a sample with a claimed LOD concentration. Claimed LOD is verified if the 95% CI for the population proportion, calculated from observed proportion of positive results, contains the expected detection rate of 95% (CLSI EP17-A2; Clin Chem 2004;50:732–40). Claimed LOD, verification sample concentration, and observed rate of positive results are subjects to systematic and random errors that can cause false failure or false acceptance of the LOD verification. The aim of this study was to assess the probability to pass or fail verification of claimed LOD with various numbers of tests as function of the ratio of test sample concentration and actual LOD for PCR-based molecular diagnostics tests and provide recommendations for study design.

**Methods:** A method of calculating the probability of passing the claimed LOD verification following CLSI EP17-A2 guideline recommendations, based on the Poisson–binomial probability model, have been developed for PCR-based assays.

**Results:** Calculations and graphs have shown that the probability of passing LOD verification depends on the number of tests and has local minima and maxima between 0.975 and 0.995 for the number of tests from 20 to 1000 on samples having actual LOD concentration. The probability of detecting the difference between claimed LOD and actual LOD increases with the number of tests performed. Graphs and tables with examples are included.

**Conclusions:** Method, tables, and graphs helping in planning LOD verification study in molecular diagnostics are provided along with the recommendations on what to do in case of failure to verify the LOD claim.

### Impact Statement

Proposed methods will help with planning manufacturer-claimed LOD verification with high probabilities of passing verification when claimed LOD does not exceed the actual LOD significantly and failing verification otherwise. Verification of LOD will assure that the analytical sensitivity of molecular diagnostics assays is adequate for (*a*) minimizing risk for recipients of blood donations, (*b*) reliable screening of patients for cervical cancer, (*c*) reliable companion diagnostics, (*d*) reliable monitoring of treatment of patients with viral and bacterial infections, etc. The proposed method advances the method recommended in the CLSI EP17-A2 guideline consisting of testing the hypothesis of equality between the actual and claimed LOD in verification studies.

## INTRODUCTION

The limit of detection (LOD)^{4} is an important performance characteristic of clinical laboratory diagnostic, therapy monitoring, and screening tests. In the CLSI EP17-A2 guideline (1) and elsewhere (2), LOD is defined as the lowest concentration of analyte that can be consistently detected (typically, in 95% of samples tested under routine clinical laboratory conditions and in a defined type of sample, e.g., plasma, whole blood, urine, cell swabs, or tissue sections). For the purpose of the statistical analysis, this definition is formalized here: LOD is the analyte concentration corresponding to 95% probability of detection. Many users of diagnostic tests verify the LOD claimed by the manufacturer for each analyte and sample type. Manufacturers sometimes use the LOD verification approach with reduced amount of testing, for example, to show equivalency of multiple genotypes of viral test LODs to the one claimed for the major virus type determined with full LOD study.

The method of verification of claimed LOD by a laboratory, after the manufacturer had established the LOD following recommendations of the CLSI EP17-A2 guidelines (1), consists of testing at least *n* = 20 replicates with 1 reagent lot, 1 instrument, 2 samples at the claimed LOD concentration, over 3 days, and 2 replicates per day of each sample. The described minimal experimental design does not result in 20 replicates but only in 3 × 2 × 2 = 12, and, for this reason, one has to increase the number of some of the design factors (e.g., the number of replicates, not necessarily the same on each instrument, tested each day). Power to detect differences between the claimed LOD and the actual LOD can be improved by increasing the number of replicates above the allowed minimum of 20. The claimed LOD passes verification if the 2-sided 95% CI for the population proportion of positive results calculated from the observed proportion (*r*/*n*) includes the expected detection rate of 95%, since in such a case, the statistical hypothesis that the actual LOD is equal to the claimed LOD is not rejected. Note that the fact that the hypothesis is not rejected does not prove that the claimed LOD is equal to or is lower than the actual LOD. The method of assessing the probability to detect unacceptable difference between the claimed LOD and the actual LOD is discussed in sections below. The method of calculating the 95% CIs for the observed proportion of positive results is described in the Appendix. Table 1 presents numbers (*n*) of replicated tests performed with a sample of claimed LOD concentration and the passing numbers of positive results (*r*) determined from Eq. A2 in the Appendix that give an upper bound (*U*) of the 95% CI that is ≥95%. Examples of applying the statistical methods to LOD verification study design and interpretation of results are provided below.

## Statistical Method

The probability *P* (*x* ≥ *r*) to observe at least *r* positive results in *n* valid tests with a sample having the probability of detection *p* is binomial:
(1)

With the test sample concentration being equal to the actual LOD, the probability to pass verification is calculated using Eq. 1 for *p* = 0.95 and the passing number of positive results (*r*) corresponding to the specified number of tests (*n*) (see Introduction and Table 1). The second term on the right-hand side of Eq. 1 is easy to calculate using cumulative binomial probability function in Microsoft Excel^{®}: binomdist(*r* – 1, *n*, *p*, true).

In reality, both the actual LOD and the actual test sample concentration are not known exactly and are subject to systematic and random errors, and the probability of detection (*p*) depends on both the actual LOD and the actual concentration of the analyte in a test sample used for LOD verification. In a typical case of PCR-based molecular diagnostics assay of a nucleic acid run on a system with large enough number of amplification and detection cycles, a single copy of target nucleic acid at PCR input is sufficiently amplified and detected. An indication of the PCR assay's ability to detect a single copy is typically that maximum observed with the assay number of amplification and detection cycles is smaller than the number of cycles available on the instrument. The maximum observed number of amplification and detection cycles obviously corresponds to the amplification and detection of the smallest number of copies of target nucleic acid at PCR input, which is a single copy. Then the cases of nondetection take place when no copies of the target nucleic acid are available for amplification and detection. Therefore, the probability of detection (*p*) is the probability of at least 1 copy of target nucleic acid being available for amplification and detection. Such probability of detection (*p*) is complimentary to the probability of no copies available for amplification and detection. The probability of having *x* copies in a randomly drawn sample, with the mean number of copies per test sample volume being μ, is well approximated, ignoring finite volume of the bulk, by Poisson distribution. The probability to have *x* copies in the sample (described with Poisson probability distribution) and extract *u* copies out of *x* (described with binomial probability distribution) is the product of the two probabilities. Based on the above, a formula for calculating the probability of detection (*p*) as a function of the mean number of copies in the test samples (μ) has been derived:
(2)

Eq. 2 is easily verified for 3 concentrations: μ = 0, μ = LOD, and μ → ∞, with the obvious expected probabilities of detection being equal to the calculated probabilities of detection 0, 0.95, and 1, respectively.

For example, in studies involving verification of claimed LOD for *m* genotypes of a virus, the randomly varying numbers of positives observed in *m* verification events are independent, and, therefore, assuming that the LODs for each genotype are the same as for the major genotype, the probability to pass verification in all *m* LOD verification events (*P*_{m}) is the *m*-th power of the probability of passing a single verification given by Eq. 1 above, and it can be expressed as:
(3)

In a more realistic case, the LODs for the genotypes may be only close to the LOD for the major genotype but different from each other. Then the probability to pass verification for all *m* genotypes is a product of the probabilities to pass verification of LOD for each genotype. The probability of detection of the actual LOD for any genotype exceeding the claimed LOD for the major genotype increases with an increased number of tests (*n*).

## Results

### Test sample concentration equal to the actual LOD

The graph in Fig. 1 presents the probability to pass verification of the claimed LOD calculated with Eq. 1 as a function of the number of valid tests (*n*) obtained during testing of a sample having actual LOD concentration with respective probability of detection (*P* = 0.95). In the graph, the probability to pass verification varies approximately between 0.975 and 0.995 and attains local maxima and minima at certain numbers of tests (*n*).

The local maxima of the probability to pass verification seen in the graph are attained at values of the number of replicates (*n*) corresponding to the local minima of the observed proportions of positives (*r*/*n*) providing for the smallest *U* ≥0.95 as calculated with Eq. A2 in the Appendix. Conversely, the local minima of the probability to pass verification are attained at values of the number of replicates (*n*) corresponding to the local maxima of the observed proportions of positives (*r*/*n*) providing for the smallest *U* that is not smaller than 0.95. For example, a local maximum for the probability to pass LOD verification, 0.995 (it is also the global maximum for the number of replicates *n* ≥20), is attained at *n* = 23, *r* = 19, providing for the smallest *U* = 0.9505, which is not smaller than 0.95, calculated with Eq. A2 in the Appendix, with the observed proportion of positives being at a local minimum of 19/23 = 0.826. The nearest local minimum for the probability to pass (0.978) is attained at *n* = 22, *r* = 19, providing for the smallest *U* = 0.9709, which is not smaller than 0.95, with observed proportion of positives at its local maximum of 19/22 = 0.864. From the above, it is clear that the probability to pass LOD verification is higher with *n* that allows for passing with smaller observed proportion of positives (*r*/*n*) having *U* ≥0.95.

In practice, the number of tests (*n*) can be specified to be larger than the one corresponding to a local maximum of the probability to pass by the number of invalid tests allowed without retesting. Then with the allowed number of invalid tests, the probability to pass verification with a reduced number of valid tests is not decreased. For example, if the specified number of tests is 25 and the minimum acceptable number of valid tests is 23, the probability to pass LOD verification with *n* = 25 is 0.993. With 1 invalid test and 24 valid tests, the probability to pass is 0.994, and with 2 invalid tests and 23 valid tests, the probability to pass is 0.995. If the specified number of tests for the LOD verification was *n* = 23, and one invalid result was allowed, then with *n* = 22 valid tests the probability to pass verification would drop significantly from 0.995 to 0.978, and the probability of failure would increase 4.4 times from 0.005 to 0.022. The above is an example of a practical application of this theoretical work to optimizing LOD verification study design.

The graph in Fig. 1 suggests that, with increase of *n*, the probability to pass LOD verification asymptotically converges at *P* ≈ 0.975. Table 2 presents the number of tests (*n*) (ranged from 23 to 264) along with the respective passing numbers of positive results (*r*) and the probabilities (*P*) of passing LOD verification at the local maxima.

### Test sample concentration not equal to the actual LOD

The LOD verification sample can have a concentration different from the actual LOD because of the following: (*a*) the concentration of the verification sample targets the claimed LOD, but it is a subject to systematic and random pool preparation errors and therefore it is generally different from the claimed LOD, and (*b*) the claimed LOD estimate generally is a subject to systematic and random errors and therefore generally is not equal to the actual LOD. Following recommendations of the CLSI EP17-A2 guideline, the highest of the LOD estimates obtained with 2 or 3 lots of reagent is claimed. This scenario creates a positive bias of the claimed LOD vs actual LOD. In a less common case of LOD estimation with 4 or more lots of reagent, combined across reagent lots data for the LOD estimation are used with the probit analysis, as recommended in EP17-A2. In such case, the claimed LOD can have positive, negative, or no bias, depending on the concentration levels tested, and the random error is reduced because more tests per concentration level are used in the analysis.

The difference between the actual verification sample concentration and the actual LOD determines the probability to pass verification. Probability (*p*) of detection with the ratio of actual test sample concentration and the actual LOD (μ/LOD) is calculated with Eq. 2. The required minimum number of positive results (*r*) to pass verification with *n* tests is calculated using Eq. A2 in the Appendix, and the probability to pass verification (*P*) is calculated using Eq. 1.

The probability to pass LOD verification as a function of difference between the actual log_{10}(LOD) and the actual log_{10}(μ) was calculated and shown in Fig. 2 for (*n*, *r*) pairs from Table 2. The differences log_{10}(LOD) − log_{10}(μ) = log_{10}(LOD/μ), corresponding to the respective log_{10} of the ratios of actual LOD and actual sample concentration, were used for the calculations of detection probabilities (*p*) from Eq. 2, and the probabilities to pass verification (*P*) were calculated using Eq. 1.

The curves in Fig. 2, located in the graph from right to left, in the order they are listed in the legend, show that with LOD lower than the test sample titer [that is, log_{10}(LOD) − log_{10}(μ) < 0], the probability to pass LOD verification is close to 1. With log_{10}(LOD) − log_{10}(μ) = 0, the probability to pass is >0.98. Also, as the number of tests (*n*) increases, the curves, describing the relationship between the probability to pass verification (*P*) and the difference log_{10}(LOD) − log_{10}(μ), become steeper. In other words, the probability, or power, to detect an increase in the difference log_{10}(LOD) − log_{10}(μ) (or the ratio of the LOD and the test sample concentration) increases as the number of tests increases. Therefore, the number of tests to verify the LOD can be determined to provide for adequate power of detecting a specified increase of LOD/μ ratio. The optimal number of tests to verify LOD would be a tradeoff between those for acceptable probability to pass verification and the probability of detection of specified LOD/μ > 1 ratio.

The curves in Fig. 3 show the boundaries of actual difference, log_{10}(LOD) − log_{10}(μ), corresponding to specified probabilities, 0.1 and 0.95, of passing LOD verification vs the replicated number of tests. The differences, log_{10}(LOD) − log_{10}(μ), corresponding to both probabilities (0.1 and 0.95) get smaller with the increase of the number of tests. These curves help in determining the number of tests (*n*) so as to provide for high probability 0.95 of passing the LOD verification with a specified small actual difference [log_{10}(LOD) − log_{10}(μ)] and for low probability 0.10 of passing the verification with a larger specified difference [log_{10}(LOD) − log_{10}(μ)]. For example, with *n* = 100 tests, the probability to pass LOD verification is 0.95 when log_{10}(LOD) − log_{10}(μ) = 0.036, which corresponds to LOD/μ = 10^{0.036} = 1.086. The probability is 0.1 when log_{10}(LOD) − log_{10}(μ) = 0.2, which corresponds to LOD/μ = 10^{0.2} = 1.585. In other words, with *n* = 100 tests and the test sample concentration equal to the claimed LOD, the probability to pass verification is 0.95 when the actual LOD is 8.6% higher than the claimed LOD, and it is 0.1 when the actual LOD is 58.5% higher than the claimed LOD. Respectively, the power to detect the LOD increase by 58.5% is 1.0 − 0.1 = 0.9. Therefore, the graph in Fig. 3 allows choosing the number of tests (*n*) that provides for acceptable trade-off between passing verification with 95% probability when there is no significant increase of the actual vs claimed LOD while failing verification with 90% probability when there is a significant increase in actual vs claimed LOD. This is another example of a practical application of this theoretical work for optimizing LOD verification study design.

### Verification of LOD for multiple analytes

The following example illustrates the verification of LOD for multiple analytes. In a real case experiment, LOD verification of 6 hepatitis C virus (HCV) genotypes with 2 sample matrices (plasma and serum) resulted in 12 independent verifications performed in a single study. Tests were performed with 3 lots of reagent on a sample with the concentration equal to the LOD estimated with the major genotype to verify the LOD for each of the genotypes and sample matrices. A total of 63 replicates were tested for each combination of genotype, matrix, and reagent lot. At least 60 valid test results were required per reagent lot. The CLSI guideline EP17-A2 does not address the verification of multiple LODs in a single study, and we chose to combine the test results across 3 lots of reagent for verification of LOD for each genotype/matrix. The total number of valid tests across 3 lots of reagent was from *n* = 180 [3 invalid results with each lot of reagent, *n* = (63 − 3) × 3 = 180] to *n* = 189 (no invalid results with each lot of reagent, *n* = 63 × 3 = 189). Assuming that the LODs were the same with each lot of reagent, sample matrix, and genotype, the probabilities of passing all 12 LOD verifications were calculated with equations Eq. 1, Eq. 2, and Eq. 3 for *m* = 12 and plotted vs log_{10}(LOD) − log_{10}(μ) differences in Fig. 4. Each curve represents a pair of (*n*, *r*). The highest probability to pass verification for a genotype/matrix combination takes place with *n* = 186, *r* = 170 (the top curve in Fig. 4), and the lowest with *n* = 185, *r* = 170 (the bottom curve in Fig. 4). Each curve in Fig. 4 corresponds to the same pair (*n*, *r*) and same log_{10}(LOD) − log_{10}(μ) difference for each of 12 genotype/matrix combinations.

For the number (*n*) of tests across 3 reagent lots for each of the 12 verification events in the range (180, 189), the probability to pass all 12 verifications is ≥0.95 with the log_{10}(LOD) − log_{10}(μ) difference in the range from approximately −0.028 to −0.015. This result corresponds to the actual LOD being from 10^{−0.028} × 100% = 93.7% to 10^{−0.015} × 100% = 96.6% of the claimed LOD. The verification sample concentration (μ) targets the claimed LOD, established as the worst case of 3 reagent lots, as recommended by the CLSI EP17-A2 guideline, and it is likely positively biased. The LOD verification is based on data combined across 3 lots of reagent with no bias. For this reason, it is quite likely that the log_{10}(LOD) − log_{10}(μ) difference is negative with magnitude larger than 0.028, which corresponds to the actual LOD being 93.7% of the claimed LOD. This result means that when the LODs of the genotypes are equal to the LOD of the major genotype, with actual LOD quite likely to be at least 6.3% lower than the claimed LOD for the major genotype, the probability to pass verification for all 12 genotype/matrix combinations is at least 0.95. Indeed, all 12 verifications passed successfully, supporting the hypotheses of equivalency of the LODs of the genotypes to the LOD for the major genotype.

### What to do when LOD verification fails?

Verification of LOD fails when the 95% CI for the population proportion of positive results calculated from the observed proportion of positive results does not include 95%. With the ratio of the claimed LOD and the actual LOD not exceeding a specified value, the failure of verification is a false failure; otherwise, it is a true legitimate failure. Without investigation, the reasons for failure of LOD verification are not known; hence, to rule out false failures, it is proposed to independently prepare and test a new sample pool. If the LOD verification passes with the additional testing, the initial failure can be attributed to random errors and considered to be resolved, particularly when the number of replicated tests is sufficient to detect a specified ratio of the claimed and actual LODs with high power. If the verification fails with additional testing, the diagnostic product manufacturer should be provided all the information of the study design and results of verification testing to investigate the problem and to work out a corrective action, based on the results of investigation.

## Conclusions

The claimed LOD typically being the highest estimate for 2 or 3 reagent lots, as recommended in CLSI EP17-A2 guideline, has a positive bias. With the verification sample concentration targeting the claimed LOD, the probability of passing LOD verification usually exceeds 0.95. The method of calculating the probability of passing LOD verification described and the graphs help in planning LOD verification studies in molecular diagnostics to provide for both high probability to pass verification of the claimed LOD when it does not exceed the actual LOD significantly and to fail verification otherwise. Recommendations for resolving LOD verification failures in a clinical laboratory have been provided.

## Acknowledgments

The authors are grateful to Dr. Collinson, the Editor-in-Chief, and to the anonymous reviewer for constructive suggestions that helped improve the paper.

## Appendix: Calculation of the Limits of 95% CI

While some of the asymptotic approximations reviewed (3) provide for tighter CIs for some combinations of the sample size and the number of observed successes (positive test results in our case), the Clopper–Pearson, often called “exact,” is the only method that consistently provides for at least the specified level of confidence. The other popular Wilson score CI, while on average being tighter than the Clopper–Pearson CI, can have as low as 93% CI for certain combinations of the number of tests and the number of positives with a specified level of confidence 95%. For this reason, Clopper–Pearson formulas were used in this study. The lower (*L*) and upper (*U*) bounds of the Clopper–Pearson CI for the population proportion of positives are used in this report. It can be calculated from the observed number of positive (reactive) results (*r*) in *n* tests using the formulas Eq. A1 and Eq. A2 [notation modified from Clopper and Pearson (4)]:
(A1)
(A2)

With α = 0.05 in Eq. A1 and Eq. A2, [L, U] limits have 95% CI. Using, for example, *finv*() function in Microsoft Excel for calculating quantiles of the *F*-distribution in Eq. A1 and Eq. A2 yields values for the confidence bounds that are in agreement with those provided in the *CRC Handbook of Tables for Probability and Statistics* (5).

## Footnotes

↵4 Nonstandard abbreviations:

- LOD
- limit of detection.

**Authors' Disclosures or Potential Conflicts of Interest:***Upon manuscript submission, all authors completed the author disclosure form.***Employment or Leadership:**J.E. Vaks, Roche Molecular Diagnostics; M. Rullkoetter, Roche Diagnostics; M.J. Santulli, Roche Molecular Systems; N. Schoenbrunner, Roche Molecular Systems.**Consultant or Advisory Role:**None declared.**Stock Ownership:**M.J. Santulli, Roche Molecular Systems; N. Schoenbrunner, Roche Molecular Systems.**Honoraria:**None declared.**Research Funding:**None declared.**Expert Testimony:**None declared.**Patents:**None declared.**Role of Sponsor:**No sponsor was declared.

- Received May 18, 2016.
- Accepted August 10, 2016.

- © 2016 American Association for Clinical Chemistry