1 / 49

Performance of a diagnostic test

Performance of a diagnostic test. Dagmar Rimek EPIET-EUPHEM Introductory Course 2012 Lazareto, Menorca, Spain. Based on the Lecture of 2011 by Steen Ethelberg. Outline. Performance characteristics of a test Sensitivity Specificity Choice of a threshold

marlin
Download Presentation

Performance of a diagnostic test

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Performance of a diagnostic test Dagmar Rimek EPIET-EUPHEM Introductory Course 2012 Lazareto, Menorca, Spain Based on the Lecture of 2011 by Steen Ethelberg

  2. Outline • Performance characteristics of a test • Sensitivity • Specificity • Choice of a threshold • Performance of a test in a population • Positive predictive value of a test (PPV) • Negative predictive value of a test (NPV) • Impact of disease prevalence, sensitivity and specificity on predictive values

  3. Performance characteristics of a test in a laboratory setting

  4. Population with affected and non-affected individuals Affected Non-affected

  5. A perfect diagnostic test identifies the affected individuals only Affected Non-affected

  6. In reality, tests are not perfect Affected Non-affected

  7. Sensitivity of a test • The sensitivity of a test is the ability of the test to identify correctly the affected individuals • Proportion of persons testing positive among affected individuals Affected persons + - True positive (TP) Test result False negative (FN) Sensitivity (Se) = TP / (TP + FN) 7

  8. Estimating the sensitivity of a test • Identify affected individuals with a gold standard • Obtain a wide panel of samples that are representative of the population of affected individuals • Recent and old cases • Severe and mild cases • Various ages and sexes • Test the affected individuals • Estimate the proportion of affected individuals that are positive with the test

  9. Example: Estimating the sensitivity of a new ELISA IgM test for acute Q-fever • Identify persons with acute Q-fever with a gold standard (IgM Immunofluorescence Assay) • Obtain a wide panel of samples that are representative of the population of individuals with acute Q-fever • Recent and old cases • Severe and asymptomatic cases • Various ages and sexes • Test the persons with acute Q-fever • Estimate the proportion of persons with acute Q-fever that are positive with the ELISA IgM test

  10. Example: Sensitivity a new ELISA IgM test for acute Q-fever Sensitivity = TP / (TP + FN)148 / 150 = 98.7% 10

  11. What factors influence the sensitivityof a test? • Characteristics of the affected persons? • YES: Antigenic characteristics of the pathogen in the area(e.g., if the test was not prepared with antigens reflecting the population of pathogens in the area, it will not pick up infected persons in the area) • Characteristics of the non-affected persons? • NO: The sensitivity is estimated on a population of affected persons • Prevalence of the disease? • NO: The sensitivity is estimated on a population of affected persons Sensitivity is an INTRINSIC characteristic of the test

  12. Specificity of a test • The specificity of a test is the ability of the test to identify correctly non-affected individuals • Proportion of persons testing negative among non-affected individuals Non-affected persons + - False positive (FP) Test result True negative (TN) Specificity (Sp) = TN / (TN + FP) 12

  13. Estimating the specificity of a test • Identify non-affected individuals • Negative with a gold standard • Unlikely to be infected • Obtain a wide panel of samples that are representative of the population of non-affected individuals • Test the non-affected individuals • Estimate the proportion of non-affected individuals that are negative with the test

  14. Example: Estimating the specificity of a new ELISA IgM test for acute Q-fever • Identify persons without Q-fever • Persons without sign and symptoms of the infection • Persons at low risk of infection, negative with gold standard (IgM Immunofluorescence Assay) • Obtain a wide panel of samples that are representative of the population of individuals without Q-fever • Test the persons without Q-fever • Estimate the proportion of persons without Q-fever that are negative with the new ELISA IgM test

  15. Specificity of a new ELISA IgM testfor acute Q-fever Specificity = TN / (TN + FP)190 / 200 = 95% 15

  16. What factors influence the specificity of a test? • Characteristics of the affected persons? • NO: The specificity is estimated on a population of non-affected persons • Characteristics of the non-affected persons? • YES: The diversity of antibodies to various other antigens in the population may affect cross reactivity or polyclonal hypergammaglobulinemia may increase the proportion of false positives • Prevalence of the disease? • NO: The specificity is estimated on a population of non-affected persons Specificity is an INTRINSIC characteristic of the test

  17. Performance of a test Disease Yes No + TP FP Test ­ - FN TN TP Se = TP + FN TN Sp = TN + FP

  18. To whom sensitivity and specificity matters most? INTRINSIC characteristics of the test ► To laboratory specialists!

  19. Distribution of quantitative test results among affected and non-affected people Threshold for positive result Ideal situation Non-affected: Affected: Number of people tested TN TP 0 5 10 15 20 Quantitative result of the test

  20. Distribution of quantitative results among affected and non-affected people Realistic situation Non-affected: Threshold for positive result Affected: TN TP Number of people tested FN FP 0 5 10 15 20 Quantitative result of the test

  21. Effect of Decreasing the Threshold Non-affected: Threshold for positive result Affected: FP Number of people tested TP TN FN 0 5 10 15 20 Quantitative result of the test

  22. Effect of Decreasing the Threshold Disease Yes No + TP FP Test ­ - FN TN TP Se = TP + FN TN Sp = TN + FP

  23. Effect of Increasing the Threshold Non-affected: Threshold for positive result Affected: TN Number of people tested TP FN FP 0 5 10 15 20 Quantitative result of the test

  24. Effect of Increasing the Threshold Disease Yes No + TP FP Test ­ FN TN - TP Se = TP + FN TN Sp = TN + FP

  25. Performance of a test and threshold • Sensitivity and specificity vary in opposite directions when changing the threshold (e.g. the cut-off in an ELISA) • The choice of a threshold is a compromise to best reach the objectives of the test • consequences of having false negatives? • consequences of having false positives?

  26. Using several tests • One way out of the dilemma is to use several tests that complement each other • First use test with a high sensitivity(e.g. screening for HIV by ELISA, or for syphilis by TPHA) • Second use test with a high specificity(e.g. confirmation of HIV or syphilis by western blot)

  27. ROC curves • Receiver Operating Characteristics curve • Representation of relationship between sensitivity and specificity for a test • Simple tool to: • Help define best cut-off value of a test • Compare performance of two tests

  28. Prevention of blood transfusion malaria:Choice of an indirect IFA threshold Sensitivity (%) 100 1/10 1/20 1/40 80 1/80 1/160 60 IFA Dilutions 1/320 40 1/640 20 0 0 20 40 60 80 100 100 - Specificity (%): Proportion of false positives

  29. Comparison of performance of IFA and ELISA IgM tests for detection of acute Q-fever Sensitivity (%) 100 80 IFA ELISA 60 Area under the ROC curve (AUC) 40 20 0 0 25 50 75 100 100 - Specificity (%)

  30. Performance of a test in a population

  31. How well does the test perform in a real population? • The test is now used in a real population • This population is made of • Affected individuals • Non-affected individuals • The proportion of affected individuals is the prevalence

  32. Predictive value of a positive test The predictive value of a positive test is the probability that an individual testing positive is truly affected Proportion of affected persons among those testing positive

  33. Positive predictive value (PPV) of a test PPV = A / (A+B) This is only valid for the sample of specimens tested 33

  34. What factors influence the positive predictive value of a test? • Sensitivity? • YES: To some extend. • Specificity? • YES: The more the test is specific, the more it will be negative for non-affected persons (less false-positive results). • Prevalence of the disease? • YES: Low prevalence: Low pre-test probability for positives.The test will pick up more false positives. • YES: High prevalence: High pre-test probability for positives.The test will pick up more true positives.

  35. Positive predictive value of a test according to prevalence and specificity Specificity PPV (%)

  36. Predictive value of a negative test The predictive value of a negative test is the probability that an individual testing negative is truly non-affected Proportion of non-affected persons among those testing negative

  37. Negative predictive value (NPV) of a test NPV = D / (C+D) This is only valid for the sample of specimens tested 37

  38. What factors influence the negative predictive value of a test? • Sensitivity? • YES:The more the test issensitive, the more it captures affected persons (less false negatives). • Specificity? • YES: But to a lesser extend. • Prevalence of the disease? • YES: Low prevalence: High pre-test probability for negatives. The test will pick up more true negatives. • YES: High prevalence: Low pre-test probability for negatives. The test will pick up more false negatives.

  39. Negative predictive value of a test according to prevalence and sensitivity Sensitivity NPV (%)

  40. Relation between predictive values and sensitivity (Se), specificity (Sp), prevalence (Pr) Disease Yes No + (1-Sp)(1-Pr) Se Pr + (1-Sp)(1-Pr) Se Pr Test - (1-Se)Pr Sp(1-Pr) (1-Se)Pr + Sp(1-Pr) Pr 1-Pr

  41. Se Pr = PPV + - - Se Pr (1 Sp)(1 Pr) Sp(1 - Pr) = NPV + - Sp(1 - Pr) (1 Se) Pr Calculate PPV and NPV

  42. Relation between predictive values and sensitivity/ specificity Se Pr = PPV + - - Se Pr (1 Sp)(1 Pr) Sp(1 - Pr) = NPV + - Sp(1 - Pr) (1 Se) Pr Increasing specificity increasing PPV Increasing sensitivity increasing NPV

  43. Relation between predictive values and prevalence Se Pr = PPV + - - Se Pr (1 Sp)(1 Pr) Sp(1 - Pr) = NPV + - Sp(1 - Pr) (1 Se) Pr Increasing prevalence  increasing PPV Decreasingprevalence increasingNPV

  44. Example: Screening for acute Q-fever in two settings • ELISA IgM test • Sensitivity = 98% • Specificity = 95% • Population in low endemic area • Prevalence = 0.5% • Patients with atypical pneumonia • Prevalence = 20% • 10,000 tests performed in each group

  45. Example: Screening for acute Q-fever in a population in a low endemic area IgM ELISA test sensitivity = 98% IgM ELISA test specificity = 95% Prevalence = 0.5% PPV = 8.97% NPV = 99.98%

  46. Example: Screening for acute Q-fever in patients with atypical pneumonia IgM ELISA test sensitivity = 98% IgM ELISA test specificity = 95% Prevalence = 20% PPV = 83.05% NPV = 99.48%

  47. To whom predictive values matters most? • Look at denominators! • Persons testing positive • Persons testing negative ► To clinicians • probability that a individual with a positive test is really sick? • probability that a individual with a negative test is really healthy? ► To epidemiologists! • proportion of positive tests corresponding to true patients? • proportion of negative tests corresponding to healthy subjects?

  48. Summary • Sensitivity and specificity matter to laboratory specialists • Studied on panels of positives and negatives • Intrinsic characteristics of a test • Capacity to identify the affected • Capacity to identify the non-affected • Predictive values matter to clinicians and epidemiologists • Studied on homogeneous populations • Dependent on the disease prevalence • Performance of a test in real life • How to interpret a positive test • How to interpret a negative test

  49. Where will you do your rain dance? There? Here?

More Related