1 / 32

Borrowed Liberally from Riegelman and Hirsch, 2 nd Edition

Studying a Study and Testing a Test: Sensitivity Training, “Don’t Make a Good Test Bad”, and “Analyze This”. Borrowed Liberally from Riegelman and Hirsch, 2 nd Edition. What you need to know:. When will ordering a given test be helpful in making a decision on therapy?

byron
Download Presentation

Borrowed Liberally from Riegelman and Hirsch, 2 nd Edition

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Studying a Study andTesting a Test:Sensitivity Training, “Don’t Make a Good Test Bad”, and “Analyze This” Borrowed Liberally from Riegelman and Hirsch, 2nd Edition

  2. What you need to know: • When will ordering a given test be helpful in making a decision on therapy? • When is a test more likely to be misleading? • Which test should I order when? • How do I evaluate the literature regarding the value of different tests?

  3. Sensitivity and Specificity: • Sensitivity = TP/(TP+FN) • Specificity = TN/(TN+FP)

  4. Scenario #1 • You read an article describing a new rapid test for diagnosis of Herpes Simplex virus infection. It was used on samples from 1000 neonatal patients with CSF pleocytosis (>30 wbc/mm3) elevated protein (>50), with negative gram stain, and the results were compared with a gold-standard test. The following results were obtained.

  5. Sensitivity and Specificity: • Question 1: What are the Sensitivity and Specificity of this test? • Question 2: Is this a Good Test? 500 500

  6. Sensitivity and Specificity:Answer to Questions 1 & 2 • Sensitivity = True Positives Divided by Gold Standard Diseased (True positives plus false negatives) = 400/500 = 80% • Specificity = True Negatives Divided by Gold Standard Disease-free (True negatives plus false positives = 450/500 = 90% • Good Test?

  7. Take Home Message #1 • Sensitivity and Specificity are Properties of the Test!!!

  8. Scenario #2 • The same diagnostic test becomes commercially available and your hospital decides that for “medicolegal reasons”, it should be done on all CSF samples, regardless of cell count and protein results. The results of the next 1,000 tests are shown on the following table.

  9. Sensitivity and Specificity • Question 3: What are the sensitivity and specificity now? • Question 4: Is this a good test? 100 900

  10. Answers to 3 and 4 • Sensitivity = 80/100=80% • Specificity = 810/900=90% • Specificity and Sensitivity do not change when you overuse the test, but the value of the positive result is less • Good test? • …. 90 of the 170 patients with positive tests are actually disease-free…

  11. Predictive Values • Predictive value of a positive test = TP/(TP+FP) • Predictive value of a negative test = TN/(TN+FN) 100 900

  12. Positive and Negative Predictive Value • Predictive value of a positive test = Proportion of those with a positive test who have the disease = 80/170 = 47.1% • Predictive value of a negative test = Proportion of those with a negative test who are disease-free = 810/830 = 97.6%

  13. Take Home Message #2 • The Predictive Value of a test depends upon the prevalence of the disease in the population in which it is applied!! • Corollary: You can make a good test into a bad test by using it in a population with a very low prevalence of the disease

  14. Scenario #3 • A new extended screening test is being piloted for medium chain acyl-CoA dehydrogenase deficiency (MCAD) on newborn blood spots. The following results are obtained:

  15. Predictive Value • Question 3: What are the sensitivity, specificity, positive and negative predictive values now? • Question 4: Is this a good screening test? 100 9900

  16. Answers to Questions 5 and 6 • Sensitivity = 99/100 = 99% • Specificity = 8100/9900 = 82% • Predictive value of positive = 99/1899 = 5.2% • Predictive value of negative= 8100/8101= 99.99%

  17. Characteristics of a Good Screen • Very high NEGATIVE PREDICTIVE value (implies high sensitivity) • Availability of follow-up Gold Standard test to confirm (may be repetition of original test) • Availability of counseling and education • Intervention that affects outcome

  18. Risks and Odds • Sometimes you don’t know the prevalence of the disease within a given situation, so you need to evaluate that from the literature • Two questions: • What is the risk of a given pathology in the context of a given risk factor? • What are the odds before and after you assess for the presence of a risk factor?

  19. Scenario #4: You see a febrile infant in the midst of an aseptic meningitis outbreak, but this one has never had Prevnar. You have access to some raw prospective data on the use of Prevnar in your area: What is the relative risk of Invasive disease in an unimmmunized patient?

  20. Relative Risk • Relative Risk = probability of disease with the risk factor ÷ probability without the risk factor • 0.030/0.003 = 10 • BUT, this only works if the data is generated PROSPECTIVELY

  21. What if this data were generated RETROSPECTIVELY, i.e., taking kids who had invasive disease and selecting matched controls. What is the odds ratio of Invasive disease in an unimmmunized patient?

  22. Odds ratio • Odds ratio: odds of having a risk factor in the diseased population ÷ odds of having the risk factor in a matched control population • 90/10 ÷ 45/55 = 9 ÷ 0.82 = 11

  23. Scenario #5 • An article outlines guidelines for referral to a pediatric cardiologist for infants and children with cardiac murmurs. The article describes a decision analysis methodology that allows you to compare the effectiveness of a two different paradigms, one in which EKG is combined with CXR at the same time, the other in which only abnormal quality murmurs are sent on for Echo.

  24. Decision Analysis:Serial Application of Tests

  25. Decision Analysis:Parallel Application of Tests

  26. Which of the following is true? A. In a serial testing approach, the initial test must meet most criteria of a good screening test. B. In a parallel approach, more diagnostic value is gained if the outcomes of the 2 tests are independent [e.g., CXR is less useful in asthmatics, since wheeze and atelectasis occur together in a non-random fashion] C. A and B are both true.

  27. Interventional Studies • Assignment to Groups • Assessment of outcomes • Analysis • Interpretation

  28. Assignment (Avoid Bias) • Prospective vs. Retrospective • Randomized vs. Selected • Blinded vs. Open-label • Placebo

  29. Assessment • Define variables prior to initiation of study • Choose a good test!!!

  30. Analysis • State Hypothesis as null: “There is no difference between treatment and control groups” • Type I error: Falsely reject the null hypothesis, P value (0.05) is likelihood of type I • Type II error: Falsely accept the null hypothesis, Power = 1.0 - type II error • Power of 0.8 is standard

  31. 3 determinants of Power • Variability of the test • Incremental change • Sample size

  32. Interpretation • What was the study population? • Do the results apply to your population?

More Related