Statistical Methods for Analysis of Diagnostic Accuracy Studies
This presentation is the property of its rightful owner.
Sponsored Links
1 / 54

Measures of diagnostic accuracy PowerPoint PPT Presentation


  • 105 Views
  • Uploaded on
  • Presentation posted in: General

Statistical Methods for Analysis of Diagnostic Accuracy Studies Jon Deeks University of Birmingham with acknowledgement to Hans Reitsma. Measures of diagnostic accuracy. Positive and negative predictive values Sensitivity and specificity Likelihood ratios Area under the ROC curve

Download Presentation

Measures of diagnostic accuracy

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Measures of diagnostic accuracy

Statistical Methods for Analysis of Diagnostic Accuracy StudiesJon DeeksUniversity of Birminghamwith acknowledgement to Hans Reitsma


Measures of diagnostic accuracy

Measures of diagnostic accuracy

  • Positive and negative predictive values

  • Sensitivity and specificity

  • Likelihood ratios

  • Area under the ROC curve

  • Diagnostic odds ratio


Diagnostic accuracy studies

Diagnostic accuracy studies

  • Results from the index test are compared with the results obtained with the reference standard on the same subjects

  • Accuracy refers to the degree of agreement between the results of the index test and those from the reference standard


Basic design

Basic Design

Series of patients

Index test

Reference standard

Cross-classification


Clinical problem

Clinical problem

  • Diagnostic value of B type natriuretic (BNP) measurement

  • Does BNP measurement distinguish between those with and without left ventricular dysfunction in the elderly?

  • Smith et al. BMJ 2000; 320: 906.


Anatomy of diagnostic study

Anatomy of diagnostic study

  • Target population: unscreened elderly

  • Index test: BNP

  • Target condition: LVSD

  • Final diagnosis (reference standard): echocardiography – global and regional assessment of ventricular function including measurement of LV ejection fraction


Our example

Our example

Elderly patients

BNP measurement

Echocardiography for LVSD

Cross-classification


Results of bnp study

Results of BNP study


Measures of test performance

Measures of test performance

  • sensitivity

    • 11 / 12 = 92% < Pr(T+|D+) >

  • specificity

    • 93 / 143 = 65% < Pr(T-|D-) >


Measures of test performance1

Measures of test performance

  • positive predictive value

  • 11 / 61 = 18% < Pr(D+|T+) >

  • negative predictive value

  • 93 / 94 = 99% < Pr(D-|T-) >


Sensivity and specificity not directly affected by prevalence

Sensivity and Specificity not directly affected by prevalence

  • sensitivity

    • 131 / 143 = 92%

  • specificity

    • 93 / 143 = 65%


Predictive values directly affected by prevalence

Predictive values directly affected by prevalence

  • positive predictive value

  • 131 / 181 = 72%

  • negative predictive value

  • 93 / 105 = 89%


Do sensitivity and specificity vary with prevalence

Do sensitivity and specificity vary with prevalence?

  • Test performance is sometimes observed to be different in different settings, patient groups, etc.

  • Occasionally attributed to differences in disease prevalence, but:

    • diseased and non-diseased spectrums differ as well.

  • e.g. using a test in primary care and secondary care referrals

    • the diseased group are different (cases more difficult)

    • the non-diseased group are different (conditions more similar)

    • sensitivity may decrease, specificity certainly decreases


Likelihood ratios

Likelihood ratios

  • Why likelihood ratios?

  • Applicable in situations with more than 2 test outcomes

  • Direct link from pre-test probabilities to post-test probabilities


Likelihood ratios1

Likelihood ratios

  • Information value of a test result expressed as likelihood ratio


Likelihood ratio of positive test

Likelihood Ratio of positive test

  • How more often a positive test result occurs in persons with compared to those without the target condition


Likelihood ratios2

Likelihood ratios

  • Likelihood ratio of a negative test result

  • How less likely a negative test result is in persons with the target condition compared to those without the target condition


Likelihood ratios3

Likelihood ratios


Calculate likelihood ratios from column percentages

Calculate likelihood ratios from column percentages


Interpreting likelihood ratios

Interpreting likelihood ratios

  • A LR=1 indicates no diagnostic value

  • LR+ >10 are usually regarded as a ‘strong’ positive test result

  • LR- <0.1 are usually regarded as a strong negative test result

  • But it depends on what change in probability is needed to make a diagnosis


Measures of diagnostic accuracy

92%

LR+ = 10

55%

10%

50%


Advantages of likelihood ratios

Advantages of likelihood ratios

  • Still useful when there are more than 2 test outcomes


Bnp is a continuous measurement

BNP is a continuous measurement

  • Dichotomisation of BNP(high vs. low) means loss of information

  • Higher values of BNP are more indicative of LVSD


Results bnp study

Results BNP study


Likelihood ratios4

Likelihood ratios

  • Stratum specific likelihood ratios in case of more than 2 test results


Compute lr from column percentages

Compute LR from column percentages


Bayes rule

Bayes’ rule

Post-test odds for disease

=

Pre-test odds for disease x Likelihood ratio


Bayes rule1

Bayes’ rule

  • Pre-test odds

    • chance of disease expressed in odds

    • example: if 2 out of 5 persons have the disease: probability = 2/5 in odds = 2/3


Bayes rule2

Bayes’ rule

  • odds = probability / (1 – probability)

  • probability = odds / (1 + odds)


Bayes rule patient with bnp 26 7

Bayes’ rulepatient with BNP >26.7

  • Pre-test probability = 0.5

  • Pre-test odds = 0.5 / (1-0.5) = 1

  • LR(BNP >26.7) = 3.83

  • Post-test odds = 1x3.83 = 3.83

  • Post-test probability = 3.83 / (1+3.83) = 0.79


Bayes rule patient with bnp lower than 18 7

Bayes’ rulepatient with BNP lower than 18.7

  • Pre-test probability = 0.5

  • Pre-test odds = 0.5 / (1-0.5) = 1

  • LR(CK< 40) = 0.13

  • Post-test odds = 1 x 0.13 = 0.13

  • Post-test probability = 0.13 / (1+0.13)

    = 0.12


Probability for lvsd after bnp

Probability for LVSD after BNP


Measures of diagnostic accuracy

79%

52%

12%

50%


Measures of diagnostic accuracy

5%

17%

5%

1%


Probability for lvsd after bnp1

Probability for LVSD after BNP


Confidence intervals

Confidence intervals

  • Sample uncertainty should be described for all statistics, using confidence intervals

+ gives upper limit - gives lower limit

Standard error of estimate

estimate of effect

Normal deviate (1.96 for 95% CI)


Confidence intervals for proportions

Confidence Intervals for Proportions

  • Sensitivity, specificity, positive and negative predictive values, and overall accuracy are all proportions


Exact or asymptotic ci

Exact or Asymptotic CI?

  • Asymptotic CI are approximations

  • Inappropriate when

    • proportion is near 0% or near 100%

    • sample sizes are small

      (confidence intervals are not symmetric in these cases)

  • Preferable to use Binomial exact methods

    • can be computed in many statistics packages

    • or refer to tables


Comparison of asymptotic and exact methods

Comparison of Asymptotic and Exact Methods


Confidence intervals for ratios of probabilities and odds

Confidence Intervals for Ratios of Probabilities and Odds

  • Odds ratios are ratios of odds

  • Likelihood ratios are ratios of probabilities


Cis for study

CIs for study

  • Sensitivity = 92% (62%, 100%)

  • Specificity = 65% (57%, 73%)

  • PPV = 82% (70%, 91%)

  • NPV = 99% (94%, 100%)

  • LR(>= 26.7) = 3.8 (2.4, 6.1)

  • LR(18.7 < 26.7) = 1.1 (0.3, 4.1)

  • LR(<18.7) = 0.13 (0.02, 0.84)


Roc curve

ROC-curve

  • ROC stands for Receiver Operating Characteristic

  • ROC-curve shows the pairs of sensitivity and specificity that correspond to various cut-off points for the continuous test result


Continuous diagnostic test results

Continuous diagnostic test results


Heterogeneity in threshold

Heterogeneity in Threshold


Heterogeneity in threshold1

Heterogeneity in Threshold


Heterogeneity in threshold2

Heterogeneity in Threshold


Heterogeneity in threshold3

Heterogeneity in Threshold


Heterogeneity in threshold4

Heterogeneity in Threshold


Threshold effects

Threshold effects

Decreasing threshold increases sensitivity but decreases specificity

Increasing threshold increases specificity but decreases sensitivity


Change in cut off value and effect on sens spec

Change in cut-off valueand effect on sens & spec


Roc curve bnp

ROC-curve BNP

Cut-off:  18.7

Cut-off:  19.8

Cut-off:  26.7


Roc curve1

ROC curve

  • Shows the effect of different cut-off values on sensitivity and specificity

  • Better tests have curves that lie closer to the upper left corner

  • Area under the ROC is a single measure of test performance (higher is better)

  • Shape

    • RAW continuous data gives steps

    • GROUPED data gives straight sloping lines

    • FITTED ROC curves are smoothed.


Variation in diagnostic threshold

Variation in diagnostic threshold

At what level, is a test result categorised as +ve, and how should the threshold be selected?

Threshold affects the performance of the test, as described by ROC curves, and likelihood ratios

Depends on

disease prevalence (affects +ve and -ve predictive values)

relative costs of false positive and false negative misdiagnoses

relative benefits of true positive and true negative diagnoses


Workshop exercise erratum

LVSD

+ve

-ve

MI or BNP

+ve

36

63

-ve

4

23

40

86

Workshop exercise – erratum

  • Q16 page 8

    Compute post-test probabilities for a high risk patient, pre-test prob=50%

    Q19 page 10


  • Login