Receiver Operating Characteristic Curve
This presentation is the property of its rightful owner.
Sponsored Links
1 / 51

Stanford University, Department of Psychiatry and Behavioral Sciences PowerPoint PPT Presentation


  • 63 Views
  • Uploaded on
  • Presentation posted in: General

Receiver Operating Characteristic Curve (ROC) Analysis for Prediction Studies Ruth O’Hara, Helena Kraemer, Jerome Yesavage, Jean Thompson, Art Noda, Joy Taylor, Jared Tinklenberg. Stanford University, Department of Psychiatry and Behavioral Sciences Stanford University School of Medicine

Download Presentation

Stanford University, Department of Psychiatry and Behavioral Sciences

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Stanford university department of psychiatry and behavioral sciences

Receiver Operating Characteristic Curve (ROC) Analysis for Prediction StudiesRuth O’Hara, Helena Kraemer, Jerome Yesavage, Jean Thompson, Art Noda, Joy Taylor, Jared Tinklenberg

Stanford University, Department of Psychiatry and Behavioral Sciences

Stanford University School of Medicine

Sierra Pacific MIRECC

Veterans Affairs Palo Alto Health Care System


The clinical need for signal detection procedures

The Clinical Need forSignal Detection Procedures

  • Clinical practice is often “hit or miss” therapy

  • Try one thing, if that does not work, try another

  • This is frustrating for the patient and expensive

  • The Goal: find the best treatment for the patient with specific characteristics

  • New news in psychiatry; old hat in internal medicine


Receiver operating characteristic curve roc analysis

Receiver Operating Characteristic Curve (ROC) Analysis

  • Signal Detection Technique

  • Traditionally used to evaluate diagnostic tests

  • Now employed to identify subgroups of a population at differential risk for a specific outcome (clinical decline, treatment response)

  • Identifies moderators


Receiver operating characteristic curve roc analysis1

Receiver Operating Characteristic Curve (ROC) Analysis

Historical Development


Roc analysis historical development 1

ROC Analysis:Historical Development (1)

  • Derived from early radar in WW2 Battle of Britain to address: Accurately identifying the signals on the radar scan to predict the outcome of interest – Enemy planes – when there were many extraneous signals (e.g. Geese)?


Roc analysis historical development 2

ROC Analysis: Historical Development (2)

  • True Positives = Radar Operator interpreted signal as Enemy Planes and there were Enemy planes (Good Result: No wasted Resources)

  • True Negatives = Radar Operator said no planes and there were none (Good Result: No wasted resources)

  • False Positives = Radar Operator said planes, but there were none (Geese: wasted resources)

  • False Negatives = Radar Operator said no plane, but there were planes (Bombs dropped: very bad outcome)


Roc analysis historical development

ROC Analysis:Historical Development

  • Sensitivity = Probability of correctly interpreting the radar signal as Enemy planes among those times when Enemy planes were actually coming

    • SE = True Positives / True Positives + False Negatives

  • Specificity = Probability of correctly interpreting the radar signal as no Enemy planes among those times when no Enemy planes were actually coming

    • SP = True Negatives / True Negatives + False Positives


Roc prediction of enemy planes by raf radar operators

ROC: Prediction of Enemy Planes by RAF Radar Operators


Receiver operating characteristic curve roc analysis applications

Receiver Operating Characteristic Curve (ROC) Analysis Applications:

Evaluating Medical Tests


Roc analysis evaluating medical tests

ROC Analysis: Evaluating Medical Tests

  • The evaluation of the ability of a diagnostic test to identify a disease involves considering:

  • P=Prevalence = occurrence in the population of the outcome of interest (e.g. disease)

  • True Positives

  • True Negatives

  • False Positives

  • False Negatives

  • P=Prevalence=True Positives + False Negatives


Roc analysis medical test evaluation

ROC Analysis: Medical Test Evaluation

  • True Positives = Test states you have the disease when you do have the disease

  • True Negatives = Test states you do not have the disease when you do not have the disease

  • False Positives = Test states you have the disease when you do not have the disease

  • False Negatives = Test states you do not have the disease when you do


Roc analysis evaluating medical tests1

ROC Analysis: Evaluating Medical Tests

  • Sensitivity =The probability of having a positive test result among those with a positive diagnosis for the disease

    • SE = True Positives / True Positives + False Negatives

  • Specificity = The probability of having a negative test result among those with a negative diagnosis for the disease

    • SP = True Negatives / True Negatives + False Positives


The basic tool 2x2

The Basic Tool: 2X2

Sensitivity (SE)=a/PSpecificity (SP)=d/P’


Roc gds test for diagnosis of clinically confirmed depression

ROC: GDS (Test) for Diagnosis of Clinically Confirmed Depression


Which test do you use medical tests evaluation

Which Test Do You Use: Medical Tests Evaluation

  • GDS: SE = .80; SP = .85

  • Beck Depression Inventory: SE = .85; SP = .75

  • Major Depression Inventory = SE = .66; SP = .63


Roc analysis

ROC Analysis

  • ROC first calculates Sensitivity and Specificity

  • Quality Indices measures the quality of the sensitivity and specificity

  • ROC computes the quality indices for each predictor to find the ones with optimal sensitivity and specificity


To detect the optimal sensitivity and specificity

To Detect the Optimal Sensitivity and Specificity

  • Depends on the relative CLINICAL importance of false negatives versus false positives.

    • W=1 means only false negatives matter.

    • W=0 means only false positives matter.

    • W=1/2 means both matter equally.

  • Analytically: Use weighted kappa.


Roc analysis1

ROC Analysis

  • P = TP + FNP’= 1- (TP + FN)

  • Q = TP + FPQ’= 1- (TP + FP)

  • EFF = TP + TN

  • κ(0.5, 0) = [ (TP + TN) - (TP + FN)(TP+FP) - (1-(TP + FN)(1-(TP + FP))]

    [1 – (TP + FN)(TP+FP) - (1-(TP + FN))(1-(TP + FP))]


Roc plane and curve

ROC Plane and “Curve”

ROC “curve”

Ideal Point

Random ROC

(Q,Q)

(P,P)


Receiver operating characteristic curve roc analysis applications1

Receiver Operating Characteristic Curve (ROC) Analysis Applications

Identifying Predictors of Clinical Outcome


Roc analysis prediction studies dr kraemer

ROC Analysis: Prediction Studies (Dr. Kraemer)

  • ROC can identify predictors/characteristics

    of patients that are at differential risk for a specific outcome of interest. e.g. What are the Characteristics of AD Patients at risk for rapid decline and are high priority for treatment?

  • What are the clinical predictors of Alzheimer Disease patients who are “good responders” (or “poor responders”) to cholinesterase inhibitor treatments?

  • Useful in “real world” clinical medicine where multiple variables affect the clinical outcome and patients seldom have one pure diagnosis


Roc identifying predictors of an outcome

ROC: Identifying Predictors of an Outcome

  • 1. ROC relates a predictor (test) to the clinical outcome of interest (Diagnosis/Gold Standard)

  • 2. ROC searches all predictors and their associated cut-points

  • 3. ROC determines which predictor and associated cut-point yields the optimal sensitivity and specificity for identifying the outcome of interest yielding two groups at differential risk for the outcome


Roc identifying predictors of an outcome1

ROC: Identifying Predictors of an Outcome

  • 4. ROC is an iterative process that is then rerun automatically for each group yielded in Step 3. in order to examine which predictor and associated cut-point may further divide the groups

  • 5. ROC will keep searching within each group yielded until one of three stopping rules apply (see Stopping rule slide)

  • 6. ROC thus identifies subgroups of individuals that are at increased risk for the outcome of interest


Roc analysis advantages and disadvantages

ROC Analysis:Advantages and Disadvantages

  • No assumptions of normal distribution

  • Multiple predictors can be evaluated simultaneously

  • Indicates interactions among predictors

  • Indicates cut-points on these predictors

  • Yields clinically relevant information

  • Non-hypothesis testing

  • Requires large samples

  • Capitalizes on chance: needs stringent stopping rule


Roc analysis procedure

ROC Analysis: Procedure

  • Start with large sample size

  • Define the outcome of interest (always binary)

  • Choose Success/Failure criteria

  • Select predictor variables of interest (as many as you like)

  • Run ROC Program that systematically finds best predictors for Success/Failure


The basic tool 2x21

The Basic Tool: 2X2

Sensitivity (SE)=a/PSpecificity (SP)=d/P’


Roc identifying predictors their cut points

ROC: Identifying Predictors & Their Cut-points

  • Dichotomous Variables such as Gender:

    • ROC calculates the Se and Sp for Female vs. Male

  • For Continuous Variables such as Age:

    • ROC would calculate Se and Sp for the cut-point of 60 vs. 61+62+63 ….85; then could calculate for cut-point of 60+61 vs. 62+63+64 ….85, and so forth.


Roc gender as predictor of clinically confirmed depression

ROC: Gender as Predictor ofClinically Confirmed Depression


Roc identifying predictors their cut points1

ROC: Identifying Predictors & Their Cut-points

  • Dichotomous Variables: ROC calculates the Se and Sp for Female vs. Male, Aphasia vs. No Aphasia, etc.

  • For Continuous Variables such as Age:

    • ROC would calculate Se and Sp for the cut-point of 60 vs. 61+62+63 ….85; then could calculate for cut-point of 60+61 vs. 62+63+64 ….85, and so forth.


Roc age as predictor of clinically confirmed depression

ROC: Age as Predictor of Clinically Confirmed Depression


Roc age as predictor of clinically confirmed depression1

ROC: Age as Predictor of Clinically Confirmed Depression


Receiver operating characteristic curve roc analysis2

Receiver Operating Characteristic Curve (ROC) Analysis

Conducting the ROC: An Example


Roc analysis procedure1

ROC Analysis: Procedure

  • Start with large sample size

  • Define the outcome of interest

  • Choose Success/Failure criteria

  • Identify predictor variables of interest

  • Run ROC Program that systematically finds best predictors for Success/Failure


Roc analysis example

ROC Analysis: Example

  • Population under investigation: 1, 472 AD patients from 10 Centerswith a 12 month follow-up

  • Clinically significant outcome:More rapid decline as defined by a loss of 3 or more MMSE points per year, post-visit

    O'Hara R et al. (2002). Which Alzheimer patients are at risk for rapid cognitive decline? J Geriatr Psychiatry Neurol;15(4):233-8.


Predictor variables

Predictor Variables

  • Age-at -patient-visit

  • Reported age of symptom onset

  • Gender

  • Years of education

  • Ethnicity

  • MMSE score

  • Living Arrangement

  • Presence of Aphasia

  • Presence of Hallucinations

  • Presence of Extrapyramidal Signs


Stopping rules

Stopping Rules

  • No more possibilities (rare!)

  • Inadequate sample size

  • Optimal test (if ‘a priori’) would not have been statistically significant (p<.001)


Stanford university department of psychiatry and behavioral sciences

N=61 (12%)P=.02

N=43 (8%)P=.19

N=57 (11%)P=.30

Figure 10.3

N=512 (100%)P=.53

Non-minority

Minority

N = 191 (37%)P=.25

N = 321 (63%)P=.70

Bayley Mental Dev. Index < 115

Bayley Mental Dev. Index ≥ 115

Mother neverattended college

Mother attended college

N=110 (21%)P=.48

N=87 (17%)P=.45

N=104 (20%)P=.09

N=211 (41%)P=.81

Bayley Mental Dev. Index<106

Bayley Mental Dev. Index≥106

Bayley Mental Dev. Index<106

Bayley Mental Dev. Index≥106

Attended, didnot graduate

Graduatedfrom college

N=131 (26%)P=.91

N=80 (16%)P=.65

N=30 (6%)P=.73

ROC Decision Tree for IHDP Control group with outcome of low IQ at age 3. (w= 0.5)


Roc plane and swarm of points

ROC Plane and “Swarm” of Points

ROC

“curve”


To detect the optimal sensitivity and specificity1

To Detect the Optimal Sensitivity and Specificity

  • Depends on the relative CLINICAL importance of false negatives versus false positives.

    • W=1 means only false negatives matter.

    • W=0 means only false positives matter.

    • W=1/2 means both matter equally.

  • Analytically: Use weighted kappa.

  • Geometrically: Draw a line through the Ideal Point with slope determined by P and w. Push this line down until it just touches the ROC “curve”. That point is optimal.


Roc analysis conclusion

ROC Analysis: Conclusion

  • Yields Clinically Relevant Information

  • Identifies complex interactions

  • Identifies individuals with different characteristics but at the same risk for the clinically relevant outcome

  • Identifies individuals at the least risk

  • Can take differential clinical costs of false positives and false negatives into account


Conclusion

Conclusion

  • It is not sufficient to identify risk factors or even to identify moderators and mediators etc. or a structural model.

  • It is necessary to present and interpret the results so that clinicians, policy makers, consumers, other researchers can apply them.

  • ROC trees are one method to accomplish this purpose.


Receiver operating characteristic curve roc analysis3

Receiver Operating Characteristic Curve (ROC) Analysis

Using the ROC Program


Using the roc program a how to get the roc program

Using the ROC ProgramA. How to Get the ROC Program

  • Go to http://mirecc.stanford.edu

  • Go to “Top Information Requests”

  • Go to “ROC4 is available for download HERE.”

  • Double Click on “HERE”

  • A pop-up window will give you the option to open or save the ROC4 zip file

  • Best option is to save it to a folder you have already created e.g. ROC analysis


Using the roc program b opening the roc program

Using the ROC ProgramB. Opening the ROC Program

  • Go to your “ROC analysis” folder

  • Unzip the ROC4.zipfile (Some computers will automatically unzip when you double click on it or you may need to use an unzip program)

  • Once unzipped the following 5 files will appear

    • Read_Me.docA help file which explains what to do

    • ROC4.19.exeThe actual ROC program

    • rDemoData.batBatch file that gets ROC program to run

    • Demo.txtA demo data input file

    • runDemoData.docA demo data output file


What the files look like

What the Files Look Like


Using the roc program c preparing data for roc program

Using the ROC ProgramC. Preparing Data for ROC Program

  • First prepare your data file

  • Put your data in Excel form

  • Your outcome measure should always be:

    • Dichotomous

    • Coded as a 1 or 0

    • In the far right column

  • All dichotomous predictor variables coded as a 1 or 0

  • All missing data coded as –9999.99

  • Remove all IDs or other non-predictor information

  • Save your Excel data file as Text (Tab delimited)

  • Give it a name that has no spaces: This will be your data input file


What your data input file looks like

What Your Data Input File Looks Like


Using the roc program d executing the roc program

Using the ROC ProgramD. Executing the ROC Program

  • Open up Microsoft Word

  • Within Word open the rDemoData batch file

  • It will open to read as follows

    echo "Program running- check folder with output file and REFRESH to confirm running “

    roc4.19 Demo.txt 50> runDemoData.doc

  • Where you see “Demo”, you replace with the name of your data input file

  • Where you see “runDemoData”, you replace with the name of your data output file

  • Then save your new batch file with a new name and put .bat at the end of the name (easiest name is one associated with the data names you have assigned).


Using the roc program d an example of executing the roc program

Using the ROC ProgramD. An Example of Executing the ROC Program

  • Helena has data entitled “Workshop” saved as text and now called “Workshopdata.txt”

  • Within Word open the rDemoData batch file to read as follows

    echo "Program running- check folder with output file and REFRESH to confirm running “

    roc4.19 Demo.txt 50> runDemoData.doc

  • “Demo” is replaced with “Workshopdata”

  • “runDemoData” is replaced with “runWorkshopdata”

  • New batch file is saved as “rWorkshopdata.bat”

    echo "Program running- check folder with output file and REFRESH to confirm running “

    roc4.19 Workshopdata.txt 50> runWorkshopdata.doc

  • Double Left Click on new batch file and as if by magic your output file entitled “runWorkshopdata.doc” will appear


Using the roc program e how to read your output file

Using the ROC ProgramE. How to Read Your Output File

  • Open up your data output file which will be in Word

  • Select All

  • Change Font to 6

  • Go to Page Setup and change from Portrait to Landscape

  • Expand your margins if you are still getting wrap around


  • Login