Loading in 2 Seconds...

Stanford University, Department of Psychiatry and Behavioral Sciences

Loading in 2 Seconds...

- 88 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about 'Stanford University, Department of Psychiatry and Behavioral Sciences' - kalia-randall

Download Now**An Image/Link below is provided (as is) to download presentation**

Download Now

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

### Receiver Operating Characteristic Curve (ROC) Analysis

### Receiver Operating Characteristic Curve (ROC) Analysis Applications:

### Receiver Operating Characteristic Curve (ROC) Analysis Applications

### Receiver Operating Characteristic Curve (ROC) Analysis

### Receiver Operating Characteristic Curve (ROC) Analysis

Receiver Operating Characteristic Curve (ROC) Analysis for Prediction StudiesRuth O’Hara, Helena Kraemer, Jerome Yesavage, Jean Thompson, Art Noda, Joy Taylor, Jared Tinklenberg

Stanford University, Department of Psychiatry and Behavioral Sciences

Stanford University School of Medicine

Sierra Pacific MIRECC

Veterans Affairs Palo Alto Health Care System

The Clinical Need forSignal Detection Procedures

- Clinical practice is often “hit or miss” therapy
- Try one thing, if that does not work, try another
- This is frustrating for the patient and expensive
- The Goal: find the best treatment for the patient with specific characteristics
- New news in psychiatry; old hat in internal medicine

Receiver Operating Characteristic Curve (ROC) Analysis

- Signal Detection Technique
- Traditionally used to evaluate diagnostic tests
- Now employed to identify subgroups of a population at differential risk for a specific outcome (clinical decline, treatment response)
- Identifies moderators

Historical Development

ROC Analysis:Historical Development (1)

- Derived from early radar in WW2 Battle of Britain to address: Accurately identifying the signals on the radar scan to predict the outcome of interest – Enemy planes – when there were many extraneous signals (e.g. Geese)?

ROC Analysis: Historical Development (2)

- True Positives = Radar Operator interpreted signal as Enemy Planes and there were Enemy planes (Good Result: No wasted Resources)
- True Negatives = Radar Operator said no planes and there were none (Good Result: No wasted resources)
- False Positives = Radar Operator said planes, but there were none (Geese: wasted resources)
- False Negatives = Radar Operator said no plane, but there were planes (Bombs dropped: very bad outcome)

ROC Analysis:Historical Development

- Sensitivity = Probability of correctly interpreting the radar signal as Enemy planes among those times when Enemy planes were actually coming
- SE = True Positives / True Positives + False Negatives
- Specificity = Probability of correctly interpreting the radar signal as no Enemy planes among those times when no Enemy planes were actually coming
- SP = True Negatives / True Negatives + False Positives

Evaluating Medical Tests

ROC Analysis: Evaluating Medical Tests

- The evaluation of the ability of a diagnostic test to identify a disease involves considering:
- P=Prevalence = occurrence in the population of the outcome of interest (e.g. disease)
- True Positives
- True Negatives
- False Positives
- False Negatives
- P=Prevalence=True Positives + False Negatives

ROC Analysis: Medical Test Evaluation

- True Positives = Test states you have the disease when you do have the disease
- True Negatives = Test states you do not have the disease when you do not have the disease
- False Positives = Test states you have the disease when you do not have the disease
- False Negatives = Test states you do not have the disease when you do

ROC Analysis: Evaluating Medical Tests

- Sensitivity =The probability of having a positive test result among those with a positive diagnosis for the disease
- SE = True Positives / True Positives + False Negatives
- Specificity = The probability of having a negative test result among those with a negative diagnosis for the disease
- SP = True Negatives / True Negatives + False Positives

The Basic Tool: 2X2

Sensitivity (SE)=a/P Specificity (SP)=d/P’

Which Test Do You Use: Medical Tests Evaluation

- GDS: SE = .80; SP = .85
- Beck Depression Inventory: SE = .85; SP = .75
- Major Depression Inventory = SE = .66; SP = .63

ROC Analysis

- ROC first calculates Sensitivity and Specificity
- Quality Indices measures the quality of the sensitivity and specificity
- ROC computes the quality indices for each predictor to find the ones with optimal sensitivity and specificity

To Detect the Optimal Sensitivity and Specificity

- Depends on the relative CLINICAL importance of false negatives versus false positives.
- W=1 means only false negatives matter.
- W=0 means only false positives matter.
- W=1/2 means both matter equally.
- Analytically: Use weighted kappa.

ROC Analysis

- P = TP + FN P’= 1- (TP + FN)
- Q = TP + FP Q’= 1- (TP + FP)
- EFF = TP + TN
- κ(0.5, 0) = [ (TP + TN) - (TP + FN)(TP+FP) - (1-(TP + FN)(1-(TP + FP))]

[1 – (TP + FN)(TP+FP) - (1-(TP + FN))(1-(TP + FP))]

Identifying Predictors of Clinical Outcome

ROC Analysis: Prediction Studies (Dr. Kraemer)

- ROC can identify predictors/characteristics

of patients that are at differential risk for a specific outcome of interest. e.g. What are the Characteristics of AD Patients at risk for rapid decline and are high priority for treatment?

- What are the clinical predictors of Alzheimer Disease patients who are “good responders” (or “poor responders”) to cholinesterase inhibitor treatments?
- Useful in “real world” clinical medicine where multiple variables affect the clinical outcome and patients seldom have one pure diagnosis

ROC: Identifying Predictors of an Outcome

- 1. ROC relates a predictor (test) to the clinical outcome of interest (Diagnosis/Gold Standard)
- 2. ROC searches all predictors and their associated cut-points
- 3. ROC determines which predictor and associated cut-point yields the optimal sensitivity and specificity for identifying the outcome of interest yielding two groups at differential risk for the outcome

ROC: Identifying Predictors of an Outcome

- 4. ROC is an iterative process that is then rerun automatically for each group yielded in Step 3. in order to examine which predictor and associated cut-point may further divide the groups
- 5. ROC will keep searching within each group yielded until one of three stopping rules apply (see Stopping rule slide)
- 6. ROC thus identifies subgroups of individuals that are at increased risk for the outcome of interest

ROC Analysis:Advantages and Disadvantages

- No assumptions of normal distribution
- Multiple predictors can be evaluated simultaneously
- Indicates interactions among predictors
- Indicates cut-points on these predictors
- Yields clinically relevant information
- Non-hypothesis testing
- Requires large samples
- Capitalizes on chance: needs stringent stopping rule

ROC Analysis: Procedure

- Start with large sample size
- Define the outcome of interest (always binary)
- Choose Success/Failure criteria
- Select predictor variables of interest (as many as you like)
- Run ROC Program that systematically finds best predictors for Success/Failure

The Basic Tool: 2X2

Sensitivity (SE)=a/P Specificity (SP)=d/P’

ROC: Identifying Predictors & Their Cut-points

- Dichotomous Variables such as Gender:
- ROC calculates the Se and Sp for Female vs. Male
- For Continuous Variables such as Age:
- ROC would calculate Se and Sp for the cut-point of 60 vs. 61+62+63 ….85; then could calculate for cut-point of 60+61 vs. 62+63+64 ….85, and so forth.

ROC: Identifying Predictors & Their Cut-points

- Dichotomous Variables: ROC calculates the Se and Sp for Female vs. Male, Aphasia vs. No Aphasia, etc.
- For Continuous Variables such as Age:
- ROC would calculate Se and Sp for the cut-point of 60 vs. 61+62+63 ….85; then could calculate for cut-point of 60+61 vs. 62+63+64 ….85, and so forth.

Conducting the ROC: An Example

ROC Analysis: Procedure

- Start with large sample size
- Define the outcome of interest
- Choose Success/Failure criteria
- Identify predictor variables of interest
- Run ROC Program that systematically finds best predictors for Success/Failure

ROC Analysis: Example

- Population under investigation: 1, 472 AD patients from 10 Centerswith a 12 month follow-up
- Clinically significant outcome:More rapid decline as defined by a loss of 3 or more MMSE points per year, post-visit

O'Hara R et al. (2002). Which Alzheimer patients are at risk for rapid cognitive decline? J Geriatr Psychiatry Neurol;15(4):233-8.

Predictor Variables

- Age-at -patient-visit
- Reported age of symptom onset
- Gender
- Years of education
- Ethnicity
- MMSE score
- Living Arrangement
- Presence of Aphasia
- Presence of Hallucinations
- Presence of Extrapyramidal Signs

Stopping Rules

- No more possibilities (rare!)
- Inadequate sample size
- Optimal test (if ‘a priori’) would not have been statistically significant (p<.001)

N=43 (8%)P=.19

N=57 (11%)P=.30

Figure 10.3

N=512 (100%)P=.53

Non-minority

Minority

N = 191 (37%)P=.25

N = 321 (63%)P=.70

Bayley Mental Dev. Index < 115

Bayley Mental Dev. Index ≥ 115

Mother neverattended college

Mother attended college

N=110 (21%)P=.48

N=87 (17%)P=.45

N=104 (20%)P=.09

N=211 (41%)P=.81

Bayley Mental Dev. Index<106

Bayley Mental Dev. Index≥106

Bayley Mental Dev. Index<106

Bayley Mental Dev. Index≥106

Attended, didnot graduate

Graduatedfrom college

N=131 (26%)P=.91

N=80 (16%)P=.65

N=30 (6%)P=.73

ROC Decision Tree for IHDP Control group with outcome of low IQ at age 3. (w= 0.5)

To Detect the Optimal Sensitivity and Specificity

- Depends on the relative CLINICAL importance of false negatives versus false positives.
- W=1 means only false negatives matter.
- W=0 means only false positives matter.
- W=1/2 means both matter equally.
- Analytically: Use weighted kappa.
- Geometrically: Draw a line through the Ideal Point with slope determined by P and w. Push this line down until it just touches the ROC “curve”. That point is optimal.

ROC Analysis: Conclusion

- Yields Clinically Relevant Information
- Identifies complex interactions
- Identifies individuals with different characteristics but at the same risk for the clinically relevant outcome
- Identifies individuals at the least risk
- Can take differential clinical costs of false positives and false negatives into account

Conclusion

- It is not sufficient to identify risk factors or even to identify moderators and mediators etc. or a structural model.
- It is necessary to present and interpret the results so that clinicians, policy makers, consumers, other researchers can apply them.
- ROC trees are one method to accomplish this purpose.

Using the ROC Program

Using the ROC ProgramA. How to Get the ROC Program

- Go to http://mirecc.stanford.edu
- Go to “Top Information Requests”
- Go to “ROC4 is available for download HERE.”
- Double Click on “HERE”
- A pop-up window will give you the option to open or save the ROC4 zip file
- Best option is to save it to a folder you have already created e.g. ROC analysis

Using the ROC ProgramB. Opening the ROC Program

- Go to your “ROC analysis” folder
- Unzip the ROC4.zipfile (Some computers will automatically unzip when you double click on it or you may need to use an unzip program)
- Once unzipped the following 5 files will appear
- Read_Me.doc A help file which explains what to do
- ROC4.19.exe The actual ROC program
- rDemoData.bat Batch file that gets ROC program to run
- Demo.txt A demo data input file
- runDemoData.doc A demo data output file

Using the ROC ProgramC. Preparing Data for ROC Program

- First prepare your data file
- Put your data in Excel form
- Your outcome measure should always be:
- Dichotomous
- Coded as a 1 or 0
- In the far right column
- All dichotomous predictor variables coded as a 1 or 0
- All missing data coded as –9999.99
- Remove all IDs or other non-predictor information
- Save your Excel data file as Text (Tab delimited)
- Give it a name that has no spaces: This will be your data input file

Using the ROC ProgramD. Executing the ROC Program

- Open up Microsoft Word
- Within Word open the rDemoData batch file
- It will open to read as follows

echo "Program running- check folder with output file and REFRESH to confirm running “

roc4.19 Demo.txt 50> runDemoData.doc

- Where you see “Demo”, you replace with the name of your data input file
- Where you see “runDemoData”, you replace with the name of your data output file
- Then save your new batch file with a new name and put .bat at the end of the name (easiest name is one associated with the data names you have assigned).

Using the ROC ProgramD. An Example of Executing the ROC Program

- Helena has data entitled “Workshop” saved as text and now called “Workshopdata.txt”
- Within Word open the rDemoData batch file to read as follows

echo "Program running- check folder with output file and REFRESH to confirm running “

roc4.19 Demo.txt 50> runDemoData.doc

- “Demo” is replaced with “Workshopdata”
- “runDemoData” is replaced with “runWorkshopdata”
- New batch file is saved as “rWorkshopdata.bat”

echo "Program running- check folder with output file and REFRESH to confirm running “

roc4.19 Workshopdata.txt 50> runWorkshopdata.doc

- Double Left Click on new batch file and as if by magic your output file entitled “runWorkshopdata.doc” will appear

Using the ROC ProgramE. How to Read Your Output File

- Open up your data output file which will be in Word
- Select All
- Change Font to 6
- Go to Page Setup and change from Portrait to Landscape
- Expand your margins if you are still getting wrap around

Download Presentation

Connecting to Server..