rohling s interpretive method how can a flexible battery perform like a fixed battery n.
Skip this Video
Loading SlideShow in 5 Seconds..
Rohling’s Interpretive Method: How Can a Flexible Battery Perform Like a Fixed Battery PowerPoint Presentation
Download Presentation
Rohling’s Interpretive Method: How Can a Flexible Battery Perform Like a Fixed Battery

Loading in 2 Seconds...

play fullscreen
1 / 138

Rohling’s Interpretive Method: How Can a Flexible Battery Perform Like a Fixed Battery - PowerPoint PPT Presentation

  • Uploaded on

Rohling’s Interpretive Method: How Can a Flexible Battery Perform Like a Fixed Battery. Martin L. Rohling, Ph.D. Associate Professor Department of Psychology University of South Alabama. Clinical vs. Mechanical Diagnosis.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Rohling’s Interpretive Method: How Can a Flexible Battery Perform Like a Fixed Battery

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
    Presentation Transcript
    1. Rohling’s Interpretive Method: How Can a Flexible Battery Perform Like a Fixed Battery Martin L. Rohling, Ph.D. Associate Professor Department of Psychology University of South Alabama

    2. Clinical vs. Mechanical Diagnosis • Much research has been conducted since Meehl (1954) found clinical judgment to be less accurate than mechanical or “actuarial” judgement • e.g., Dawes, Faust, & Meehl (1989); Filskov (1981); Garb (1989); Garb (1994); Garb (1998); Grove et al. (2000);Sawyer (1966); and Wedding & Faust (1989) • Such results influential in causing NPs to turn to different versions of the HRB (Russell, 1998). • “Batteries” have been defined as the method by which one can avoid the clinical errors highlighted by Meehl an others, using “actuarial” rules for diagnosis (Russell, 1995; Russell et al., 2005). Rohling - CCPN Orlando, FL

    3. Rohling’s Interpretive Method (RIM): Development History • Conducted several meta-analysis with Dr. Laurence Binder at the Portland, OR – VA • The last of these focused on the residual cognitive effects of mild head injury. • Binder, Rohling, & Larrabee (1997) • Binder et al. grouped effect sizes (ES) into domains of neuropsychological functioning based on factor analytic studies. • e.g., Leonberger, Nicks, Larrabee, & Goldfader (1992) Rohling - CCPN Orlando, FL

    4. RIM Generated fromMeta-Analytic Procedures • Meta-analysis (MA) combines effect sizes (ES) across samples assuming that they all sample the population M for the particular effect of interest. • Common method ES calculation is a standardized mean difference score (e.g., Glass’ delta). • delta = difference between con. & exp. group’s M’s divided by con. group’s SD. • delta analogous to Z score - linear equivalent of T score used in clinical neuropsychology Rohling - CCPN Orlando, FL

    5. RIM Generated fromMeta-Analytic Procedures • Binder et al. (1997) combined ES’s generated from various tests into cognitive domains. • Why not similarly combine ES’s, or T scores, from a single patient into cognitive domains in the same way that it is accomplished in MA. • Each test score is treated as a ES that reflects the individual’s ability within a domain. • ES can be combined based on homogeneity of variance, so as to avoid combining apples and oranges. Rohling - CCPN Orlando, FL

    6. Introduction to the RIM Analysis • Flexible battery (multiple measure) use: • Is the most frequently cited model of assessment among neuropsychologists. • Only 7% of neuropsychologists use a fixed battery (Rabin et al, 2006, ACN). • Regarding the suitability, practicality, and usefulness of any fixed battery: • “We know of no batteries that fully satisfy these criteria.” (Lezak, Howieson & Loring 2004, Neuropsych. Assess., 4th ed, p 648.) Rohling - CCPN Orlando, FL

    7. Advantages of Flexible Battery • Dynamic & responsive to clinician’s needs • Covers 1 or many domains • “Flexible”, can be adapted for each patient • Can “oversample” domains • Well suited for hypothesis-driven approach Rohling - CCPN Orlando, FL

    8. Potential Problems with aFlexible Battery • Inflated error rates • Multicollinearity • Weighting decision problems • Unknown veracity/reliability of sets of tasks • Human judgment errors Rohling - CCPN Orlando, FL

    9. Human Judgment Errors(Wedding & Faust, 1989, ACN) • Hindsight bias • Confirmatory bias • Overreliance on salient data • Under-utilization of base rates • Failure to take into account co-variation Rohling - CCPN Orlando, FL

    10. Potential Benefits withRohling’s Interpretive Method (RIM) • Judgment errors can threaten reliability & validity of multiple measure test batteries. • RIM was designed to reduce these effects. • Based on meta-analytic techniques. • Uses a linear combination of scores placed on a common metric. Rohling - CCPN Orlando, FL

    11. Potential Benefits of RIM • A strategy that produces summary results analogous to those generated in a fixed-battery approach (e.g., HII, GNDS, AIR). • Takes advantage of psychometric properties of same metric data, e.g., T Scores. Rohling - CCPN Orlando, FL

    12. Today’s Presentation - Intent • Present a set of procedures that allows for a quantitatively-based comparison of an overall battery of measures. • Non-specific to battery measures themselves. • Can be used for any individual patient. • Demonstrate importance and practicality of use of established statistical indices. • (e.g., alpha, beta, effect size). Rohling - CCPN Orlando, FL

    13. Today’s Intent (cont’d) • Present a data format for any set of measures to be inspected at: • Global level (OTBM) • Domain level (DTBM) • Test measure level (ITBM) • Present a series of calculations to assist in the generation of these indices. • Present Steps in conjunction with clinical judgment from an informed position. Rohling - CCPN Orlando, FL

    14. Common RIM Domains of Functioning • Symptom Validity (SV) Tests • Emotional / Personality (EP) Measures • Meta-Cognition, Pain, or other self-ratings • Estimated Premorbid General Ability (EPGA) • Test Battery Means • Overall (OTBM), Domain (DTBM), & Instrument (ITBM) • Cognitive Domains: • VC, PO, EF, AML, VML, AW, PS • Non-Cognitive Domains: • PM, LA, SP Rohling - CCPN Orlando, FL

    15. Sample RIM: Summary Table Rohling - CCPN Orlando, FL

    16. Sample RIM: Graphic Display Rohling - CCPN Orlando, FL

    17. Brief of RIM Steps: • There are 24 steps to the RIM process • 17 calculation steps: • Advice on design of the battery • Calculation of summary statistics • Generation of graphic displays • 7 interpretative steps. • Detail a systematic procedure for use of the statistical summary table and graphic displays to: • Assess and verify summary data. • Identify strengths/limitations of current data. • Obtain a reliable diagnosis. • Develop tx plans based on sound judgments. • We briefly review each step in just a moment. Rohling - CCPN Orlando, FL

    18. Support for the RIM Process • Rational support/reasoning: Reduce clinical judgment errors. • The RIM is a Process, not a program. • Rather, the RIM is a way of formalizing thinking & interpretation of individual case data. • This is operationalizing what many flexible battery clinicians are already doing in their head. Rohling - CCPN Orlando, FL

    19. Support for the RIM Process:Specific Advantages • Psychometric properties at level with fixed, co-normed batteries, without their limitations. • Flexibility of test selection. • Flexibility of theoretical view of cognition (domain structure) Rohling - CCPN Orlando, FL

    20. Support for the RIM Process:Specific Advantages • Quantitatively support your conclusions and interpretations • Statistical evaluation • Measure of confidence in findings • Measure of limitations of findings • Ability to present data at different levels of interpretation • Greater defensibility Rohling - CCPN Orlando, FL

    21. The RIM has a Set of Procedure or Specific Steps Rohling - CCPN Orlando, FL

    22. RIM Steps 1-4: Summary Data • Design & administer battery. • Use well standardized recently normed tests. • Estimate premorbid general ability. • Use Reading (WTAR), Regression (OPIE-III), & academic records (rank, SAT, ACT). • Convert test scores to a common metric. • We recommend T scores, but z or SS OK too. • Assign scores to domains. • Factor analysis to support assignment (Tulsky et al., 2003) Rohling - CCPN Orlando, FL

    23. RIM Steps 5-8: Summary Data • Calculate domain M, sd, & n. • Calculate test battery means (TBM). • Overall TBM – All scores, large N & high power. • Domain TBM – Avoids domain over weighting. • (e.g., attention & memory). • Instrument TBM – One score per norm sample. • Calculate p for heterogeneity. • Have you put “apples & oranges” together? • Determine categories of impairment. • Recommend using of Heaton et al. (2003). Rohling - CCPN Orlando, FL

    24. RIM Steps 9-12: Summary Data • Determine % of test impaired. • Analogous to Halstead Impairment Index • # scores below cutoff / total # of of scores • Calculate ES for all summary stats. • Use Cohen’s d = (Me – Mc) / SD pooled • Calculate confidence interval for stats. • 90% CI = 1.645 x SEM • Upper limit of performance for impair. • Look for overlap between 90% CI of EPGA (lower) & Summary Stats (upper) Rohling - CCPN Orlando, FL

    25. RIM Steps 13-17: Summary Data • Conduct one-sample t tests. • Use EPGA as reference point • Conduct a between-subjects ANOVA. • Looking for strengths & weaknesses • Conduct power analyses. • Only needed for those NS differences • Sort scores for visual inspection. • Graphically display summary statistics. Rohling - CCPN Orlando, FL

    26. RIM Steps 18-20: Interpretation • Assess battery validity. • Examine the Symptom Validity scores. • Caution in accepting low power results. • Look at heterogeneity of summary stats. • Normative sample unrepresentative of patient. • Scores assigned to wrong domain. • Inconsistent performance on construct measures. • Examine influence of psychopathology. • Examine scores for heterogeneity. • Check OTBM, DTBM, & ITBM impaired. Rohling - CCPN Orlando, FL

    27. RIM Steps 21-24: Interpretation • Examine strengths/weaknesses looking at: • Confidence intervals overlap. • Results from one-sample t tests. • Results of ANOVA. • %TI show differences otherwise not evident. • Determine if pattern existed premorbidly. • Examine non-cognitive domains. • Psychomotor, Lang/Aphasia, Sensory Percept • Explore Type II errors –need more tests? • Examine sorted T-scores • Look for patterns missed by summary stats. Rohling - CCPN Orlando, FL

    28. Age: 37 Handed: Left Race: Euro-American Sex: Female Ed: 14 years Occup: Nursing Marital: Sep. 10 yrs Living: Camper in parent’s backyard Reason for Referral: TBI in head-on boat accident. Propeller hit pt in right parietal-occipital lobe (LOC = 7 days; GCS = 3). Eval. to determine capacity for medical & financial decisions, parenting skills, occupational prognosis, & disability status. Significant emotional, behavioral, occupational, and social problems pre-TBI. RIM Sample Case 1: Obvious TBI Rohling - CCPN Orlando, FL

    29. RIM Sample Case 1: Obvious TBI Rohling - CCPN Orlando, FL

    30. RIM Sample Case 1: Obvious TBI Rohling - CCPN Orlando, FL

    31. TBI Dose Response CurvesDikmen ES’s Meyers’ T Scores Rohling - CCPN Orlando, FL

    32. Combined Dikmen & Meyers Estimates: ES, T, & Difference Rohling - CCPN Orlando, FL

    33. Return to Work Study: OTBM’s for 4 Groups of TBI Survivors Rohling - CCPN Orlando, FL

    34. RIM Sample Case 1: Obvious TBI Normal Distribution of T Scores Rohling - CCPN Orlando, FL

    35. Reason for Referral: 2 yrs dangerous work habits. Eval to see if atrial fib & Type II diabetes impairs cognition. Hospitalized “TIA-like” Sx. Admitted to problems for 20 yrs, cardiac dysrhythmia & bradycardia, pacemaker, blood sugar difficult to manage, & family Hx of heart disease & diabetes. Age: 55 Handed: Right Race: Euro-American Sex: Male Ed: 13 years Occup: Mechanic Marital: Married 20 yr Living: at home w/wife RIM Sample Case 2: Subtle Diabetes Rohling - CCPN Orlando, FL

    36. RIM Sample Case 2: Subtle Diabetes Rohling - CCPN Orlando, FL

    37. RIM Sample Case 2: Subtle Diabetes Rohling - CCPN Orlando, FL

    38. RIM Sample Case 2: Subtle Diabetes Normal Distribution of T Scores Rohling - CCPN Orlando, FL

    39. RIM Critiques: Concern 1 • The method of calculating the standard deviations (SDs) for summary statistics and domain scores is incorrect. • Since many of the remaining steps of the RIM depend on the use of these SDs, this error is magnified in the subsequent steps. • SDs statistically can not exceed 9.99 and are more likely to be around 6.4 Rohling - CCPN Orlando, FL

    40. Response 1: RIM Ms 4 Datasets Rohling - CCPN Orlando, FL

    41. Inter-Individual Ms & SDs Rohling - CCPN Orlando, FL

    42. Response 1: RIM SDs 4 Datasets Rohling - CCPN Orlando, FL

    43. Intra-Individual Ms & SDs Rohling - CCPN Orlando, FL

    44. RIM Critiques: Concern 2 • More false-positives then clinical judgment. • Palmer et al. (2004) expressed concern that • We failed to distinguish “statistical” from “clinical” significance. • This failure is a critical error that precludes the prudent clinician from using the RIM. Rohling - CCPN Orlando, FL

    45. Response 2: RIM vs. Manual Detecting Differences – Overall % Rohling - CCPN Orlando, FL

    46. Response 2: RIM vs. Manual Detecting Differences – ESs Rohling - CCPN Orlando, FL

    47. Response 2: RIM vs. ManualDetecting Differences Scores Rohling - CCPN Orlando, FL

    48. RIM Critiques: Concern 3 • Clinicians who use the RIM will: • Idiosyncratically assign scores to cognitive domains. • This will result in low inter-rater reliability in analysis & diagnosis. Rohling - CCPN Orlando, FL

    49. RIM Critiques: Concern 4 • Scores on domains are unit weighted, which introduces error. • Willson & Reynolds (2004) said scores load on multiple domains. Assignment to domains & weights depend on: • Battery of tests administered. • Patients whose test scores are being examined. Rohling - CCPN Orlando, FL

    50. Response 4: Cross-Valid. Unit Wts • Conducted 4 multiple reg. on 457 pts’ WAIS-R. • Split sample in ½ - assess shrinkage. • Regressed patients’ verbal subtests onto PIQ. • Generated ideal weights for the 1st ½ of sample. • Used wts to predict PIQs in the 2nd ½ of sample. • Pre-PIQs regressed on actual PIQs 2nd ½ sample. • Also, generated weights for the 2nd ½ of sample. • Use Pre-PIQ’s regress on actual PIQs 1st ½ sample. • Repeated, except performance subtests predict VIQ • split sample ½ & generate same statistics as before. Rohling - CCPN Orlando, FL