How To Be An Evidence-Based Psychologist

1. How To Be An Evidence-Based Psychologist John Hunsley hunch@uottawa.ca

2. OVERVIEW • What Is Evidence-Based Psychological Practice (EBPP) • Evidence-base practice, empirically supported treatments, clinical practice guidelines • Examining the research evidence • Evidence-based treatments, therapeutic relationships, and assessment • Dissemination and implementation of EBPP • Implications, challenges, and opportunities

3. SOME DEFINITIONS • Evidence-Based Practice (and Evidence-Based Practice in Psychology) • Empirically Supported Treatments • Practice Guidelines • Randomized Controlled Trials (RCTs) • Efficacy Studies & Effectiveness Studies • Treatment As Usual (TAU)

4. EVIDENCE-BASED PRACTICE The use of systematically collected data, clinical expertise, and patient preferences by decision-makers (including clinicians, administrators, and policy makers) when considering service options. Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: What it is and what it isn’t. British Medical Journal, 312, 71-72. Institute of Medicine. (2001). Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academy Press. American Psychological Association Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61, 271-285.

5. EVIDENCE-BASED PRACTICE • Providing the right health care services—services that have been demonstrated to work—for each client’s needs • Services are based on empirical evidence but are individually tailored to take into account client characteristics, needs, and resources • Services may also need to be adjusted in order to fit the demands and constraints of real world clinical practice

6. IMPLEMENTATION OF EVIDENCE-BASED SCURVY PREVENTION PROGRAMS: AN EXAMPLE WE DO NOT WANT TO REPEAT • 1497 Vasco da Gama sailed around Cape of Good Hope: 100 of 160 sailors died of scurvy (due to deficiency of vitamin C) • 1601 James Lancaster RCT (4 ships England to India, 1 on which sailors received 3 tsps of lemon juice each day): Halfway, 110/278 died of scurvy on 3 ships; 0 on the “lemon” ship • 1747 James Lind RCT of 6 treatments (including citrus juice) replicates Lancaster finding • 1768-1780 James Cook (early adopter), 3 voyages, required sauerkraut in the rations Only 3 sailors died of scurvy • 1795 British Navy required citrus fruit as part of diet (limey) • 1865 British Board of Trade required citrus fruit as part of diet Berwick. (2003). Disseminating innovations in health care. Journal of the American Medical Association

7. THE EBP MODEL Best Available Evidence EBP Clinical Expertise Client/Patient Preferences

8. BEST AVAILABLE RESEARCH EVIDENCE (TREATMENT)

9. EVIDENCE-BASED PSYCHOLOGICAL PRACTICE • Influenced by multiple factors in North America • Evidence-Based Medicine • Scientist-Practitioner Model • Accountability/Quality Assurance • Empirically Supported Treatments

10. EBP IN PSYCHOLOGY: THE CONTROVERSY • Concerns: • loss of professional autonomy • takeover of professional psychology by specific interest groups • dehumanization of psychological services • inadequacy of the research base • impossibility of basing care on research evidence • Similar concerns raised in other health professions

11. APA DIVISION 12 EMPIRICALLY SUPPORTED TREATMENTS: CRITERIA FOR WELL-ESTABLISHED TREATMENTS • I. At least 2 good between group design experiments demonstrating efficacy in one or more of the following ways: • A. Superior (stat. sign.) to pill or psychological placebo or to another treatment • B. Equivalent to an already established treatment in experiments with adequate sample sizes OR • II. A large series of single case design experiments (n > 9) demonstrating efficacy. These experiments must have: • A. Used good experimental designs and • B. Compared the intervention to another treatment.

12. CRITERIA FOR WELL-ESTABLISHED TREATMENTS FURTHER CRITERIA FOR BOTH I AND II: • III. Experiments must be conducted with treatment manuals or equivalent clear descriptions of treatment • IV. Characteristics of the client samples must be clearly specified • V. Effects must have been demonstrated by at least 2 different investigators or teams

13. CAUTION: ESTs ≠ EBT • Difference between (a) research on a treatment meeting a pre-established set of criteria and (b) determining which treatments have strongest support for a specific condition • Despite this, terms are now being used almost interchangeably • can be confusing, especially as trend in professional psychology has been to use lists rather than encourage the use of practice guidelines or having individual clinicians conducting their own literature searches

14. EBTs FOR CHILDREN & ADOLESCENTS (Multiple Single Case Designs or Higher) • Autistic Disorder • Attention-Deficit/Hyperactivity Disorder • Anxiety Disorders • Chronic Pain • Conduct Problems & Oppositional Defiant Disorder • Major Depressive Disorder • Eating Disorders • Elimination Disorders • Obesity • Tic Disorders

15. EBTs FOR ADULTS • Anxiety Disorders • Specific Phobias • Social Phobia • Panic Disorder (with/without Agoraphobia) • GAD • OCD • PTSD • Major Depressive Disorder • Bipolar Disorder

16. EBTs FOR ADULTS • Eating Disorders • Anorexia Nervosa • Bulimia Nervosa • Binge-Eating Disorder • Sleep Disorders • Substance-Related Disorders • Alcohol Abuse • Cocaine Abuse • Opiate Abuse

17. EBTs FOR ADULTS • Tic Disorders • Sexual Disorders • Schizophrenia • Marital/Couple Conflict • Personality Disorders • Avoidant PD • Borderline PD • Somatoform Disorders • Pain Disorders • Body Dysmorphic Disorder • Hypochondriasis

18. SOME ADDITIONAL EBTs • Anger Management • Anxiety/fear associated with medical/dental procedures • Assertiveness Skills • Parent Training • Social Skills • Stress Management • Many also available for specific illnesses/chronic health conditions (e.g., Chronic Fatigue Syndrome, Irritable Bowel Syndrome) • Psychometrically strong assessment measures available for assessing outcomes from all EBTs

19. CLINICAL PRACTICE GUIDELINES • Common in many health professions; in professional psychology, they are almost totally absent (but coming soon from APA) • In general, they are consensus statements from experts/professional organizations/healthcare organizations that present best clinical practices (screening, assessment, consultation, treatment, referral, etc.) • Not the same as guidelines for reimbursement or other administrative purposes • Usually encourage appropriate initial assessment, without any further guidance on how to do this (more on this later)

20. CLINICAL PRACTICE GUIDELINES • National Institute for Health and Clinical Excellence (NICE) sponsored by National Health Service in England and Wales • Use of explicit evidence hierarchy • Extensive consultations undertaken with stakeholder organizations (both consumer and professional groups) • Each guideline has a limited “life,” ensuring review in near future http://www.nice.org.uk/

21. CLINICAL PRACTICE GUIDELINES: NICE RECOMMENDATIONS Some current guidelines recommending use of a psychological intervention • ADHD • Anxiety Disorders (Panic Disorder, Agoraphobia, Generalized Anxiety Disorder) • Bipolar Disorder • Chronic Fatigue Syndrome • Conduct Disorder • Depression (child, adolescent, & adult) • Eating Disorders • Obsessive-Compulsive Disorder & Body Dysmorphic Disorder • Personality Disorders (Antisocial, Borderline) • Posttraumatic Stress Disorder • Schizophrenia

22. BUT AREN’T ALL THERAPIES EQUIVALENT? • Most meta-analyses suggest that this is not the case for most disorders/conditions • Wampold et al. (1997) meta-analysis of bona fide comparative treatment studies • Mean effect size = .19 Wampold, B. E., Mondin, G. W., Moody, M., Stich, F., Benson, K., & Ahn, H. (1997). A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.” Psychological Bulletin, 122, 203-215. • But, over ¾ of comparisons within types of CBT • NNT was 9 (for comparison, interferon vs. placebo to slow progression of multiple sclerosis has NNT of 9) • For depression, marital conflict, PTSD, & bipolar disorder, however, there do appear to be multiple treatments with similar results

23. EBT COMPARED WITH TAU • Meta-analysis of 32 studies of youth interventions found majority of EBTs yielded outcomes superior to TAU (most studies focused on externalizing disorders) • Mean effect sizefor EBT versus TAU was 0.30 • This indicates that the average “EBT” youth was better off after treatment than 62% of “TAU” youth • Alternatively, this can be represented as a NNT of 6 (similar NNT for average effects of adding radiation treatment to chemotherapy for a range of cancers) Weisz, Jensen-Doss, & Hawley (2006). Psychological Bulletin.

24. EBT COMPARED WITH TAU • Data on treatment outcome for adults, in real world clinics: • Of over 6000 clients, 35% rate of improvement/recovery (Hansen, Lambert, & Forman, 2002, Clinical Psychology: Science and Practice, 9, 329-343) • Of over 6100 clients, 29% rate of improvement/recovery (Wampold & Brown, 2005, Journal of Consulting and Clinical Psychology, 73, 914-923) • Data on treatment outcome for adults, in RCTs with EBTs • Of over 2100 clients, 67% rate of improvement/recovery (Hansen, Lambert, & Forman, 2002, Clinical Psychology: Science and Practice, 9, 329-343)

25. BUT ARE EBTs CLINICALLY RELEVANT? • Are the participants in treatment studies similar enough to patients routinely seen in practice settings to warrant generalizing research results to clinical practice? • Is the research literature on psychological treatments sufficiently developed to be applicable to the broad range of conditions encountered in practice?

26. BUT ARE EBTs CLINICALLY RELEVANT? • Participants in RCTs/efficacy trials are unlike “real” clients because ……….. • They are only the “worried well” • They are only “pure” cases

27. BUT ARE EBTs CLINICALLY RELEVANT? • Patients included in RCTs for cocaine dependence had symptoms comparable to, but more severe than, those found reported by patients receiving outpatient services Carroll et al. (1999, Drug and Alcohol Dependence) • Compared patient files from a managed behavioral health care network to the inclusion and exclusion criteria used in numerous RCTs for adults with mental disorders • Over ½ of patients would have been ineligible for RCTs because symptoms were not severe enough to warrant inclusion (in most cases, due to patient diagnosis of adjustment disorder) Stirman et al. (2003, Journal of Consulting and Clinical Psychology)

28. BUT ARE EBTs CLINICALLY RELEVANT? • For youth RCTs, approx. half used no exclusion criteria related to comorbidity Weisz et al. (2004, Child and Adolescent Psychiatric Clinics of North America) • In adult disorder efficacy RCTs, only “clinical appropriate” exclusion criteria used for many years now • Can see this even almost 20 years ago in US NIMH Collaborative Depression Treatment study, with approximately ¾ having personality disorders

29. BUT ARE EBTs CLINICALLY RELEVANT? • Single RCTs, or more, available for most DSM IV Axis I and Axis II disorders, and many nondiagnosable conditions • Hawaii study of 2,200 youth receiving services • 89% had a primary diagnosis for which an EBT was available • In terms of treatment targets, 90% had 1 or more problems for which an EBT was available • On the other hand, for only 3% were there EBTs for all treatment targets Schiffman et al. (2006). Evidence-based services in a statewide public mental health system: Do the services fit the problems? Journal of Clinical Child and Adolescent Psychology, 35, 13-19.

30. EFFECTIVENESS RESEARCH • Reviewed EBT effectiveness studies published prior to April 2006 • 21 studies of adult treatment and 13 of child/adolescent treatment met criteria (including at least 2 effectiveness trials) • Compared results to benchmarks from reviews of efficacy studies (mainly meta-analysis) Hunsley & Lee (2007). Research-informed benchmarks for psychological treatments: Efficacy studies, effectiveness studies, and beyond. Professional Psychology: Research and Practice, 38, 21-33.

31. EFFECTIVENESS RESEARCH • Completion rates higher than usually reported in studies of “real world” psychotherapy • For both adult and youth disorders, the average improvement rates were similar to efficacy benchmarks • Some examples: • Adult Depression: 74% completed, 51% improved • Adult OCD: 88% completed, 64% improved • Youth Anxiety Disorders: 87% completed, 63% improved

32. EFFECTIVENESS RESEARCH PTSD Treatments Omagh Bombing Gillespie et al. (2002). Behaviour Research and Therapy, 40, 345-357. 9/11 World Trade Center Levitt et al. (2007). Behaviour Research and Therapy, 45, 1419-1433. • Vast majority completed treatment, with results very similar to efficacy RCTs • Up to 25-30 sessions; most clinicians had limited CBT background, received training over several days, weekly/monthly supervision

33. EBTs: CONCLUSIONS • Does treatment research generalize to practice? • Compelling evidence that it does, but always need to exercise caution in applying results to a particular individual • Much more research on “mild” conditions and Axis II conditions needed • How effective are evidence-based treatments (EBT) in clinical practice? • More evidence needed, but EBTs are usually as effective as in efficacy trials Hunsley, J. (2007). Addressing key challenges in evidence-based practice in psychology. Professional Psychology: Research and Practice, 39, 113-121. Hunsley, J. (2007). Training psychologists for evidence-based practice. Canadian Psychology, 47, 32-42.

34. EBPP: “FLEXIBILITY WITHIN FIDELITY” • Treatment manuals now focus attention on key elements of treatments, not just session by session list of activities/strategies • Increased recognition of importance of tailoring treatment to clients, especially aspects related to cultural diversity and presence of multiple disorders/problems components • Call for attention to principles of change and commonly used techniques/strategies, not “trademarked” therapies • Some examples: David Barlow: Unified treatment for mood and anxiety disorders Bruce Chorpita: Modular CBT for youth disorders

35. EBPP: INCLUDING CLINICAL EXPERTISE • Evidence-Based Therapy Relationships Norcross, J. C. (Ed.). (2011). Psychotherapy relationships that work: Evidence-based responsiveness (2nd ed.). New York: Oxford University Press. • There is more to EBT than RCTs • Client expectations • Therapist empathy • Therapeutic alliance • Repairing therapeutic ruptures • Culturally adapted treatments

36. EVIDENCE-BASED ASSESSMENT (EBA) Use of research and theory to guide: • The selection of constructs to be assessed for a specific assessment purpose • The methods and measures to be used in the assessment • The manner in which the assessment process unfolds (including integration and interpretation of assessment data) Hunsley, J., & Mash, E. J. (2007). Evidence-based assessment. Annual Review of Clinical Psychology, 3, 29-51.

37. WHY IS EBA NEEDED? “…blanket recommendations to use reliable and valid measures when evaluating treatments are tantamount to writing a recipe for baking hippopotamus cookies that begins with the instruction “use one hippopotamus,” without directions for securing the main ingredient.” Mash, E. J., & Hunsley, J. (2005). Evidence-based assessment of child and adolescent disorders: Issues and challenges. Journal of Clinical Child and Adolescent Psychology, 34, 362-379.

38. WHY IS EBA NEEDED? • Almost no overlap among the psychological measures found in surveys (USA, UK) to be commonly used by psychologists (and those commonly taught to clinical graduate students) and the measures necessary to • implement and monitor EBTs • adapt treatments based on the consideration of Evidence-Based Therapy Relationship (EBTR) elements • How can we use EBT and EBTR information without using appropriate assessment tools in practice?

39. A GUIDE TO ASSESSMENTS THAT WORK Hunsley and Mash (2008) “good enough” criteria for use of instruments • Must balance psychometric ideals with clinical realities • Must keep in mind issues such as age, gender, ethnicity in determining relevance of instruments and supporting data • Presented criteria used in rating norms, reliability indices (internal consistency, inter-rater reliability, test-retest reliability), validity indices (content validity, construct validity, validity generalization, treatment sensitivity), and clinical utility

40. A GUIDE TO ASSESSMENTS THAT WORK Youth: ADHD, Conduct Problems, Depression, Self-Injurious Thoughts and Behaviors, Anxiety Disorders, Pain Adults: Depression, Bipolar Disorder, Self-Injurious Thoughts and Behaviors, Anxiety Disorders, Substance Abuse Disorders, Alcohol Use Disorders, Gambling Disorders, Schizophrenia, Personality Disorders, Couple Distress, Sexual Dysfunction, Paraphilias, Eating Disorders, Sleep Disorders, Pain Hunsley, J., & Mash, E. J. (Eds.). (2008). A guide to assessments that work. New York: Oxford University Press.

41. WHY IS EBA NEEDED? • When asked to rate 3 most important problems to address in treatment, 77% of child-parent-therapist triads failed to agree on a single problem • Correlations between clinician and youth symptom and self-esteem measures <.23 Hawley, K. M., & Weisz, J. R. (2003). Child, parent, and therapist (dis)agreement on target problems in outpatient therapy: The therapist’s dilemma and its implications. Journal of Consulting and Clinical Psychology, 71, 62-70. Love et al. (2007). Meeting the challenges of evidence-based practice: Can mental health therapists evaluate their practice? Brief Treatment and Crisis Intervention, 7, 184-193. .

42. WHY IS EBA NEEDED? • In a population-basedstudy of Canadian adultswhoreceivedpsychotherapy in pastyear, 43% terminated services becausetheyfeltbetter • But, 14% terminatedbecausetheyfelttherapywas not helping, and 7% were not comfortablewith the therapist’sapproach • Cliniciansidentified <50% of treatmentsuccesses (as rated by patients) and failed to identify the 10% of patients whoterminatedbecausetheyfelttreatmentwasworseningtheirproblems Westmacott & Hunsley. (2010). Reasons for terminating psychotherapy: A general population study. Journal of Clinical Psychology, 66, 965-977 Hunsley et al. (1999). Comparingtherapist and client perspectives on reasons for psychotherapytermination. Psychotherapy, 36, 380-388. .

43. WHY IS EBA NEEDED? Limited evidence of clinical utility for commonly used instruments • MMPI-2 completed by all clients prior to treatment • Half of clinicians (randomly assigned) received test results; half did not • All clients received an appropriate EBT (based on diagnosis) • Having MMPI-2 information had no impact on • the number of sessions patients attended • whether therapy ended prematurely • overall patient improvement in functioning assessed in the end of treatment. Lima et al. (2005). The incremental validity of the MMPI-2: When does therapist access not enhance treatment outcome? Psychological Assessment, 17, 462-468.

44. A GUIDE TO ASSESSMENTS THAT WORK • Focus on specific assessment purposes directly pertinent to clinical interventions: • diagnosis (including screening issues and the importance of addressing comorbidity) • case conceptualization and treatment planning • treatment monitoring and treatment evaluation

45. DIAGNOSIS • Diagnostic information allows access to relevant research on psychopathology, epidemiology, prognosis, and treatment • Client characteristics (common comorbid conditions, likely health concerns) and social/interpersonal characteristics (common problems or limitations associated with social networks and intimate relationships, work functioning, and healthcare utilization) that are likely to merit further evaluation or consideration in treatment planning • e.g., clients meeting criteria for a substance abuse disorder are likely to abuse additional substances, and those who abuse multiple substances are least likely to benefit from treatment Rohsenow, D. (2008). Substance use disorders. In J. Hunsley & E. J. Mash (Eds.), A guide to assessments that work (pp. 319-338). New York: Oxford University Press.

46. DIAGNOSIS • Diagnosis has utility for EBTs • i.e., points in the direction of treatment options based on research evidence (e.g., NICE guidelines) • Can give directions for treatment planning with respect to comorbidity • e.g., presence of depression among people with OCD can diminish the effectiveness of exposure and response prevention Abramowitz, J. S., Franklin, M. E., Kozak, M. J., Street, G. P., & Foa, E. B. (2000). The effects of pre-treatment depression on cognitive-behavioral treatment outcome in OCD clinic patients. Behavior Therapy, 31, 517-528.

47. DIAGNOSIS • Symptom profile can be used to guide treatment selection • e.g., bipolar disorder treatments to reduce manic symptoms should address medication adherence and recognition of mood changes; treatments focusing on cognitive and interpersonal coping strategies reduce depressive symptoms Miklowitz, D. J. (2008). Adjunctive psychotherapy for bipolar disorder: State of the evidence. American Journal of Psychiatry, 165, 1408-1419. • Symptom profile can also lead to emphases in treatment strategies or addition of treatment strategies • Chorpita’s modular treatment approach Chorpita, B. F. (2006). Modular cognitive– behavioral therapy for childhood anxiety disorders. New York: Guilford Press. Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments for 322 randomized trials. Journal of Consulting and Clinical Psychology, 77, 566-579..

48. DIAGNOSIS • Study of community based services for adolescents • Disagreement between clinician-generated and research-based diagnoses were associated with a host of treatment implementation problems including: • Increased number of client “no-shows” • Cancelled treatment appointments • Treatment drop-outs • Inaccurate clinician-generated diagnoses were also associated with smaller treatment gains Jensen-Doss, A., & Weisz, J. R. (2008). Diagnostic agreement predicts treatment process and outcomes in youth mental health clinics. Journal of Consulting and Clinical Psychology, 76, 711-722.

49. TREATMENT MONITORING AND EVALUATION • What do all RCTs, used as evidence for EBTs, have in common? • Routine collection of monitoring data, typically reviewed by clinicians during the supervision of services being provided • Hypothesis: The fact that treatment is repeatedly monitored, and information provided to clinicians, is one of the most important contributors to successful treatment outcome in EBTs • Collection of assessment data as treatment unfolds allows for making any needed adjustments as required • Does this apply to routine practice?

50. TREATMENT MONITORING AND EVALUATION • The potential benefits of treatment monitoring • Outcome Questionnaire – 45 http://www.oqmeasures.com • Over 2,500 clients from a range of clinics • All completed OQ weekly, half of clinicians received feedback, namely RED,YELLOW, & GREEN dots • No feedback: 21% improved, 21% worsened • Feedback: 35% improved (i.e., 66% increase), 13% worsened (i.e., 33% decrease) Lambert et al. (2003). Is it time to track patient outcome on a routine basis? A meta-analysis. Clinical Psychology: Science and Practice, 10, 288-301.