Critical appraisal of clinical research evidence

1. Critical appraisal of clinical research evidence Chris Lewis – May 2008

2. How to read a “paper”

3. Objectives: To enable VTS members to have a good working knowledge of: • Processes of EBM • Conduct of a RCT • Sources of bias in a RCT • Risk assessment terminology (RR ARR NNT)

4. Evidence Based Medicine Incorporating the best available research evidence into clinical decision making

5. Processes of Evidence Based Medicine • Asking answerable questions (PICO) • Accessing the best information • Appraising the information for validity and relevance • Applying the information to patient care

6. Asking an answerable question • Population • Intervention • Comparator • Outcome(s)

7. Types of “paper” research evidence • Primary studies • Case studies • Experiments • Surveys • Clinical Trials • Secondary studies • Non-systematic reviews • Systematic reviews • Meta-analyses • Guidelines • Decision analyses • Economic analyses

8. Types of evidence

9. Topics of Primary Study and Types of Study Design Phenomena Observation / qualitative studies Aetiology Cohort studies (or Case-control studies) Diagnosis and screening Cross-sectional analytical studies Prognosis Cohort studies Intervention Randomised Controlled Trials

10. How do you read a clinical research paper?

11. How to read a (clinical research) paper • Scan abstract for a few seconds • Are the authors conclusions of interest? • Briefly assess study design • Briefly assess statistical precision of results • Formulate a brief summary • Critically appraise methods & results sections for validity • Critically appraise results section (especially the tables and figures) for relevance • Draw your own conclusions about clinical applicability

12. What conclusions have you drawn concerning clinical application of the Heart Protection Study?

13. How to read a (clinical research) paper • Scan abstract for a few seconds • Are the authors conclusions of interest? • Briefly assess study design • Briefly assess statistical precision of results • Formulate a brief summary • Critically appraise methods & results sections for validity • Critically appraise results section (especially the tables and figures) for relevance • Draw your own conclusions about clinical applicability

14. What is the essential information you want to know about any clinical research evidence? What information would you include in a brief summary of a clinical research paper?

15. Information to include in summary • Type of study • Size • Study Population • Intervention • Comparator • Duration • Outcome(s) • Main findings (with relevant statistics) • Conclusion(s)

16. Produce a brief summary for the Heart Protection Study (6 to 8 short sentences) Note which parts of the paper you have to read to produce your summary.

17. Heart Protection Study (Lancet.v360.pp7-22.6/7/2002) • Randomised placebo-controlled trial • 20536 UK adults, aged 40 to 80, with CHD, other occlusive arterial disease or diabetes. • Effect of Simvastatin 40mg vs placebo on mortality, fatal and non-fatal vascular events. • 5 years follow-up

18. All cause mortality reduced in the simvastatin group 1328/10269 (12.9%) vs 1507/10267 (14.7%); p=0.0003; RR 0.87 (0.81-0.94); NNT=55 • First vascular event rate reduced in the simvastatin group 2033/10269 (19.8%) vs 2585/10267 (25.2%); p<0.0001; RR 0.76 (0.72-0.81); NNT=18. • No significant harms identified • All people similar to the study population should be treated with Simvastatin 40mg

19. How much of the paper have we actually read to get to this summary?

20. How to read a (clinical research) paper • Scan abstract for a few seconds • Are the authors conclusions of interest? • Briefly assess study design • Briefly assess statistical precision of results • Formulate a brief summary • Critically appraise methods & results sections for validity • Critically appraise results section (especially the tables and figures) for relevance • Draw your own conclusions about clinical applicability

21. Flow Diagram for a RCT / cohort study 1 Selection and sampling 4 Outcomes 2 Allocation (with or without Randomisation) 5 Analysis 3 Follow-up

22. Validity - External Validity To whom do the results of this trial apply? Can the results be reasonably applied to a definable group of patients in a particular clinical setting in routine practice? Are the results generalisable beyond the trial setting?

23. Appraisal of External Validity • Where were the participants recruited from (primary care / referral centre)? • Do the inclusion and exclusion criteria make sense? • What proportion of the screened population was recruited?

24. Where were the participants recruited from?

25. Where were the participants recruited from? Methods: Recruitment p8 • 69 UK hospitals

26. Inclusion Criteria:

27. Inclusion Criteria: • Methods: Eligibility p8 • Men and women aged 40 to 80 + • Blood total cholesterol >= 3.5mmol/L + • Past medical history of any one or more of CHD, CVA, TIA, PVD, DM OR • Men, 65 to 80, treated for hypertension

28. Exclusion criteria:

29. Exclusion criteria: • Anyone already on a statin or Dr considered statin to be clearly indicated. • Contraindications • Chronic liver disease • ALT >67 IU/L (1.5 x ULN) • Child-bearing potential • Conditions requiring a dose reduction • Severe renal disease • Creatinine >200 mmol/L • Interactions • Treatment with ciclosporin, fibrates, niacin • Conditions similar to known unwanted effects • Inflammatory muscle disease • CK >750 IU/L (3 x ULN) • Patient unlikely to survive 5 years follow-up • Severe heart failure • Another life threatening condition • Conditions limiting compliance • Severely disabling stroke • Dementia

30. What proportion of the screened population was recruited?

31. What proportion of the screened population was recruited? Results: patient enrolment & Fig.1 p10 • 49% (31458/63603) of screened population excluded or refused We are not given a break-down of the reasons • 36% (11609/32145) of population accepted for run-in were not subsequently randomised. • 26% chose not to enter or “did not seem likely to be compliant for 5 years” • 5% considered to have clear indication for statin • 3% raised ALT, CK or Creatinine at pre-treatment screen • 2% attributed various problems to run-in treatment • 1% cholesterol <3.5mmol/L • Only 32% (20536/63603) of screened population were randomised.

32. Validity - Internal Validity The extent to which the observed difference in outcomes between the two comparison groups can be attributed to the intervention rather than other factors.

33. What are the possible causes of an “effect” in a RCT?

34. What are the possible causes of an “effect” in a RCT? • Bias • Placebo • Chance • Real effect

35. Bias • Allocation (Selection) Bias – Failure of randomisation Systematic differences in comparison groups • Performance Bias Systematic differences in interventions received by the two groups • Attrition Bias Systematic differences in withdrawals from the trial • Detection (Measurement) Bias – Failure of blinding Systematic differences in outcome assessment

36. Internal Validity - Sources of Bias in a RCT 2 Allocation Bias (Failure of Randomisation) 3 Follow-up – Performance Bias and Attrition Bias 4 Outcomes – Detection Bias (Failure of Blinding)

37. CONSORT definition: Selectionbias—a systematic error in creating intervention groups,causing them to differ with respect to prognosis. The groupsdiffer in measured or unmeasured baseline characteristics becauseof the way in which participants were selected for the studyor assigned to their study groups.

38. Confounding—a situationin which the estimated intervention effect is biased becauseof some difference between the comparison groups apart fromthe planned interventions - such as baseline characteristics,prognostic factors, or concomitant interventions. For a factorto be a confounder, it must differ between the comparison groupsand predict the outcome of interest.

39. Comparison of Cohort and RCT Cohort RCT Population highly selected Allocation by chance Outcomes defined prospectively Outcomes must be common Follow-up pre-determined and usually short-term Analysis relatively simple • Population diverse • Allocation by clinical decision • Outcomes can be defined retrospectively • Outcomes may be rare • Follow-up may be retrospective and may be long-term • Analysis complex multivariate

40. Possible types of comparisons in cohort study • Generalpopulation • Intervention v alternative intervention • Interventionv no intervention • Restricted population • Intervention valternative intervention • Intervention v no intervention

41. Do patients who receive atypical antipsychotic drugs have an increased risk of hip fracture? Effect on age distribution and sample size of restricting comparison of atypical antipsychotic with no intervention to individuals with dementia

42. Consider the difference between the three sets of figures here:

43. Selection / Allocation Bias Assessed by looking at the Table of Baseline Characteristics

44. Womens Health InitiativeJAMA v288, pp321-333, 17th July 2002 • A randomised placebo-controlled trial. • 16608 American women, aged 50-79, with intact uterus. • Effect of conjugated equine oestrogens 0.625mg od + medroxyprogesterone acetate 2.5mg od on incidence of CHD and Breast Cancer • 8.5 years follow-up planned, but stopped after 5.2 years.

45. CHD rate increased in oest+prog group 164/8506 (1.93%) vs 122/8102 (1.51%); RR 1.29 (1.02-1.63); ARI 0.42%; NNH 238 • Breast Ca rate increased in oest+prog group 166/8506 (1.95%) vs 124/8102 (1.53%); RR 1.26 (1.00 – 1.59); ARI 0.42%; NNH 238 • Treatment group also had increased rate of venous thrombo-embolism and reduced rates of fractures and colo-rectal carcinoma. • Overall long-term harms exceeded benefits 751/8506 (8.83%) vs 623/8102 (7.69%); RR 1.15 (1.03-1.28); ARI 1.14%; NNH 88 • When prescribing combined HRT in the over-50’s short-term benefits should be balanced by consideration of long-term harms.

46. Table of Baseline Characteristics (WHI) • Are all important characteristics listed? • Are any of the differences between the treatment and placebo groups statistically significant? • Are there any differences in the two groups that may bias the results? • What age range includes 95% of the Placebo group? • Assuming HRT has no effect – which group would you expect to have more heart attacks? • Does this introduce a bias? • If so, in which direction does it operate?

47. Performance Bias • Contamination: Provision of the intervention to the control group • Compliance: Poor compliance with the allocated intervention • Co-interventions Provision of unintended additional interventions to either group

48. Attrition Bias • Count (drop-out) Loss to follow-up rate should not exceed outcome event rate and should be equal in all groups.

49. Detection (Measurement) Bias Best: Double-blind Both patient and investigator unaware of treatment allocation Less important if outcome is objective (e.g. death) Critical if outcome is subjective Impossible for some comparisons eg medical vs surgical intervention

50. What are the possible causes of an “effect” in a RCT? • Bias • Placebo • Chance • Real effect