Critical Reading of Medical Articles

1. Critical Reading of Medical Articles VOLGOGRAD - October 2011 Pr. Pierre-Emmanuel Falcoz Strasbourg University Hospital

2. After all, WHY? • Notion of « Evidence Based Medicine » • It is not possible to read all the medical literature • We must learn to retain ONLY good publications that will modify our medical behavior when faced with a given situation

3. Which sources of information? • The « reference » book • The « expert » • The medical literature • google: « pubmed » • medline : 20-30 000 articles/month

4. Quality criteria of an article • The structure of the article • The type of study • The methodological quality (internal validity) • The practical feasibility (extrapolation)

5. The structure of the article • The title: • Interesting, useful and linked to our question • The abstract: • informative • IMRaD structure • One introductory sentence: the objective • 2 or 3 sentences regarding material and methods: appropriate to the question raised • 3 or 4 sentences illustrating the results: accurate and relevant • One concluding sentence

6. The structure of the article • Introduction • The justification • The objective • Material and methods • The patient(s) (inclusion and exclusion criteria) • Description of the method • End point and outcome measure • The plan of the analysis

7. The structure of the article • Results • Description of the included patient(s) • They should correspond to patients announced in the material and methods paragraph • Data should be clear, precise and pertinent • Calculation should be easily tested • Discussion: comparison of the results with those of other articles • Written paragraphs should be well differentiated  do not mix sub-headings!!!

8. The different types of studies • Therapeutic studies • Diagnostic studies • Prognostic studies • Epidemiologic studies • Economic studies

9. The randomized controlled trials • « The gold standard » enables one to answer the question asked • Only way to obtain reliable proof of the effectiveness of a given treatment • The other types of studies: • Non-randomized prospective trials, retrospective studies, e.g. … • Lower level of scientific proof

10. A short story... • After myocardial infarction, the existence of ventricular extra systoles (VES) increased the risk of sudden death • Class 1 anti-arrhythmics erase VES • From this data, such treatments have been given to prevent sudden death… • Without any trials of clinical criteria…

11. CAST study (1991) Death / n mortality Anti-arrhythmic group 39 / 432 9% Control group 18 / 423 4% RR=2.13, p<0.05 • 80 000 Deaths induce by the treatment in the USA • More than Vietnam War (Moore)

12. Anti-arrhythmics in post myocardial infarction period Proven Ventricular extrasystoles Sudden death Proven Speculative Not confirmed in real settings by actual findings Flecaine

13. Comparative trial End point5-y mortality No treatment 10 % Difference = effect of the treatment Treatment studied 12 %

14. The quantification of scientific proof (ACCP) • Level 1 : • Randomized trials with definitive (unquestionable) results • Meta-analyses • Level 2 : • Randomized trials (low power) • Non randomized controlled trials (well designed) • Level 3 : • Non controlled prospective trials (well designed) • Level 4 : • Controlled trials with bias • Comparative trials (historical series) • Level 5 : • Retrospectives studies or case-series

15. The bias • Definition: there is a bias when the difference observed between two groups at the conclusion of the trial is due to a factor other than the studied treatment itself Population Randomization Selection bias Control group Experimental group Execution bias Treated Untreated Exclusion bias Follow-up Follow-up Detection bias Results Results

16. The different biases – a summary • Selection bias • Difference in the base prognosis of the patient(s) • Execution bias • difference in follow-up and care given to the patient(s) • Exclusion bias • difference with regard to the “departure” from the studies • Evaluation bias (measure) • difference in evaluation of the end-point

17. Grp T Treatment Grp C Comparable groups • Similar groups • Same type(s) of patient(s) • Same stage of the disease, etc… • That are different ONLY due to the applied treatment • If a difference exists, it is ONLY due to the treatment

18. Composition of groups • The distribution should not depend on: • the patient • the disease • the physician • the medicine • By random drawing of lots: • random allocation • assures that , on average, the 2 groups are strictly comparable

19. Poor control groups • Historical controls • For example, patients treated 5 years ago • Those patients are not comparable to those currently treated • Geographical control • patients of another department

20. Statistical reality of the results • An observed difference between 2 groups could be either real or due to chance • Statistical tests allow the statistical reality of the difference to be assessed • The alpha risk is the risk of wrongly drawing a conclusion regarding the effectiveness of a treatment, when in reality this difference is due to chance  this risk is set up to 5% • The danger would be to to increase the alpha risk and thus wrongly conclude that a difference exists between the 2 groups

21. It is unlikely that the observed difference would be due to chance Significant difference The probability that the observed difference would be due to chance is high Non significant difference Statistical test: interpretation P<5% Test Observed difference P>5%

22. A priori definition of a primary endpoint 1 single statistical test Risk of wrongly drawing a conclusion regarding the effectiveness of the treatment = 5% No definition of the primary endpoint 6 statistical tests Risk of wrongly drawing a conclusion regarding the effectiveness of the treatment = 30% Endpoint • Death (all causes) • Cardiovascular death • Sudden death • Myocardial infarction • Cerebrovascular accident • Surgery • Primary endpoint • Death from all causes • Secondary endpoint • Cardiovascular death • Sudden death • Myocardial infarction • Cerebrovascular accident • Surgery

23. Test 1 Test 2 Test 3 Test 4 Risk to wrongly conclude to a difference = 5% Risk to wrongly conclude to a difference = 5% Risk to wrongly conclude to a difference = 5% Risk to wrongly conclude to a difference = 5% Multiple comparisons At dice game, the probablity to obtain one SIX is higher with 3 dices than with only one n overall risk 2 0.10 3 0.13 5 0.23 10 0.40 Overall, the risk of wrongly drawing a conclusion regarding a difference when utilizing these four comparisons is much greater than 5% Inflation of the alpha risk

24. The methodological value of the result • The trial respects the experimental design • A hypothesis formulated A PRIORI • The entire experimental plan is devised A PRIORI • the primary endpoint • the calculation of the number of required subjects • an analysis planned • The results should correspond to the formulated hypothesis • One cannot draw a conclusion in terms of causality if a result is not derived from this approach • does not concretely demonstrate anything • at best it suggests an effect

25. The clinical relevance • It allows to ascertain that: • The benefit of the treatment is sufficiently important • The clinical criteria is relevant • The ratio benefit/risk is acceptable • The treatment is applicable in daily practice

26. The methodological relevance • The main objective is clearly formulated • The therapeutic methods are describe with precision: • the comparison treatment is adapted • the evaluated treatment is applicable in daily practice • The primary endpoint: • Clinical, relevant, validated • The patient(s) included in the trial is/are representative: • the studied population resembles patients seen in daily practice

27. The importance of the therapeutic effect • The fact that the trial is significant (p<5%) is not good enough!!!! • If the therapeutic effect is minimal, it might be of little or no interest to our daily practice…

28. The accuracy of the therapeutic effect • The confidence interval • Indicates the accuracy of the estimation of the size of the therapeutic effect • A 95% confidence interval assess the range of values within which we are 95% certain to find the true sought value

29. 95% confidence interval Trial A Trial B Trial C 0.4 0.6 0.8 1 1.2 1.4 Relative Risk

30. The quantification of the therapeutic effect • It allows me to know how much my chances of improvement could be increased if I use this therapeutic ! • The relative risk (RR) or the odds ratio (OR) • The relative risk reduction (RRR) • The absolute risk reduction (ARR) • The number needed to treat (NNT)

31. The definitions • Relative risk: risk of treated patients in relation to the risk of the control • Relative risk reduction: is the amplitude of the risk reduction • Absolute risk reduction: is the absolute value of the risk reduction • The number needed to treat: is the number of patients we must treat to avoid, on average, one event

32. Number needed to treat (NNT) Interpretation: At 2 years: we must treat 26 patients by chemotherapy to avoid 1 death !!! N. Engl J Med 2004

33. The benefit / risk ratio • The decision should weigh the beneficial effects against the adverse events • Are the adverse events acceptable? • Do the adverse events offset the totality of the benefit? • The benefit / risk ratio is unfavorable in the following situations: • the seriousness of the adverse events represents an unacceptable risk with regard to the seriousness of the disease • the frequency of the adverse events is high • the new therapeutics are less well tolerated than the available therapeutics

34. Practical guide • Evaluation of the internal validity (methodological quality) • the research of bias • the statistical reality of the result • the methodological value of the result • The clinical relevance and the use in daily practice • the objective of the trial • the relevance of the main end point • the representativity of the included patient(s) • the feasibleness of the therapeutic • the size of the therapeutic effect • the benefit / risk ratio

35. The end… Thank you for your attention!