Evidence Based Medicine Capitol Conference 2007

1. Evidence Based MedicineCapitol Conference 2007 Jeff Weinfeld, MD Assistant Professor Georgetown University Department of Family Medicine 4/27/2012

2. Acknowledgement Janelle Guirguis-Blake, MD

3. Learning Objectives Describe study designs and their advantages and disadvantages Describe applications for chi-square test, sample and student T tests, interpret p values Define, calculate, and interpret sensitivity, specificity, and measures of risk including RRR, ARR, NNT, OR

4. Evidence-Based Medicine �The conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence-based medicine requires integration of individual clinical expertise and patient preferences with the best available external clinical evidence from systematic research.� -- Gordon Guyatt, MD

5. EBM Process Ask clinical question Search literature Critically evaluate the medical literature User�s guides to the Medical Literature published in JAMA �> Book Apply to your patient

6. The PICO Format for Foreground Questions For the ideal study I would like to find P: The Population (kids, women, CHF pts) I: The Intervention (or test) C: Comparison (control group) O: Outcome

7. Information Mastery Goal is to answer clinicians� questions quickly at point of care Modifies/Incorporates EBM into more usable form Helps identify what information is important JFP 38(5) 1994: 505-13. Slawson DC, Shaughnessy AF, Bennett JH. Becoming a medical information master: feeling good about not knowing everything. JFP 38(5) 1994: 505-13.Slawson DC, Shaughnessy AF, Bennett JH. Becoming a medical information master: feeling good about not knowing everything. JFP 38(5) 1994: 505-13.

8. Usefulness Equation Usefulness = Relevance x Validity Work For medical information to be useful Relevant Valid Reasonable amount of work Next lets talk about some basic principals that allow us to use EBM IN PRACTICENext lets talk about some basic principals that allow us to use EBM IN PRACTICE

9. Relevance: POEs and DOEs DOE = Disease oriented evidence �Ologies� (path-, etiol-, pathophys-) POE = Patient oriented evidence Morbidity, mortality or quality of life Something a patient would care about without explanation Highest quality evidence Next important part of usefulness eqn is relevance. Proportional to utility A WAY TO DIVIDE EVIDENCE = ALL EVIDENCE IS NOT CREATED EQUAL!!!!!Next important part of usefulness eqn is relevance. Proportional to utility A WAY TO DIVIDE EVIDENCE = ALL EVIDENCE IS NOT CREATED EQUAL!!!!!

10. Study Designs Randomized Controlled Trial N-of-1 Systematic Review Meta-Analysis Cohort Cross Sectional Case Control Case Series/Report

11. Hierarchy of Studies: The Best Randomized trials Meta-analysis of randomized controlled trials Randomized Controlled Trial (RCT) N-of-1

12. Hierarchy of Studies: The Rest Non-Randomized Trials Case-Control Cohort Prospective Retrospective Population-based studies Case series/case report

13. Randomized Control Trial Advantages Unbiased distribution of confounders Blinding more likely Randomization facilitates statistical analysis Disadvantages Expensive - time and money Volunteer bias Can be ethically problematic

14. N-of-1 Trials Advantages Best study for individual patient Blinding Disadvantages Cost, time Need pharmacist for placebo Not reimbursed

15. Systematic Review A review which answers a defined question by summarizing multiple studies on a single topic with. Comprehensive (published) literature search strategy Defined inclusion and exclusion criteria Evaluation of evidence quality Summarizes evidence From guyatt p 158 fig 1e-1Summarizes evidence From guyatt p 158 fig 1e-1

16. Meta-analysis Systematic Review PLUS Pooled data of similar (homogeneious studies) Combine results (studies or patients)

17. Disadvantages of Systematic Review/Meta analysis Advantages Summarize literature Rigorous One RCT could be wrong Increase power by combining multiple studies Disadvantages Requires many RCTs Limited by existing data Meta-analysis not always possible (lack of homogeneity) Does not eliminate bias or underlying quality issues

18. Cohort Study Advantages Establishes Incidence Ethically safe Subjects matched Can establish causation Eligibility criteria and outcome assessments can be standardized Easier and cheaper than RCT Disadvantages Controls may be difficult to identify Exposure may be linked to a hidden confounder Blinding difficult Not randomized Rare disease -- large sample or long follow-up Cell phones and brain tumors � 100 people � some will use/have used cells and some never use cell phones follow both individual with and without exposure for 20 years to see what the odds of getting brain tumors is in each group. Can be prospective or retrospective. Tie to confounder of those with cell phones more techn. Advanced so may also have a microwave. Or community nuclear powerplant was giving cell phones to all of its employees...Cell phones and brain tumors � 100 people � some will use/have used cells and some never use cell phones follow both individual with and without exposure for 20 years to see what the odds of getting brain tumors is in each group. Can be prospective or retrospective. Tie to confounder of those with cell phones more techn. Advanced so may also have a microwave. Or community nuclear powerplant was giving cell phones to all of its employees...

19. Cross-Sectional Survey Advantages Establishes prevalence Cheap and simple Ethically safe Disadvantages Can�t establish incidence Association, not causality Recall bias susceptibility Confounders may be unequally distributed Group sizes may be unequal Specific population Take this room and I do an upper endoscopy on each of you to determine how many of you have erosive gastritis or PUD. Tie this to assoc not causality from studying for the boards. Neyman bias= case group is not representative of population you intended to study This can make a critical difference for a case control design where you have risk factors that are associated not with the disease itself, but with mortality. Any risk factor that makes a person die quickly is going to be underrepresented among prevalent cases and could lead to a spurious finding. This is sometimes called Neyman's bias. Take this room and I do an upper endoscopy on each of you to determine how many of you have erosive gastritis or PUD. Tie this to assoc not causality from studying for the boards. Neyman bias= case group is not representative of population you intended to study This can make a critical difference for a case control design where you have risk factors that are associated not with the disease itself, but with mortality. Any risk factor that makes a person die quickly is going to be underrepresented among prevalent cases and could lead to a spurious finding. This is sometimes called Neyman's bias.

20. Case-Control Studies Advantages: Quick and cheap Feasible for very rare disorders or long lag time Fewer subjects needed Disadvantages: Reliance on recall or records to determine exposure status Confounders Selection control groups is difficult Back to microwaves and brain tumors�Since brain tumors are rare, it would be more feasible to do a case control (than a cohort) and to start with 100 pts with brain tumors and 100 pts without brain tumors and ask how many of each group had ever use cell phoneBack to microwaves and brain tumors�Since brain tumors are rare, it would be more feasible to do a case control (than a cohort) and to start with 100 pts with brain tumors and 100 pts without brain tumors and ask how many of each group had ever use cell phone

21. Case report/Case series Advantages: Relatively inexpensive Short time course Appears early when learning about new disease Disadvantages: Anecdotal No statistical testing can be applied This is how the brain cancer cell phone association was first put out there as a question. Physicians/patients noticed individual cases. Then the study was done.This is how the brain cancer cell phone association was first put out there as a question. Physicians/patients noticed individual cases. Then the study was done.

22. Statistical methods Sensitivity Specificity Positive Predictive Value Negative Predictive Value CHARACTERISTICS of screening or diagnostic testsCHARACTERISTICS of screening or diagnostic tests

23. Ugggh!Sensitivity and Specificity Sensitivity is the proportion of people with disease who have a positive test = TP/TP+FN = a/a+c Specificity is the proportion of people free of a disease who have a negative test = TN/TN+FP = d/d+b

24. Ugggh! PPV, NPV PPV = probability actually having the disease in a positive test = TP/TP+FP = a/a+b Increasing prevalence increases PPV NPV = probability not having the disease in a negative test = TN/TN+FN = d/d+c

25. Sensitivity and Specificity Sensitivity = TP/TP+FN = a/a+c Specificity = TN/TN+FP = d/d+b

26. Sn/Sp Question Secrets

27. Sn/Sp Question Secrets Prevalence = TP+FN / NPrevalence = TP+FN / N

29. Sensitivity and Specificity A colleague has told you that there is a new blood test for diagnosing thyroid cancer. Before you decide if you want to use it in your clinic, you look at one of the published studies. In the study,there were a total of 2300 enrolled patients. All of these patients had the new blood test and a subsequent thyroid biopsy. There were 463 people who had positive blood test results, but only 215 of these people who tested positive actually had thyroid cancer on biopsy. There were 1837 people who had negative test results, and 1822 of these people actually had negative biopsy results. What are the sensitivity and specificity of this new blood test? What is the positive predictive value and negative predictive value of this test?A colleague has told you that there is a new blood test for diagnosing thyroid cancer. Before you decide if you want to use it in your clinic, you look at one of the published studies. In the study,there were a total of 2300 enrolled patients. All of these patients had the new blood test and a subsequent thyroid biopsy. There were 463 people who had positive blood test results, but only 215 of these people who tested positive actually had thyroid cancer on biopsy. There were 1837 people who had negative test results, and 1822 of these people actually had negative biopsy results. What are the sensitivity and specificity of this new blood test? What is the positive predictive value and negative predictive value of this test?

30. Sensitivity and Specificity Denominator of sensitivity is ALL PEOPLE WITH THE DISEASE. Denominator of specificity is ALL PEOPLE WITHOUT THE DISEASE. Denominator of PPV is ALL POSITIVE TESTS. Denominator of NPV is ALL NEGATIVE TESTS. NOTE: Sensitivity and specificity are CHARACTERISTICS OF THE TEST. PPV AND NPV are CHARACTERISTICS OF THE TEST and REFLECT PREVALENCE OF DISEASE. Sens and spec look pretty decent but b/c thyroid cancer is SO RARE, the PPV is pretty low-- if a pt has a positive test result, there�s a 50/50 chance that the pt actually has thyroid cancer.Denominator of sensitivity is ALL PEOPLE WITH THE DISEASE. Denominator of specificity is ALL PEOPLE WITHOUT THE DISEASE. Denominator of PPV is ALL POSITIVE TESTS. Denominator of NPV is ALL NEGATIVE TESTS. NOTE: Sensitivity and specificity are CHARACTERISTICS OF THE TEST. PPV AND NPV are CHARACTERISTICS OF THE TEST and REFLECT PREVALENCE OF DISEASE. Sens and spec look pretty decent but b/c thyroid cancer is SO RARE, the PPV is pretty low-- if a pt has a positive test result, there�s a 50/50 chance that the pt actually has thyroid cancer.

31. How to use them SnOUT Very sensitive test can rule condition OUT if NEGATIVE Eg. Ottowa Knee and Ankle rules SpIN Very specific test can rule condition IN if POSITIVE Eg. Rapid strep test 80-90% sensitive, 95-99% specific Snout = if all rules neg, no fracturesSnout = if all rules neg, no fractures

32. The DeTesto Company markets a new quick and painless test for Syndrome Z. Syndrome Z has a 2-fold higher prevalence in women compared to men. The �gold standard� for the diagnosis of Syndrome Z is a biopsy, which is painful and costly. You work in the gynecology clinic for a year, seeing only female patients. During that time, you use the test on 150 patients with the following results, presented in table form. Syndrome Z The specificity of this test in this population is: 25% 50% 67% 85%

33. Risk measurements Absolute Risk Reduction Risk Ratio Relative Risk Reduction Odds Ratio Number Needed to Treat Now we will look at the concept of RISK. WAYS TO COMMUNICATE BENEFITS OR RISKS OF TREATMENTS.Now we will look at the concept of RISK. WAYS TO COMMUNICATE BENEFITS OR RISKS OF TREATMENTS.

34. Absolute Risk Reduction (ARR) Difference in the event rate between control group (CER) and treated group (EER) ARR = CER � EER The ABSOLUTE arithmetic difference in rates of bad outcomes b/w the untreated and treated groups Start withStart with

35. ARR You found that, in the untreated group, 50/65 (77%) had an MI. In the treated group, only 5/65 (8%) had an MI. You found that, in the untreated group, 50/65 (77%) had an MI. In the treated group, only 5/65 (8%) had an MI.

36. Absolute risk reduction Start with one hundred and thirty pts with CAD- you treat 65 if those pts with ASA for one year and then you look for how many of those pts had recurrent MI. If untreated, there was a seventy-seven percent chance of MI. If treated, there was a eight percent chance of MI. The ARR therefore is sixty nine percent.Start with one hundred and thirty pts with CAD- you treat 65 if those pts with ASA for one year and then you look for how many of those pts had recurrent MI. If untreated, there was a seventy-seven percent chance of MI. If treated, there was a eight percent chance of MI. The ARR therefore is sixty nine percent.

37. Absolute risk reduction Start with one hundred and thirty pts with CAD- you treat 65 if those pts with ASA for one year and then you look for how many of those pts had recurrent MI. If untreated, there was a seventy-seven percent chance of MI. If treated, there was a eight percent chance of MI. The ARR therefore is sixty nine percent.Start with one hundred and thirty pts with CAD- you treat 65 if those pts with ASA for one year and then you look for how many of those pts had recurrent MI. If untreated, there was a seventy-seven percent chance of MI. If treated, there was a eight percent chance of MI. The ARR therefore is sixty nine percent.

38. Number Needed to Treat (NNT) 100/ARR% or 1/Absolute Risk Reduction Always associated with time, CIs Number to treat to get one result Can also NNS, NNH Lower better, but always relative http://www.jr2.ox.ac.uk/bandolier/band50/b50-8.html NNT = 1 / EER - CER EXPRESS BOTH AS FRACTIONS FIRSTNNT = 1 / EER - CER EXPRESS BOTH AS FRACTIONS FIRST

39. Number needed to treat Same exact scenario of CAD and treatment with ASA. Chance of MI in untreated group remains seventy seven percent and the chance of MI in the treated group is eight percent. So ARR is seventy seven minus eight which = sixty nine percent. One divided by sixty nine is one point four. You need to treat less then one and a half patients to prevent one MI.Same exact scenario of CAD and treatment with ASA. Chance of MI in untreated group remains seventy seven percent and the chance of MI in the treated group is eight percent. So ARR is seventy seven minus eight which = sixty nine percent. One divided by sixty nine is one point four. You need to treat less then one and a half patients to prevent one MI.

40. You are considering how useful a new treatment might be in preventing stroke. A well designed study is reported with 200 patients in the treated group and 200 patients in the untreated group. The study finds a 5-year risk of stroke of 3% in the treated group versus 5% in the untreated group. Assuming this study is valid and applicable to your patient population, how many patients would you have to treat for 5 years to prevent one stroke (number needed to treat, or NNT)? A) 400 B) 200 C) 100 D) 50 E) 25 D = 50 5% - 3% = 2% So 100% / 2% = NNT of 50 D = 50 5% - 3% = 2% So 100% / 2% = NNT of 50

41. Risk Ratio/Relative Risk RR is ratio of risk in the treated group (EER) to the risk in the control group (CER): RR=EER/CER Relative Risk Reduction (RRR) is the percent reduction in events in the treated group event rate (EER) compared to the control group event rate (CER): RRR = (CER - EER) / CER * 100 PROPORTIONAL reduction in rates of bad outcomes b/w the untreated and treated groups used in randomized trials and cohort studies. Expressed in probability. Prospective studyExpressed in probability. Prospective study

42. Relative risk reduction Same exact scenario of CAD and treatment with ASA. Chance of MI in untreated group remains seventy seven percent and the chance of MI in the treated group is eight percent. RRR is seventy seven percent minus eight percent divided by seventy seven percent. There is a ninety percent relative risk reduction with the use of ASA.Same exact scenario of CAD and treatment with ASA. Chance of MI in untreated group remains seventy seven percent and the chance of MI in the treated group is eight percent. RRR is seventy seven percent minus eight percent divided by seventy seven percent. There is a ninety percent relative risk reduction with the use of ASA.

43. Odds Ratio Odds are a ratio of events to non-events. If the event rate for a disease is 0.1 (10%), its nonevent rate is 0.9 and therefore its odds are 1:9, or 0.111. Odds Ratio is the ratio of an experimental patient suffering an adverse event relative to a control patient. Used in case control studies

44. OR You have 100 people with lung cancer and find 100 more people without lung cancer and then look back to see who smoked. Of the 100 people with lung cancer, 90 of those people smoked and 10 did not smoke. Of the 100 people without lung cancer, 40 of those people smoked and 60 of those people did not smoke. Calculate the odds ratio for getting lung cancer if you are a smoker. What type of study - (case control design) Exposure is smoking and disease is lung cancer. What type of study - (case control design) Exposure is smoking and disease is lung cancer.

45. Odds ratio Exposure is smoking and disease is lung cancer; start with 100 people with lung cancer and 100 people without lung cancer and then look back to see who smoked. Of the one hundred people w/lung ca, ninety of those people smoked and ten did not smoke. Of the one hundred people without lung cancer, forty of those people smoked and sixty of those people did not smoke. Ratio of risk of getting lung ca if you smoke over the risk of getting lung cancer if you don�t smoke. There is a thirteen to one odds.Exposure is smoking and disease is lung cancer; start with 100 people with lung cancer and 100 people without lung cancer and then look back to see who smoked. Of the one hundred people w/lung ca, ninety of those people smoked and ten did not smoke. Of the one hundred people without lung cancer, forty of those people smoked and sixty of those people did not smoke. Ratio of risk of getting lung ca if you smoke over the risk of getting lung cancer if you don�t smoke. There is a thirteen to one odds.

46. Interpretation Relative risk (RR) = Ie/Io; Cohort Odds Ratios (OR) = Oe/Oo ; Case-control Confidence interval Overlaps 1, not significant > 1 = exposure increases risk < 1 = exposure decreases risk

47. Statistical analysis methods Central tendency P value Chi-Square Test T test

48. Central tendencies Median: Value that exactly one-half of the values are less than and one-half of the values are more than when the values are sorted in numerical order. Mean: Average value of a data set and mathematically is the sum of all values divided by the number of values. Mode: Most common data value, which is the highest peak of a frequency distribution. Median example grades on a test- if median was score of eighty percent, then half the students scored below eighty and half the students scored over eighty percent. Mean example. If the mean score was eighty percent then that was the average of all grades. Mode example. If the mode was eighty percent then that means that of all the possible scores, students most often scored an eighty percent. You can apply this to clinical lab tests.Median example grades on a test- if median was score of eighty percent, then half the students scored below eighty and half the students scored over eighty percent. Mean example. If the mean score was eighty percent then that was the average of all grades. Mode example. If the mode was eighty percent then that means that of all the possible scores, students most often scored an eighty percent. You can apply this to clinical lab tests.

49. P value Probability that an outcome as large as or larger than the outcome was due entirely to chance variability of individuals or measurements alone. If Null hypothesis states that the means of two groups are equal, then the P value states that there is an X% chance that this difference seen in the study is due to chance. P<.05 statistically significant - 5% chance that this difference was found by chance alone. P-value: The p-value is the probability that an outcome as large as or larger than that observed would occur in a properly designed, executed, and analyzed analytical study if in reality there was no difference between the groups, i.e., that the outcome was due entirely to chance variability of individuals or measurements alone. A p-value isn�t the probability that a given result is wrong or right, the probability that the result occurred by chance, or a measure of the clinical significance of the results. A very small p-value cannot compensate for the presence of a large amount of systematic error (bias). If the opportunity for bias is large, the p-value is likely invalid and irrelevant. The p-value the probability of observing the results from your sample of data or a sample with results more extreme, assuming the null hypothesis is true. The smaller the p-value, the greater the inconsistency.P-value: The p-value is the probability that an outcome as large as or larger than that observed would occur in a properly designed, executed, and analyzed analytical study if in reality there was no difference between the groups, i.e., that the outcome was due entirely to chance variability of individuals or measurements alone. A p-value isn�t the probability that a given result is wrong or right, the probability that the result occurred by chance, or a measure of the clinical significance of the results. A very small p-value cannot compensate for the presence of a large amount of systematic error (bias). If the opportunity for bias is large, the p-value is likely invalid and irrelevant. The p-value the probability of observing the results from your sample of data or a sample with results more extreme, assuming the null hypothesis is true. The smaller the p-value, the greater the inconsistency.

50. Significance Doesn�t indicate size of effect Doesn�t indicate clinical significance of effect Doesn�t rule out biased result

51. Chi-Square Test Tests the null hypothesis, which states that there is no significant difference between the expected and observed result Nominal data (categories) for proportions Compares observed data with data we would expect to obtain according to a specific hypothesis For example, if, according to Mendel's laws, you expected 10 of 20 offspring from a cross to be male and the actual observed number was 8 males, then you might want to know about the "goodness to fit" between the observed and expected. Were the deviations (differences between observed and expected) the result of chance, or were they due to other factors Or you toss a coin 20 times and you get �heads� 13 times (when you might expect heads 10 times) . Was this deviation from the expected a result of chance alone---or is there some other factor at play?? For example, if, according to Mendel's laws, you expected 10 of 20 offspring from a cross to be male and the actual observed number was 8 males, then you might want to know about the "goodness to fit" between the observed and expected. Were the deviations (differences between observed and expected) the result of chance, or were they due to other factors Or you toss a coin 20 times and you get �heads� 13 times (when you might expect heads 10 times) . Was this deviation from the expected a result of chance alone---or is there some other factor at play??

52. T-tests Sample T test: assesses whether the means of two groups are statistically different from each other (no assumptions of distribution) Student T test: same but assumes a Gaussian distribution Paired Student T test: assesses whether means of same group tested in different points in time are different from one another Example: I am completing a medical malpractice study where I look at the mean malpractice payments in states with and without damage caps. The two groups are: states with damage caps and states without damage caps. The mean payments in the states with caps is one hundred thousand dollars and then mean payments in the states without caps is one hundred and twenty thousand dollars. So I performed a sample T test to determine if the means of these two groups is statistically different. What is the value of T???? To report the variety of possible outcomes: from means not "significantly" different to means in fact "significantly" different, the probability that the difference is do to chance is reported. Reject the null hypothesis if P is "small� Student's t test for independent samples: is used to determine whether two samples were drawn from populations with different means. (Assumes Gaussian distribution) Paired student T test: Very often the two samples to be compared are not randomly selected: the second sample is the same as the first after some treatment has been applied. Children�s mean reading speed before and after a head start program. Example: I am completing a medical malpractice study where I look at the mean malpractice payments in states with and without damage caps. The two groups are: states with damage caps and states without damage caps. The mean payments in the states with caps is one hundred thousand dollars and then mean payments in the states without caps is one hundred and twenty thousand dollars. So I performed a sample T test to determine if the means of these two groups is statistically different. What is the value of T???? To report the variety of possible outcomes: from means not "significantly" different to means in fact "significantly" different, the probability that the difference is do to chance is reported. Reject the null hypothesis if P is "small� Student's t test for independent samples: is used to determine whether two samples were drawn from populations with different means. (Assumes Gaussian distribution) Paired student T test: Very often the two samples to be compared are not randomly selected: the second sample is the same as the first after some treatment has been applied. Children�s mean reading speed before and after a head start program.

53. Become An Optimal Clinician Training in EBM Computers available at the point of care Use evidence-based secondary sources Inquisitive Open to new approaches, willing to question standard approaches Persistent in pursuing answers Ebell, M. Information mastery. FP essentials, ed 318, AAFP home study, Nov. 2005 To conclude� EBM can help you to become an optimal physician Here are some of the things you need to do. Training .. You already have it! But we�ll look forward to continuing in second year Keep an open mind! Question authority (but in a nice way) And last but not least, lets not forget that ultimately, it is about the patient.To conclude� EBM can help you to become an optimal physician Here are some of the things you need to do. Training .. You already have it! But we�ll look forward to continuing in second year Keep an open mind! Question authority (but in a nice way) And last but not least, lets not forget that ultimately, it is about the patient.

54. References Users' Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. Guyatt G. Rennie D Eds. 2002. Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Evidence-Based Medicine: How to Practice and Teach EBM. London: Churchill Livingstone; 1997:12-16. http://www.cebm.net/glossary.asp. Centre for Evidence-Based Medicine, University Department of Psychiatry, Warneford Hospital, Headington, Oxford.

55. Thanks!