Foolish & Fatal Flaws

1. Foolish & Fatal Flaws When medical science goes bad

2. Outline • Background on statistical and methodological error • On Pigs and PCCs • Crossing the species barrier • Correct group assignment • What? You mean some of the control patients actually got transfused? • Large starting group consents, but few randomized & deceptive statistics • You mean only 6.8% of patients consenting to the trial were actually randomized? You mean there are missing statistical calculations? • ‘Temporal ambiguity’ • Outcome before exposure – What? The patient got pneumonia before the transfusion?

3. The use of statistics in medical diagnoses and biomedical research may affect whether individuals live or die, whether their health is protected or jeopardized, and whether medical science advances or gets sidetracked. [...] Because society depends on sound statistical practice, all practitioners of statistics, whatever their training and occupation, have social obligations to perform their work in a professional, competent, and ethical manner.” [Ethical Guidelines for Statistical Practise, American Statistical Association,1999].

4. When transfusion medicine gets sidetracked • Recombinant factor VIIa • Transfusion-related immunomodulation • Formula-driven resuscitation • Albumin use in critical care • RBC transfusion in patients with ischemic heart disease

5. Where can you go wrong?Strasak AM, et al. Swiss Med Wkly 2007; 137: 44-49 • Study design • Failure to a priori define outcomes • Failure to perform sample calculations • Failure to blind (or disclose who the blinding was done) • Control and treatment groups not comparable • Data analysis • Messing up on simple statistical tests – i.e. using a t-test without meeting test requirements or appropriate correction tests • When comparing multiple groups – can’t use ‘two-group’ tests • Post-hoc subgroup analysis – ‘shopping’ for statistically significant results

6. Where can you go wrong?Strasak AM, et al. Swiss Med Wkly 2007; 137: 44-49 • Documentation • ‘Where appropriate’ use of a statistical test • Failure to state the number of ‘tails’ used for a specific analysis • Presentation • Correct use of standard deviation vs. 95% confidence interval vs. inter-quartile ranges • Median vs. mean • P values – record exactly – not ‘ns’, ‘<0.05’, ‘>0.05’ – no cheating when you round your p values

7. Incongruence between test statistics and P values in medical papersBMC Med Res Methodol. 2004; 4: 13. • 11.6% and 11.1% of the statistical results published in Nature and BMJ respectively during 2001 were incongruent, mostly due to rounding, transcription, or type-setting errors • At least one such error appeared in 38% and 25% of the papers of Nature and BMJ, respectively • In 12% of the cases, the significance level might change one or more orders of magnitude • The frequencies of the last digit of statistics deviated from the uniform distribution and suggested ‘digit preference’ in rounding and reporting (a.k.a. lying)

8. Should see uniform distribution

9. More haste, less science?Nature. 1999 Aug 5;400(6744):498

10. Details 74 types of medical bias

11. What are the Contributing Factors to Misuse? • Pressures to publish, produce results, or obtain grants • Career ambitions or aspirations • Conflicts of interest and economic motives • Inadequate supervision, education, or training Gardenier JS, Resnik DB. The misuse of statistics: concepts, tools, and a research agenda. Account Res. 2002;9:65–74.

12. Bottom line: Be on the look out for medical science flaws!

13. On Pigs and PCCs Prothrombin complex concentrate vs fresh frozen plasma for reversal of dilutional coagulopathy in a porcine trauma model. Dickneite G, Pragst I. Brit J Anesthesia 2009; 102: 345-54. The study was funded and conducted by representatives of the company that makes the PCC product

14. The study design • 47 pigs, 20-30 kg, anesthetized, 70% isovolemic blood loss with replacement with RBCs/HES • The pigs were randomized to (5-7 each group): • 15 mL/kg of porcine FFP • 40 ml/kg of porcine FFP • 25 U/kg of human-derived PCC (Beriplex P/N) • 15 mL/kg saline • Porcine FFP was used instead of human FFP because infusion of human FFP into pigs can result in transfusion reactions • Following this resuscitation, the pigs underwent a controlled injury • 3 mm hole into the femur or 7 cm x 1cm incision into the spleen

15. Outcomes – measured by blinded observers • Time to hemostasis and blood loss were monitored for 120 min after injury • Skin bleeding time (SBT) in duplicate • Blood samples were collected at baseline, after the completion of hemodilution and 5 min after study treatment administration • Coagulation factors (2/7/9/10) were measured • They did not measure the non-vitamin K dependent factors

16. The effect on the PT – all 3 work

17. Factor X – only PCC works Expected results Human equiv 11 units FFP

18. Proportion with hemostasisBetter off with saline than FFP? PCC Saline FFP - low FFP – high

19. Spleen blood loss

20. Bone blood loss FFP-low FFP-high PCC Saline

21. Skin bleeding time

22. Fibrinogen level 2 ½ doses Of FFP?

23. The investigators conclusion • In view of the unmet clinical need for more efficacious haemostatic agents in such patients, clinical studies are now justified to confirm the observed favorable effects of PCC in the present preclinical model system

24. A lot of unanswered questions • The laboratory tests used to measure clotting factor levels are based on human factor-deficient plasma • Do the lab assays work for human and pig samples for all factor assays performed? • Perhaps this explains why PT corrects but not factor assays • Why would massive FFP exposure increase blood loss in these pigs? Was there something wrong with their pig FFP? Is hemodilution with FFP a bad thing? Did they fail to give enough RBCs? • Why would human PCC immediately stop bleeding in these pigs and FFP would make you bleed more than saline?

25. Comment • Pig vs. Human comparison • Infusion of human factors into pig recipients via PCC may have induced enhanced coagulation compared with what those same proteins would achieve in a human recipient? • The clinical bleeding outcomes observed between animals infused with pig FFP versus animals infused with human PCC in this experiment are difficult to compare directly • The real value of PCCs compared with FFP in human trauma cannot be answered by this experimental design • If you are a pig and you get injured, you want human PCC!

26. Caution • Make sure your trauma surgeons do not take this study at face value

27. Correct group assignmentWhen you untransfused control group might be transfused Intraoperative transfusion of 1 U to 2 U packed red blood cells is associated with increased 30-day mortality, surgical-site infection, pneumonia and sepsis in general surgery patients. Bernard AC,Davenport DL,Chang PK, et al.J Am Coll Surg 2009; 208: 931-37

28. Design • Prospective study of patients undergoing major surgical procedures at 121 hospitals • Nurses prospectively collected preoperative, intraoperative and postoperative variables for 30 days after the operation on the first 40 operations in each 8-day cycle • Database was queried for 05-06 for all general surgery procedures

29. Transfused vs. not transfused • ‘Transfusion’ = • number of PRBC units transfused intraoperatively • Transfused >4 U RBCs is the first 72 hours post-op • 2nd group = yes/no definition, not number of units

30. Time course Data variable only 72 hours OR >4 units >4 units yes/no yes/no =Transfused Intraoperative 30 days number transfused

31. Outcomes over 30 days • Composite surgical-site infection (superficial, deep, or organ/space) • Urinary tract infection • Pneumonia (without preoperative pneumonia) • Sepsis/septic shock • Composite morbidity – all of the above • 1 or more of 20 adverse events uniformly defined by the ACS-NSQIP, excluding bleeding requiring transfusion • Mortality

32. Analysis • Estimated probability for a patient to receive a transfusion (propensity) was calculated by MLR analysis of all the available patient and operative risk factors • Risks for outcomes by level of intraoperative transfusion were calculated using logistic regression, with adjustment for: • transfusion propensity, procedure group, and complexity • other ACS-NSQIP risk factors • operative duration (proxies for technique), • level of transfusion received intraoperatively • postoperative transfusion of >4 U PRBCs

33. Results • 125,223 general surgery patients at 121 hospitals were retrieved • 4,788 patients (3.8%) received intraoperative RBC or >4 in the 72 hours post • Risk variables most predictive of transfusion were inpatient procedure, procedure group, ASA class, hematocrit >38!, preoperative transfusion >4 U, emergent procedure, esophageal varices, and age

34. Results • Patients receiving a single unit of intraop RBCs had higher rates of surgical-site infections, urinary tract infection, pneumonia, sepsis/shock, composite morbidity, and 30-day mortality • After adjustment for transfusion propensity, procedure group and complexity, wound class and operative duration, and all other important risk variables, transfusion significantly (p<0.05) increased the risk of mortality (OR 1.32), composite morbidity (OR 1.23), pneumonia (OR 1.24), and sepsis/shock (OR 1.29), but not surgical-site infection

35. Their conclusions – Can’t argue with these motherhood statements • RBC transfusions should be used very selectively during surgical procedures. • Mild anemia should be tolerated. • Blood-conservation strategies and appropriate indicators for transfusion should be used. • Additional studies must determine the mechanisms by which transfusion of PRBCs and other blood components contribute to poor patient outcomes. • Although deleterious effects are evident and some mechanisms have been suggested, reversible causes and effective treatments have yet to be definitively determined.

36. The flaw • Their statistical model was based on patients requiring intraoperative transfusions and if a patient was given >4 transfusion within 72 hours post-operatively. • As such, a patient could have received up to 8 transfusions in the perioperative period and would have been considered as not having a transfusion!

37. Large population screened (few enrolled)Deceptive statistics, ‘underpowered’ Safety and efficacy of recombinant activated factor VII. A randomized placebo-controlled trial in the setting of bleeding after cardiac surgery.Gill R, Herbertson M, Vuylsteke A, et al. Circulation 2009; 120: 21-27. industry-funded

38. Phase 2 dose-escalation study Safety and possible benefits 30 sites in 13 countries Aug 2004 - Nov 2007 Cohort 1 (n=70) 1:1 randomization placebo 40 ug/kg rVIIa Design Cohort 2a (n=51) 1:2 randomization placebo 80 ug/kg rVIIa Cohort 2b (n=51) DSMB required repeat due to safety concerns placebo 80 ug/kg rVIIa Cohort 3 (planned, but not done) Steering/safety Comm advised against Placebo vs. 160ug/kg

39. Primary endpoint • The primary endpoint was the incidence of critical serious adverse events (death, myocardial infarction, stroke, and venous thromboembolic complications) • Note: Don’t get too excited – the study is stopped early before we get the answer (maybe)

40. Patient population Inclusion criteria: • Pt admitted to CVICU  30 min (excluded patients bleeding intraop) • Post-op bleeding into drains in the cardio-thoracic cavity: 200 ml/hr or 2ml/kg for 2 consecutive hours • Urgent re-op not required ‘per investigator judgment’ Examples of exclusion criteria: • History of CVA/DVT/PE • Hereditary thrombophilia • VAD, ECMO, aortic arch +/or descending thoracic aorta • 1st time CABG + none or only 1 antiplatelet medication within 5 days • Unacceptable thrombotic risk ‘as per site investigator’

41. Patient outcome Consented (n=2619) 6.8% Randomized (n=179) Randomized and not dosed (n=7) Randomized and dosed (n=172) Placebo (n=68) Death (n=4, 6%) 40 ug/kg (n=35) Death (n=4, 11%) 80 ug/kg (n=69) Death (n=6, 9%) Study terminated prior to introduction of cohort 3 “based on the data within the expanding cardiac literature in which doses of rVIIa were in the range of 60 ug/kg” ??? The time to drug administration was 2.8 hrs arriving in ICU

42. Overview of adverse events

43. Sample size calculationPoorly written Data on 1st 35 control Not provided • Sample size was chosen to have <20% risk of seeing >14 (of 35) on active versus <7 (of 35) on placebo in cohort 1 • What they saw: 5/68 (7.35%) placebo vs. 10/35 (28.57%) treatment – 21.22% absolute risk increase • <16.7% risk of 13 (of 34) on active versus 2 (of 17) on placebo or 7 on active versus 8 on placebo in cohorts 2a, 2b and 3, all assuming no differences and 21 events in cohort 1 and 15 events in cohorts 2a, 2b, and 3. • What they saw: 5/68 (7.35%) placebo vs. 11/69 (15.94%) treatment – 8.59% absolute risk increase

44. Sample size - continued • “Additionally, the sample size was chosen to give adequate power to detect a 35% reduction in the need for any allogeneic transfusions.” • The power for the efficacy evaluation is based on a comparison of (all) placebo patients with the highest dose of rFVIIa (ie, cohort 3) – never done • 11% (placebo) vs. 32% (40 ug/kg) vs. 27% (80 ug/kg) – 20% absolute risk reduction • “This simple comparison between 2 groups (86 on placebo versus 34 on rFVIIa) then has 80% power assuming 80% transfusion rate on placebo and a 35% reduction to 52%” (89% placebo, 68% on 40 ug/kg, 73% on 80 ug/kg)

45. r7a was associated with a statistically higher frequency of Serious Adverse Events p=0.0284, Fisher * SAE = Death, Cerebral Infarction, MI, PE, or other TE

46. Efficacy

47. Re-op rates

48. Avoiding RBCs after treatment

49. Major limitations • Statistically increased serious adverse events in rVIIa groups, although they failed to do a simple Fisher Exact test • Potential efficacy only seen at 80 ug/kg dose • Very select group of 6.8% of their population • Limitations: • Small sample size • Industry sponsored trial • Authors “compensated” by company • Study never completed

50. Outcome before intervention(a.k.a. Temporal ambiguity) Transfusion and pneumonia in the trauma intensive care unit: An examination of the temporal relationship.Vandromme MJ, McGwin G, Marques MB, et al. J Trauma 2009; 67: 97-101