Clinical research how to avoid magical thinking
1 / 59

Clinical Research: How to Avoid Magical Thinking - PowerPoint PPT Presentation

  • Updated On :

Clinical Research: How to Avoid Magical Thinking Reese H. Clark, MD Director of Clinical Research, Pediatrix And Consulting Associate Professor Duke University Objectives Define the different levels of evidence use to change clinical care

Related searches for Clinical Research: How to Avoid Magical Thinking

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Clinical Research: How to Avoid Magical Thinking' - andrew

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Clinical research how to avoid magical thinking l.jpg

Clinical Research: How to Avoid Magical Thinking

Reese H. Clark, MD

Director of Clinical Research, Pediatrix


Consulting Associate Professor

Duke University

Objectives l.jpg

  • Define the different levels of evidence use to change clinical care

  • Review how failure to do good clinical research is associated with bad outcomes

  • To discuss different study designs, their strengths and weaknesses.

  • To provide guidance on how to perform good clinical studies that improve neonatal care

What changes care l.jpg
What Changes Care?

  • Evidence

    • Publications

    • Meta-analysis

    • Consensus opinion

  • Personal opinion (in my experience)

  • Institutional history (the way we do it)

  • Personal history (the last case I had)

Experience is the ability to make make the same mistake repeatedly with increasing confidence l.jpg

Experience is the abilityto make make the same mistake repeatedly with increasing confidence

Slide5 l.jpg

The goal of research is to discover, learn, understand and teach principles that improve the quality of life

Levels of evidence l.jpg
Levels of Evidence teach principles that improve the quality of life

  • Level 1 – Results from randomized control trials with meaningful outcome measure

  • Level 2 – Case Control type of studies with treated and untreated patients and minimal evidence of selection biases

  • Level 3 – The patient acts as his or her own control or Case control with some selection bias i.e., historical controls

  • Level 4 – Case series

  • Level 5 – Expert opinion based on experience

Slide7 l.jpg

Pediatrics 2004;114(3):874 teach principles that improve the quality of life

Problem l.jpg
Problem teach principles that improve the quality of life

  • “The best available evidence, however, is not always sound or valid evidence. Sometimes, when faced with a collection of reports that do not constitute good evidence, attempts to choose the best evidence become pointless; in this case, a statement of no good evidence is preferable.”

  • Ambalavanan N et al. Clin Perinatol 2003; 30:305-31

Definition of research http www nsf gov bfa dias policy docs 45cfr690 pdf l.jpg
Definition of Research teach principles that improve the quality of life

  • Research means a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge.

  • Activities which meet this definition constitute research for purposes of this policy, whether or not they are conducted or supported under a program which is considered research for other purposes.

  • For example, some demonstration and service programs may include research activities.

What is research http www hhs gov ohrp humansubjects guidance decisioncharts htm c2 l.jpg
What is Research? teach principles that improve the quality of life

  • Human Subject Regulations Decision Charts

  • The Office for Human Research Protections (OHRP) provides the following graphic aids as a guide for institutional review boards (IRBs), investigators, and others who decide if an activity is research involving human subjects that must be reviewed by an IRB under the requirements of the U.S. Department of Health and Human Services (HHS) regulations at 45 CFR part 46. OHRP welcomes comment on these decision charts. The charts address decisions on the following:

    • whether an activity is research that must be reviewed by an IRB

    • whether the review may be performed by expedited procedures, and

    • whether informed consent or its documentation may be waived.

Most important research issue l.jpg
Most Important Research Issue teach principles that improve the quality of life

Protect the people who are volunteering to participate in a clinical research study.

Drug misadventures l.jpg

Drug Misadventures teach principles that improve the quality of life

The willowbrook study vulnerable population rules l.jpg
The Willowbrook Study teach principles that improve the quality of lifeVulnerable Population Rules

  • The site where a highly controversial medical study was conducted there between 1963 and 1966 by medical researcher Saul Krugman,

  • Healthy children were intentionally inoculated, orally and by injection, with the virus that causes the disease, then monitored to gauge the effects of gamma globulin in combating it. A public outcry forced the study to be discontinued.

  • Researchers defended the deliberate injection of these children by pointing out that the vast majority of them would acquire the infection anyway.

Iatrogenesis brought forth by a healer l.jpg
Iatrogenesis: teach principles that improve the quality of life“Brought forth by a healer"

480 BC

Since Hippocrates's time, the potential damaging effect

of a healer's actions has been recognized:

First do no harm

“primum non nocere”

The road to hell is paved with good intentions……

-St. Bernard of Clairvaux ~1150 AD


Iatrogenesis in neonatology l.jpg

Lowered thermal environment teach principles that improve the quality of lifeincreased mortality 1900–1964

Supplemental oxygenRLF (ROP) 1941–1954

Initial thirsting and starvingneurological deficits 1945–1970

Synthetic vitamin Kkernicterus1945–1961

Sulfisoxazolekernicterus 1953–1956

Chloramphenicol‘‘gray baby’’ syndrome 1956–1960


Hexachlorophenebrain lesions 1952–1971

Epsom saltsenemasmagnesium intoxication 1964–1965

Feeding gastrostomyincreased mortality 1963–1969

Benzyl alcohol‘‘gasping’’ syndrome –1982

Iatrogenesis in Neonatology

From: Robertson AF, Reflections on Errors in Neonatology (Part I,II,III) J Perinatology 2003

With the best intent we can do great harm l.jpg
With the best intent we can do great harm. teach principles that improve the quality of life

  • Eferol -- Increases NEC, liver injury and death

  • Verapamil -- Causes profound bradycardia

  • Post-natal steroids -- May increase brain injury

  • Overventilation -- Increases the risk of PVL

Slide19 l.jpg

Martone WJ et al. Illness with fatalities in premature infants: association with an intravenous vitamin E preparation, E-Ferol. Pediatrics. 1986;78:591-600.

A role for vitamin E in the prevention of retrolental fibroplasia was first reported in 1949.

A number of clinical studies between 1978 and 1983 suggested that vitamin E supplementation (to normal or supranormal serum levels) might prevent or ameliorate the course of retrolental fibroplasia and other complications of prematurity, especially following oxygen exposure.

Vitamin E’s therapeutic or preventive role had not yet been clarified before it was used.

Mmwr april 13 1984 33 14 198 9 http www cdc gov mmwr preview mmwrhtml 00000319 htm l.jpg
MMWR infants: association with an intravenous vitamin E preparation, E-Ferol. Pediatrics. 1986;78:591-600.April 13, 1984 / 33(14);198-9

CDC has received reports from two hospitals of clusters of an unusual illness occurring among low-birth weight (less than 1,500 grams), premature infants in neonatal intensive-care units.

Thirteen affected infants developed clinically significant ascites, in addition to some or all of the following abnormalities: hepatomegaly, splenomegaly, cholestatic jaundice, azotemia, and thrombocytopenia.

All affected infants had received parenteral nutrition therapy, in addition to other supportive measures.

An intravenous vitamin E preparation, containing 25 mg/ml vitamin E, 9% polysorbate 80 and 1% polysorbate 20 in 2-ml vials (E-Ferol Aqueous SolutionR, distributed by O'Neal, Jones & Feldman, St. Louis, Missouri), was introduced in each hospital for addition to parenteral nutrition solutions approximately 1 month before the onset of illness in the first infant in both clusters.

All affected infants received E-Ferol; some affected infants received up to 1 ml or more daily. Both outbreaks ceased shortly after use of E-Ferol was discontinued.

Slide21 l.jpg
Do You Remember E-Ferol? The Penalty for Selling Untested Drugs in NeonatologyJerold F. LuceyPediatrics 1992;89;159;

In 1984, E-Ferol killed at least 38 newborns.

Iatrogenic disasters are often caused primarily by well-intentioned physicians using logical therapies which turned out to have unexpected, lethal side effects.

A poorly managed, avaricious company, O’Neil, Jones and Feldman, Inc. decided to get the jump on the market and sell an untested preparation of intravenous vitamin E.

Physicians assumed E-Ferol had been tested and approved for use by the FDA. It hadn’t been tested.

An astute clinician spotted the problem.

On January 19, 1989, three defendants pleaded guilty and were sentenced to fines of $130 000 each and 6-month jail sentences. These penalties were for conspiracy, mail fraud, and Cosmetic Act Felony.

Problems l.jpg
Problems Drugs in Neonatology

  • Clinical practice is dynamic

  • Study design and execution can take years

  • Muticenter studies require coordination which can further delay the process

  • Publication delay is terrible

  • Evidence-based practice requires continuous reevaluation of practice

Change in event rate with time term neonates with meconium aspiration syndrome l.jpg
Change in Event Rate With Time Drugs in NeonatologyTerm Neonates With Meconium Aspiration Syndrome

Study types l.jpg

Study Types Drugs in Neonatology

Study designs l.jpg
Study Designs Drugs in Neonatology

  • Case Series

    • Retrospective

    • Prospective

  • Case-Control

  • Randomized Control Trial

    • Crossover design

    • No Crossover

Case series reporting our experiences l.jpg
Case Series -- Reporting Our Experiences Drugs in Neonatology

  • Evaluates the occurrence of an outcome and the factors associated with that outcome

  • Examples

    • The effect of iNO on oxygenation

    • What factors are associated with, not that cause, IVH?

  • Advantage: Easy to do. Easy to get consent.

  • Disadvantage: No concurrent controls

  • Never proves efficacy or safety

Ways to strengthen a case series l.jpg
Ways to Strengthen a Case Series Drugs in Neonatology

  • Define study plan, outcomes measures, and statistical methods prospectively

  • Avoid hunting for an effect

  • Be careful to evaluate the study sample for selection bias (e.g., ECMO centers admit other institutions treatment failures)

  • Do not over-interpret the results

Case control trials l.jpg
Case-Control Trials Drugs in Neonatology

  • Similar to case series except the cases are compared to a defined group of controls

  • Type of controls:

    • Historical

    • Same period, different location

    • Matched for factors that influence the outcome

  • Gives a better sense of efficacy but selection bias and confounding variables remain a problem

Randomized controlled trials l.jpg
Randomized Controlled Trials Drugs in Neonatology

  • Patients are randomly assigned to one of several defined groups. The management of each group is strictly defined.

  • Crossover design allows patients to “crossover” into other treatment groups. While easier to get consent for, the results are difficult to interpret.

Outcomes study endpoint what is really important l.jpg
Outcomes = Study Endpoint = What is really important? Drugs in Neonatology

  • Physiology -- Heart rate, blood pressure, PaO2

  • Health consequence -- Survival, chronic lung disease, seizures, stroke, learning disability

  • Quality of life -- Joyful participation in life

  • Health economics -- Did we produce the same outcome more efficiently?

Slide31 l.jpg

Outcomes Drugs in Neonatology

“Not everything that can be counted counts,

and not everything that counts can be counted.”

Albert Einstein

Outcomes l.jpg
Outcomes Drugs in Neonatology

  • Primary -- The outcome we are most interested in studying. Sample size is determined by estimating the number of patients needed to evaluate this endpoint. Every aspect of study design is directed at getting a clean measure of this outcome.

  • Secondary -- All other measures of outcome.

Characteristics of a good study endpoint or outcome measure l.jpg
Characteristics of a Good Study Endpoint or Outcome Measure Drugs in Neonatology

  • Easy to measure and to define

    • Survival is easy to define but hard to study

    • Chronic lung disease is easy to study but hard to define

  • Valuable

    • Healthy survival

    • Not a transient rise in PaO2

  • Occurs at a frequency that is feasible to study

  • Outcome change must be attributable to the intervention studied

Surrogate outcome measures l.jpg
Surrogate Outcome Measures Drugs in Neonatology

  • Definition -- A measure that predicts or is closely associated with another measure of outcome

  • Example -- Grade 3-4 IVH is often used as a surrogate measure (or proxy) of neurological outcome. If we decrease the rate of severe IVH, we predict that we will improve neurological outcome.

Failure of surrogate markers l.jpg
Failure of Surrogate Markers Drugs in Neonatology

  • Ment LR et al. Pediatrics 1994;93:543-550

    • Low-dose prophylactic indomethacin decreased IVH from 18 to 12% in neonates 0.6-1.25 kg

    • Also reduced the rate of grade 3-4 IVH from 5 to 1.4%

    • Survival was not significantly different but was better (92 vs. 87%) in the treated group

  • Ment LR et al. Pediatrics 1996;98:714-718

    • Follow-up showed no difference in IQ or the occurrence of cerebral palsy

Prophylactic indomethacin ment et al pediatrics 1996 98 714 718 l.jpg
Prophylactic Indomethacin Drugs in NeonatologyMent et al. Pediatrics 1996;98:714-718

Postnatal steroids l.jpg
Postnatal Steroids Drugs in Neonatology

  • Meta-analysis shows that steroids reduce the risk of CLD in premature neonates (Bhuta et al. Arch Dis Child 1998;79:F26)

  • CLD is associated with poor neurodevelopmental outcome.

  • It might be expected that steroids might improve neurodevelopmental outcome

  • Instead early steroids increase neurodevelopmental problems (Yeh et al. Pediatrics 1998;101)

Meta analysis l.jpg
Meta-analysis Drugs in Neonatology

  • Summarize the results of different research studies of related problems

  • Systematic approach to the identification and abstracting the critical information held in each study

  • Present a comprehensive best estimate meant to summarize what is known about the clinical problem

Meta analysis lelorier et al nejm 1997 337 536 l.jpg
Meta-analysis Drugs in NeonatologyLeLorier et al. NEJM 1997;337:536

Evaluation of meta analysis l.jpg
Evaluation of Meta-analysis Drugs in Neonatology

Definitions l.jpg
Definitions Drugs in Neonatology

  • Relative Risk -- The probability (risk) of being treated with ECMO if you get iNO compared to if you did not get iNO (%ECMO use in iNO treated/ %ECMO in control not treated with iNO)

  • Odds Ratios -- The rate that ECMO patients are treated with iNO compared to patients who do not get ECMO. (%iNO exposure in ECMO patients) / (%iNO use in non ECMO patients). Better applied to morbidity factors like ICH, or CLD

Definitions42 l.jpg
Definitions Drugs in Neonatology

  • Confidence intervals -- How certain are you that the observation falls within your measured result. Usually the number is 95% CI

  • Standard Deviation -- a measure of average variance from the mean (Square root of {Sum(individual values - mean value)2/number of measurements}

  • Standard Error of the Mean -- STD/Square root of the sample size.

Slide43 l.jpg

Relative Risk of Outcome Drugs in Neonatology

Decreased death

Good confidence

Effect but no


No effect

Increased death

Good confidence










Relative Risk of Outcome

Efficacy or equivalency or non inferiority l.jpg
Efficacy or Equivalency or Non-inferiority? Drugs in Neonatology

  • Efficacy trials are directed at proving that one therapy is better than another with 95% confidence

  • Equivalence implies that the two therapies produce the same outcome

    • If one therapy reduces health care cost, then we may only want to show that the two approaches produce similar outcomes

  • Non-inferiority are done to show that patients treated with X do no worse than those treated with Y. Like efficacy but only one tail-test are used. Required sample size is smaller.

Slide45 l.jpg

Relative Risk of Outcome Drugs in Neonatology

The new therapy is

better and no risker

Effect but no


Equivalent but no



Good confidence










Relative Risk of Outcome

Slide46 l.jpg

Relative Risk of Death and/or ECMO Drugs in Neonatology(% Death or ECMO iNO/ % in Control)

NINOS (n = 235)

INOSG (n = 58)

Boston (n = 90)

Ohmeda (n = 155)

Total (n = 538)





Relative Risk of Death/ECMO iNO/Control

Slide47 l.jpg

Relative Risk of Death Drugs in Neonatology

NINOS (n = 235)

INOSG (n = 58)

Boston (n = 90)

Ohmeda (n = 155)

Total (n = 538)










Relative Risk of Death

Rate of ecmo or death ninos trial nejm 1996 l.jpg
Rate of ECMO or Death Drugs in NeonatologyNINOS trial, NEJM 1996

Problems with the general application of any model l.jpg
Problems With the General Application of Any Model Drugs in Neonatology

  • Standardized Rate -- Observed outcome rate divided by the predicted rate

  • Inadequate sample size

  • Selection biases

  • Neonatal care changes and the model must be recalibrated

  • May be slow in identifying poor performers if we have to wait for adequate sample size

Sample size calculations l.jpg
Sample Size Calculations Drugs in Neonatology

  • Dependent on:

    • The absolute event rate in the population being studied

    • The absolute difference between the two groups

    • How certain you want to be in the measured difference

Effect of sample size on confidence interval and probability that the proportions are different l.jpg

p=0.6 Drugs in Neonatology



Effect of Sample Size On Confidence Interval and Probability that the Proportions are Different

Site variation l.jpg

Site Variation Drugs in Neonatology

Center differences outcomes of extremely low birth weight infants l.jpg
Center Differences & Outcomes Drugs in Neonatologyof Extremely Low Birth Weight Infants

Vohr B et al. Pediatrics April 2004, pp 781

Across site variability l.jpg
Across Site Variability Drugs in Neonatology

  • Most sites randomize within center

  • Each site is a mini-trial

  • Differences in care are variable within sites and may effect the efficacy of the drug being studied within that site

    • steroids

    • nosocomial sepsis

    • nutrition

    • saturation targets

Site variability in proportion of neonates alive off oxygen treatment study all treated l.jpg
Site Variability in Proportion of Neonates Alive & Off OxygenTreatment Study (“All Treated”)

Difference between group 1 and group2 alive and off oxygen pma36 for treatment study l.jpg
Difference between Group 1 and Group2 OxygenAlive and off oxygen PMA36 for Treatment Study

Surfactant 2 better

Surfactant 1 better

Solutions l.jpg
Solutions Oxygen

  • Create a network of centers with a common goal to answer important questions

  • Define answerable questions

  • Limit data collection to confounding variables

  • Define strict time lines and review progress on a quarterly basis

  • Present findings at national meetings

  • Interpret results carefully

Slide59 l.jpg

“For I was assailed by so many doubts and errors that the only profit I appeared to have drawn from trying to become educated, was progressively to have discovered my ignorance.”

Descartes, Discourse on Method, 1637.