SECONDARY ANALYSES IN CLINICAL TRIALS

1. SECONDARY ANALYSES IN CLINICAL TRIALS Presented by: George Bigelow, PhD Daniel J. Feaster, PhD Abigail G. Matthews, PhD December 6, 2013

2. Objectives • Review the statistical issues with analyzing and interpreting secondary analyses, and demonstrate the multiple testing burden • Explain the importance of secondary outcome and analysis identification during protocol development • Discuss reporting and interpretation of secondary analyses, including the perspective of the CTN Publications Committee

3. Outline • Introduction and motivation • Statistician’s perspective • Multiplicity • Implementing secondary analyses • Summary • Discussion

4. Publications Committee Perspective on Secondary Analyses: Opportunities & Cautions

5. Opportunities • CTN encourages multiple publications • We want to learn as much as possible • Large-N, diverse, multi-site studies • Broad study teams with diverse interests • Extensive investment in assessments • Repeated assessments over time • Assessment commonalities across studies

6. Cautions • CTN studies typically yield multiple publications • Question: Are we over-analyzing the data? • Discussions in Publications and Executive Committees • Consequence: This webinar

7. Cautions (cont.) • Multiple testing incurs risk of false conclusions • Proper planning can reduce this risk • Acknowledgement of limitations is essential

8. Example of Multiple Publications • CTN006/007: MIEDAR – Abstinence-Contingent Incentives • Report of primary outcome • Do contingent incentives reduce stimulant use?

9. Example of Multiple Publications CTN006/007: MIEDAR – Abstinence-Contingent Incentives Reports of secondary outcomes: Do contingent incentives affect… …HIV risk behavior? …gambling? …cost or cost effectiveness? …methamphetamine use? …staff attitudes?

10. Example of Multiple Publications CTN006/007: MIEDAR – Abstinence-Contingent Incentives Reports of moderator variable associations: Are incentive effects related to… …gender, race, ethnicity? …treatment history? …criminal justice involvement? …urinalysis result at intake? …gambling history?

11. A Caution About Demographic Subgroup Differences Be cautious of thinking of subgroup differences as inherent characteristics of those groups or of individuals within those groups Demographic subgroup differences are very likely the result of some correlated confounding variable; they likely reflect differences in life experiences and opportunities, and the contexts in which drugs are encountered

12. Example of Multiple Publications CTN006/007: MIEDAR – Abstinence-Contingent Incentives Reports of associations unrelated to the study intervention: What symptoms are related to dependence on various drugs?

13. Types of Analyses • Intervention Effects on Primary Outcome • Intervention Effects on Secondary Outcomes • Analysis of Moderators or Mediation • Associations

14. Common Errors • Mistaking correlations for causes • Mis-describing the study methods • Overlooking explanatory confounding variables • Failing to acknowledge limitations

15. Correlation is Not Causation Avoid language that implies causality when reporting associations

16. Describe Methods Accurately • Understand and describe original study accurately • Explain origins and methods of secondary analysis • Idea should precede looking through the data • Describe types and numbers of analyses performed

17. Consider Confounding Factors • One report examined relationship between study pay and proportion of Ss present at the final assessment • Proposed implausible conclusion that greater pay led to less retention • Failed to note that pay amount was related to study duration and difficulty

18. Proceed with Caution Many of the factors that must be considered in conducting and reporting secondary analyses are the same as those important for careful and thoughtful reporting of primary analyses However, secondary analyses can also involve some special statistical considerations, as will be discussed by the following speakers

19. Statistician’s Perspective and the issue of multiplicity

20. What do we mean by primary and secondary outcomes and analyses? • The primary outcome is the main outcome variable for the study • Research hypothesis based on this measure • Used to power study and determine statistical significance of any treatment effect • Analytic method must be specified a priori in the Statistical Analysis Plan (SAP) at a minimum

21. What do we mean by primary and secondary outcomes and analyses? • Secondary analyses are any other analyses, e.g.: • Sensitivity analysis of primary outcome measure with respect to missing data • Subgroup analyses by age, race, gender, ethnicity, disease severity, etc. • Secondary outcomes are any other outcome measures

22. Why secondary analyses? • Publish or perish!!! • Possible that primary outcome measure data ends up being unreliable • e.g., using TLFB but high rate of discordance between self-report and UDS • Analytic issues with pre-specified primary analysis • e.g., proposed distribution of the primary outcome variable does not hold and alternative methods should be used

23. Why secondary analyses? (cont.) • Possibly poor power of primary analysis if assumptions used in sample size are not appropriate • Sensitivity analyses of primary outcome with respect to missing data – key for addiction research • Subgroup analyses • Race, ethnicity, gender required by NIH • Baseline severity of disease (Nunes et. al., 2011) SPECIFY A PRIORI IN PROTOCOL OR SAP AS MUCH AS POSSIBLE!

24. Why secondary outcomes? • Publish or perish!!! • In addiction research we always focus on abstinence but is that enough? • Improved overall quality of life • Engaging in less risky sexual behaviors • Less illegal activity such as theft or prostitution • CTN TEAM Task Force recommends at least one secondary outcome measure be related to “functioning, satisfaction, or quality of life”

25. Why secondary outcomes? (cont.) • Again, what if there are unanticipated issues with the primary outcome? • Cannot “hang your hat” on only one outcome measure SPECIFY A PRIORI IN PROTOCOL OR SAP AS MUCH AS POSSIBLE!

26. Example of Utility of Secondary Analyses • CTN: Women with Trauma and Addictions • Primary outcome results (Hien et. al., 2009): • Trauma symptom severity: NS • Abstinence: NS • Secondary analyses: • Women with baseline eating disorders had significantly less improvement in PTSD severity and abstinence • SS significantly reduces unprotected sex in high risk women over time • Racial/ethnic matching with therapist associated with SS effectiveness • Examples of other positive findings: retention, sleep disorders, intimate partner violence

27. Denise Hien CPDD 2013 Presentation:

28. Words of Warning • Too many post hoc analyses opens one to accusations of data dredging • Secondary analyses/outcomes cannot be used to evaluate the trial as a whole (only primary outcome) • If there are a substantial number of pre-specified secondary outcomes and analyses, consider adjusting for multiple comparisons • Appropriate interpretation of results is key • Hypothesis generating

29. Cautionary Tale • Convicted of wire fraud: “willfully overstating in a press release the evidence for benefit of a drug his company made” • Primary outcome p-value=0.08 • Asked his statisticians to identify sub-group with significance • Patients with mild to moderate disease severity: p=0.004 • Press release acknowledged negative finding from primary outcome analysis but maintained drug associated with increased survival • Post: “…everyone agrees there weren’t any factual errors in the four-page document. The numbers were right; it’s the interpretation of them that was deemed criminal.” • During appeal court said: “Statements are fraudulent if ‘misleading or deceptive’ and need not be literally ‘false’.” Scott Harkonen, MD

30. Why such controversy? Multiplicity • Type I error is preserved (usually 5%) for primary outcome(s) • If performing multiple secondary analyses, then overall Type I error will be higher • If enough analyses are performed, there will be at least one spurious association • Adjustment not necessary for secondary analyses/outcomes, but interpretation must be cautious and presentation of results forthright and transparent

31. Illustration of Multiplicity • Generate 10 outcome variables independently from normal with mean=0 and variance=1 for 300 participants • Calculate Spearman correlation coefficient for each pair-wise combination • Test correlation coefficient ≠ 0 • Type I error estimated as the number of tests that are statistically significant divided by number of tests (45)

32. Illustration of Multiplicity (cont’d) Type I Error Rate = 6/45 = 13.3%

33. Implications • Avoid post hoc analyses • Pre-specify as much as possible (protocol or SAP) → → Avoid data dredging criticism → Can even adjust Type I error rate for number of secondary analyses performed (rare) • Interpret secondary results keeping in mind the inflated Type I error rate

34. Responsible Analysis and Reporting • Focus should always be on primary outcome • Of secondary analyses, focus should be on those that were pre-specified • Requires careful planning with statement of hypotheses in protocol/SAP (SAP should be finalized before data lock) • Report in a manuscript the number of pre-specified analyses performed and the number reported

35. Responsible Analysis and Reporting (cont’d) • Present estimates of treatment differences and CIs: “plausible range of treatment differences consistent with trial results” • Interpretation needs to be viewed as exploratory rather than confirmatory • Frame results in context of supporting or contradictory data from other studies

36. Responsible Analysis and Reporting (cont’d) • For post hoc analyses: • Acknowledge that analyses were not specified a priori (data driven) • Describe why analyses are important and the relevance of the research question • Report number of post hoc analyses performed and the number reported • Significance should be viewed as descriptive and not used for inference or decision making • Can be used to justify future research

37. Examples If primary outcome not statistically significant but some pre-specified secondary analyses were: While the primary outcome did not demonstrate statistically significant evidence of a treatment effect, some secondary analyses suggested that the treatment may be effective. Therefore, future research is warranted. If primary outcome is statistically significant but no secondary analyses are: The primary outcome was statistically significant indicating that treatment is effective in this study population. Despite the fact that numerous secondary analyses did not yield statistical significance, there is sufficient evidence to justify future research of this intervention.

38. Questions…

39. The Design Stage Implementation of secondary analyses

40. Multiple Types of Secondary Analyses • Secondary hypotheses—Utilize the design • Mediation studies—Use data post-randomization • Association Studies—Normally don’t use the design • Example: Predictors of HIV Testing (CTN0032) • Since do not use the study design—observational!

41. Multiple Types of Secondary Analyses (cont.) • Subgroup or Moderator Analyses • Risk reduction counseling impact on HIV testing by modality of substance use treatment (CTN0032) • Differential Treatment Effects by Race/Ethnicity and/or Gender • We do not randomize to subgroups—observational!

42. Why consideration at design stage? • Appropriate measures • Sample size considerations • For secondary analyses that do NOT use design: • Causal interpretation is difficult • Statistical models can help • Subject to assumptions—no unmeasured confounders • Implies need to think about and measure confounders for any secondary analysis that is not just a test of difference by randomized treatment group

43. Secondary Outcomes • Simplest type of “secondary” analysis • Other outcomes on which we feel the intervention will have an impact • Analysis strategy frequently very similar to primary outcome analysis • Need to consider multiple testing issue! • If enumerate and measure 20 secondary outcomes, the Type 1 error is .64 (if we use α = .05 for each test) • Each secondary outcome being in a separate paper does NOT change this fact

44. Mediation Analyses • Another analysis frequently included in protocols • Because pieces of the model are determined after randomization, there is difficulty making a strong causal interpretation Attendance in Treatment Treatment Assignment Count of Drug Use Days

45. But we do not randomize attendance, so even if observed for everyone: CONFOUNDERS Attendance in Treatment Treatment Assignment Count of Drug Use Days We cannot rule out confounders of both Attendance and Drug Useand, therefore, cannot make strong causal statements about the gold pathway without strong assumptions.

46. Possible Assumptions No Unmeasured Confounders Instrumental Variables If we can find exogenous factors, Z, that are correlated with Attendance, X Also, Z does not directly affect Count of Drug Use Days Can use Z as instruments to identify the causal impact of attendance on Count of Drug Days Use the instrumental variable in place of the endogenous variable • If we measure all potential confounds, then can make causal statements SUBJECT to the assumption that we have measured ALL the confounds • May want to measure potentially confounding factors • Example: Propensity Score Analysis

47. Association Studies • Like mediator models, association studies are looking at the impact of variables which the experimenter has not controlled on some chosen outcome→Do not make causal claims (unless include confounder analysis or Instrumental Variable(s))! • Frequently these studies will look at numerous predictors • Type I error • Must be very clear and honest about the way the analyses were done

48. Potential Solution for Type I Error • Machine Learning approaches • Used in data mining • Allows an exhaustive search for the best predictive model of the outcome • Like testing all covariates, sometimes can over-fit • Cross-validation • Simplest approach is split sample, explore on one sample, then replicate on a second sample (the second sample is a “test” of results on the first sample)

49. Subgroup or Moderator Analyses: Should Determine Which Subgroups Are of Interest at Design Stage • Many of us have interests in: • Racial and ethnic groups • Gender • But other subgroups may be of interest • Drug of choice • Severity of individual’s problem • Age • Socioeconomic status • Site Differences (in levels of outcome and/or treatment effects) • PTSD/No PTSD • Important to define groups a priori (Necessary to consider at the design phase)

50. Many (if not most) subgroups are not randomized. • This means these are observational models and cannot make causal statements • Should assess for confounds • Should be careful not to over-interpret • Even if assess for confounds, cannot rule out unobserved confounds • Race/Ethnic differences are examples where differences are largely NOT causal (race/ethnicity is correlated with true casual agent)