1 / 36

On the interpretation of responder analyses and NNTs

On the interpretation of responder analyses and NNTs. Stephen Senn. An apology. I will talk mainly about responder analysis I have little to say about numbers needed to treat. Genes, Means and Screens.

eydie
Download Presentation

On the interpretation of responder analyses and NNTs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. On the interpretation of responder analyses and NNTs Stephen Senn (c) Stephen Senn 2011

  2. An apology • I will talk mainly about responder analysis • I have little to say about numbers needed to treat (c) Stephen Senn 2011

  3. Genes, Means and Screens It will soon be possible for patients in clinical trials to undergo genetic tests to identify those individuals who will respond favourably to the drug candidate, based on their genotype…. This will translate into smaller, more effective clinical trials with corresponding cost savings and ultimately better treatment in general practice. … individual patients will be targeted with specific treatment and personalised dosing regimens to maximise efficacy and minimise pharmacokinetic problems and other side-effects. Sir Richard Sykes, FRS, 1997 (c) Stephen Senn 2011

  4. Soon? (c) Stephen Senn 2011

  5. (c) Stephen Senn 2011

  6. (c) Stephen Senn 2011

  7. The Pharmacogenomic Revolution? • Clinical trials • Cleaner signal • Non-responders eliminated • Treatment strategies • “Theranostics” • Markets • Lower volume • Higher price per patient day (c) Stephen Senn 2011

  8. Implicit Assumptions • Most variability seen in clinical trials is genetic • Furthermore it is not revealed in obvious phenotypes • Example: height and forced expiratory volume (FEV1) in one second • Height predicts FEV1 and height is partly genetically determined but you don’t need pharmacogenetics to measure height • We are going to be able to find it • Small number of genes responsible • Low (or no) interactive effects (genes act singly) • We will know where to look • In fact we simply don’t know if most variation in clinical trials is due to individual response let alone genetic variability (c) Stephen Senn 2011

  9. My Opinion • Most of the hype is due to a failure to understand response • Responder analysis is to blame • And related to this is an obsession with Numbers Needed to Treat • Which is increasing the pressure to use dichotomies (c) Stephen Senn 2011

  10. A Thought Experiment • Imagine a cross-over trial in hypertension • Patients randomised to receive ACE II inhibitor or placebo in random order • Then we do it again • Each patient does the cross-over twice • We can compare each patient’s response under ACE II to placebo twice (c) Stephen Senn 2011

  11. (c) Stephen Senn 2011

  12. NB These are conditional probabilities of response on the second occasion. They are not conditional probabilities of being a ‘true’ responder. (c) Stephen Senn 2011

  13. (c) Stephen Senn 2011

  14. (c) Stephen Senn 2011

  15. ? (c) Stephen Senn 2011

  16. NOTE FOR GUIDANCE ONCLINICAL INVESTIGATION OF MEDICINAL PRODUCTSIN THE TREATMENT OF HYPERTENSION 1998 P2 Arbitrarily, response criteria for antihypertensive therapy include the percentage of patients with a normalisation of blood pressure (reduction SBP < 140 mmHg and DBP < 90 mmHg) and/or reduction of SBP ≥ 20 mmHg and/or DBP ≥ 10 mmHg. Results obtained should be discussed in terms of statistical significance and in relation to their clinical relevance. The first word in this paragraph is the most important (c) Stephen Senn 2011

  17. Dichotomania • Continuous measurements taken and referred to baseline • Patients dichotomised as responder/non- responder • Inefficient • Arbitrary • Sheep versus goats • Ignores geep and shoats • Analysis on risk difference scale to calculate NNT (C) Stephen Senn 2006

  18. Y = outcome, X = baseline If (Y < 90  X > 95)  (Y < 0.9X) patient ‘responds’ (C) Stephen Senn 2006

  19. Why I mistrust the NNT • It has very poor properties as a scale • Reciprocal of risk difference • It is theoretically unlikely to be stable from study to study • And this theoretical instability has been demonstrated by empirical research • It is an impatient measure • It tries to shortcut the steps from study to practice • It is an illusion that this can be done • Those who advocate it are preferring an easy lie to a difficult truth

  20. Pharmacogenetics: A cutting-edge science that will start delivering miracle cures the year after next. (c) Stephen Senn 2011

  21. Moerman and Placebos • Paper of 1984 • Investigated 31 placebo-controlled trials of cimetidine in ulcer • Found considerable variation in response • Considered placebo response rate was an important factor • Has been cited by others as proof of variation in treatment effect from trial to trial (c) Stephen Senn 2011

  22. (c) Stephen Senn 2011

  23. Lessons from Moerman • There is no evidence of variation in the treatment effect from trial to trial • We should be wary about concluding that apparent variation signals true variation • We need to be cautious and think carefully about analysis • Of course…it is always possible that there was exactly the same genetic mix in each trial • in which case gene by treatment would not manifest itself as trial by treatment interaction • We need to understand components of variation (c) Stephen Senn 2011

  24. Pharmacogenomics: A subject with great promise. (c) Stephen Senn 2011

  25. What you learn in your first ANOVA course • Completely randomised design • One way ANOVA • Randomised blocks design • Two way ANOVA • Randomised blocks design with replication • Two way ANOVA with interaction • No replication, no interaction (c) Stephen Senn 2011

  26. 1. Senn SJ. Individual Therapy: New Dawn or False Dawn. Drug Information Journal 2001;35(4):1479-1494. (c) Stephen Senn 2011

  27. (c) Stephen Senn 2011

  28. A Word of Caution • What is additive on one scale is not additive on another • The Moerman example suggests a constant effect on the log-odds ratio scale • If the background risk varies this translates into a varying effect on the risk-difference scale • The biological interpretation of this is then moot • However the practical implication of this is summarise on the additive scale (c) Stephen Senn 2011

  29. (c) Stephen Senn 2011

  30. The Mottos • Additive at the point of study • Relevant at the point of application • If NNTs have their place it is in decision making for individual patients • Not in reporting results from individual trials • The additive scale has to be transformed into the relevant scale at the point of treatment • The fact that NNTs might be relevant when making an individual decision is not an excuse for summarising results this way (c) Stephen Senn 2011

  31. Tiotropium v Placebo in Chronic Obstructive Pulmonary Disease From the UPLIFT Study, NEJM, 2008 Significant differences in favor of tiotropium were observed at all time points for the mean absolute change in the SGRQ total score (ranging from 2.3 to 3.3 units, P<0.001), although the differences on average were below what is considered to have clinical significance (Fig. 2D). The overall mean between-group difference in the SGRQ total score at any time point was 2.7 (95% confidence interval [CI], 2.0 to 3.3) in favor of tiotropium (P<0.001). A higher proportion of patients in the tiotropium group than in the placebo group had an improvement of 4 units or more in the SGRQ total scores from baseline at 1 year (49% vs. 41%), 2 years (48% vs. 39%), 3 years (46% vs. 37%), and 4 years (45% vs. 36%) (P<0.001 for all comparisons). (My emphasis) (c) Stephen Senn 2011

  32. Two Normal distributions with the same spread but the Active treatment has a mean 2.7 higher. If this applies every patient under active can be matched to a corresponding patient under placebo who is 2.7 worse off (c) Stephen Senn 2011

  33. A cumulative plot corresponding to the previous diagram. If 4 is the threshold, placebo response probability is 0.36, active response probability is 0.45. (c) Stephen Senn 2011

  34. In summary…this is rather silly • If there is sufficient measurement error even if the true improvement is identically 2.7, some will show an ‘improvement’ of 4 • The conclusion that there is a higher proportion of true responders by the standard of 4 points under treatment than under placebo is quite unwarranted • So what is the point of analysing ‘responders’? (c) Stephen Senn 2011

  35. Who are the authors? 1. Tashkin, DP, Celli, B, Senn, S, Burkhart, D, Kesten, S, Menjoge, S, Decramer, M. A 4-Year Trial of Tiotropium in Chronic Obstructive Pulmonary Disease, N Engl J Med 2008. Personal note. I am proud to have been involved in this important study and have nothing but respect for my collaborators. The fact that, despite the fact that two of us are statisticians, we have ended up publishing something like this shows how deeply ingrained the practice of responder analysis is in medical research. We must do something to change this. (c) Stephen Senn 2011

  36. In conclusion • Responder analysis is the source of much confusion • It is leading trialists to overestimate the individual element of response to treatment • The key to understanding response is replication and careful analysis • Stupid dichotomies do not help this understanding • NNTs may be relevant at the point of application but they are not relevant at the point of study • Personalised medicine may be about to happen ‘soon’ for quite a few years to come yet (c) Stephen Senn 2011

More Related