Dr. Stavros Petrou Presentation to Nottingham CTU 19 th June 2008

1. Economic evaluations alongside clinical trials: what they contribute, how they are performed and their limitations Dr. Stavros Petrou Presentation to Nottingham CTU 19th June 2008

2. What is economic evaluation? • Premise: scarce (health care) resources • Aim: to maximise health gain with the available resources • Method: compare cost and consequences of interventions • Balance: about costs and consequences, inputs and outputs • Economic evaluation: explicit criteria for making choices.

3. Definition of economic evaluation • Definition of economic evaluation: “The comparative analysis of alternative courses of action in terms of both their costs and their consequences” (Drummond et al, 1997) • Requires: • a comparison of two or more alternatives • examination of both costs and consequences

4. Types of economic evaluation Is there good evidence on effectiveness of interventions being compared? NO Is effectiveness of interventions equal? YES Costing study NO YES Cost minimization study YES Can all outcomes be valued in monetary terms ( e.g. willingness to pay)? Cost benefit analysis NO NO Cost-effectiveness analysis Can outcomes be measured as quality adjusted life years? YES Cost-utility analysis

5. Two independent groups Control group Treatment group Patient (Cost, Effect)1 ( CC1, EC1 ) 2 ( CC2, EC2 ) 3 ( CC3, EC3 ) . . nC ( CCn, ECn ) Mean: ( CC, EC ) Patient (Cost, Effect)1 ( CT1, ET1 ) 2 ( CT2, ET2 ) 3 ( CT3, ET3 ) . . nT ( CTn, ETn ) Mean: ( CT, ET ) Economic evaluation alongside trials

6. Economic evaluation alongside trials Two independent groups Control group Treatment group Mean: ( CT, ET ) Mean: ( CC, EC ) Incremental cost-effectiveness ratio CT - CC ET - EC

7. New treatmentmore costly NW NE New treatmentless effective New treatment more effective C New treatmentless costly SW SE The cost-effectiveness plane New treatment more effectivebut more costly Existing treatmentdominates New treatment dominates New treatment less costly but less effective

8. New treatmentmore costly NW NE Maximum acceptable ICER New treatmentcost-ineffective New treatmentless effective New treatment more effective C New treatment cost-effective New treatmentless costly SW SE The cost-effectiveness plane

9. NICE: method of operation • Preferred measure of cost-effectiveness: • Quality-adjusted life year (QALY) • Alternatives - e.g. cost per life year gained - acceptable • No absolute threshold for level of acceptability: • no empirical basis for setting a value • may in some circumstances want to ignore threshold • A set threshold implies efficiency has absolute priority over other objectives (e.g. fairness) • many technology suppliers are monopolies; a threshold would discourage price competition

10. Cost per QALY results from NICE Those in bold: rejected Source: N Devlin, D Parkin. Does NICE have a cost-effectiveness threshold and what other factors influence its decisions? A binary choice analysis. Health Economics 2004: 13(5):437-52.

11. Why QALYs as a measure of outcome? • To use cost-effectiveness as a guide to decision-making, we need to compare the c-e of different uses of resources • Therefore we need an effectiveness measure that can be used in a wide range of settings: • Life-years gained • but only where survival is main outcome • Quality adjusted life years (QALYs) • Composite of survival and quality of life

12. 1.0 QALYs gained Quality of life valuation u Health profile with intervention Health profile without intervention 0 1 2 3 4 5 6 Time in years t

13. The EuroQol EQ-5D The following questions are designed to tell us about your state of health today. Please look at each group of questions, and then tick the statement which best describes you own health state today. You should make five ticks in all, one for each group. Mobility I have no problems walking about ____ I have some problems walking about ____ I am confined to bed ____ Self-care I have no problems with self-care ____ I have some problems washing or dressing myself ____ I am unable to wash or dress myself ____ Usual activities I have no problems performing my usual activities ____ I have some problems with performing my usual activities ____ I am unable to perform my usual activities ____ Pain/discomfort I have no pain or discomfort ____ I have moderate pain or discomfort ____ I have extreme pain or discomfort ____ Anxiety/depression I am not anxious or depressed ____ I am moderately anxious or depressed ____ I am extremely anxious or depressed ____

14. Tariffs for the EuroQol EQ-5D

15. Why randomised trials? • Most treatments do not have large effects; reliably detecting moderate effects requires studies that simultaneously avoid: • moderate bias • proper randomisation • intention-to-treat analysis • avoidance of inappropriate sub-group analysis • moderate random error • adequate size

16. Why economic assessment in clinical trials? • Many health economists advocate models using lots of data sources: trial, non-trial, summary data etc • But...issues of bias and random error also affect incremental resource use and health outcomes • And, trials provide patient-level data, useful for: • Dealing with patient heterogeneity • Examining covariance, e.g. between costs and outcomes • Building and validating models, eg to extrapolate • Trials allow prospective measurement of resources & outcomes of interest • Incremental cost of economic evaluation alongside trials is low

17. UK Collaborative ECMO Trial • Pragmatic RCT • 185 mature (35 weeks, 2kgs) infants with severe respiratory failure (ox. index 40) • Infants recruited from 55 centres in 1993-5 • Randomisation to ECMO: 1 of 5 specialist centres, cannulated, ECMO support • Randomisation to CM: conventional care • Outcomes: survival without severe disability up to 7 years of age

19. Design and analytical issues • Costs: measurement and valuation • Consequences: measurement and valuation • Analytical issues: within and beyond RCTs

20. Perspectives and types of costs • Direct costs • Health care system • Other care inputs, e.g. social services • Patient, family, carer expenses • Informal care costs • Opportunity cost of unpaid informal care • Indirect costs • Time off work, reduced productivity • Early retirement • Premature mortality • Transfer payments • Payments such as social security benefits that redistribute output with no exchange of goods or services

21. Three elements of cost • Resource use (cost generating event) • a day in hospital • a GP consultation • Unit cost • cost per in-patient day / per hospitalisation • cost per GP consultation / per GP minute • Cost • the product of resource use and unit costs

22. Measurement of resource use • All significant resource inputs during first 7 years of life • Trial data collection forms: • transport (mode, distances) • duration and intensity of neonatal care • Observational research for infant death • Postal questionnaires, validated by information from hospital records: • post-discharge hospital and social service utilisation • GP records: • community service utilisation

23. Valuation of resource use • Unit costs employed to value resource use • Neonatal care – top down methodology • Readmissions - Reference cost schedules • Community service utilisation - Published cost data (previous studies, PSSRU, etc.) • Drugs – BNF • £, 2002-3 prices

24. Time horizon and discounting • Should extend far enough into the future to capture all costs and consequences of interventions being evaluated • ECMO Study – time horizon initially reflected that of RCT • Costs (and consequences) occurring beyond the first year of life must be reduced to present values • Rationale for discounting- Time preference- Opportunity cost – market basis

25. Mean costs and mean cost differences Source:Petrou S, Bischof M, Bennett C, Elbourne D, Field D, McNally H. Cost-effectiveness of neonatal ECMO based on seven year results from the UK Collaborative ECMO Trial. Pediatrics 2006; 117(5): 1640-1649.

26. Bootstrap mean cost differences Source:Petrou S, Bischof M, Bennett C, Elbourne D, Field D, McNally H. Cost-effectiveness of neonatal ECMO based on seven year results from the UK Collaborative ECMO Trial. Pediatrics 2006; 117(5): 1640-1649.

27. Measurement of outcomes at 7 years • Survival period → life years gained • Neurodevelopmental assessments performed by developmental psychologist across 6 domains: • cognitive ability - behaviour • neuromotor skills - hearing • general health - vision • Each domain defined as normal, impaired or mild, moderate or severely disabled • Overall status defined by highest degree of impairment or disability in any domain → disability-free life years gained • Limitations of QALYs in childhood context

29. Cost-effectiveness of neonatal ECMO Incremental cost-effectiveness ratio = C / E = £13,385 per life year gained or = £23,566 per disability-free life year gained NB: Natural or physical unit of outcome, which ignores full range of consequences.

33. Mean net benefits of neonatal ECMO Mean net benefits = Rc.E - C

34. Are trial-based economic evaluations sufficient? • Under the right circumstances, Yes • Like clinical evidence, economic evidence can stand alone or be synthesised • Requirements: reasonable comparators, adopts final outcomes, collects data on a sufficiently broad set of services, adequately powered, sufficiently long follow-up, representative patient population • Problems: Use of inappropriate statistical tests, lack of power, failure to handle missing data, lack of intention-to-treat analysis, follow-up too short, lack of transferability

35. Was the ECMO trial-based economic evaluation sufficient? • Reasonable comparator  • Representative patient population  • Intention to treat analysis  • Adequately powered  • Appropriate perspective  • Sufficiently long follow-up x • Adopts final outcomes ? • Appropriate statistical tests  • Handled missing data 

36. Longer-term cost-effectiveness • Simple decision-analytic model • Assumptions: • restricted to first 18 years of life • survivors to 7 years survive to 18 years • disability status at age 7 does not vary • excess annual costs during years 4-7 continue during years 8-18  ICER: £7344 per life year gained £11802 per disability-free life year gained

37. Vehicles for economic evaluation • Prospective collection of data alongside randomised controlled trials • Least subject to bias, control over instruments, low incremental cost. May need to supplement. • Prospective collection of data alongside non-randomised studies • More subject to possible bias, control over instruments, low cost. • Retrospective analysis of available data • Low control over design and data, subject to bias. • Modelling study • Data from different sources, combined in decision-analytic models. Hard to validate. Useful and often unavoidable adjunct to trials.

38. Limitations of trials as a vehicle for economic evaluation Trial limitations Inappropriate or partial comparisons More than one trial Partial measurement Unrepresentative practice Intermediate outcomes Limited follow-up No trials NICE Examples Temozolomide (recurrent malignant glioma) Drugs for Alzheimer’s Riluzole (resource use) Glycoproteins Beta interferon (MS) Implantable cardioverter defibrillators Liquid-based cytology

39. Modelling • Hence modelling aims to address all these questions and can be used instead of or as a complement to trial evidence • Structure the economic question • Extrapolate beyond observed data • Links intermediate and final endpoints • Generalizes results to other settings/patient groups • Synthesises evidence and can create head-to-head comparisons where RCTs don’t exist • Can indicate the need for further research

40. Models should: • Represent a simplification of the real world • Encourage decision-makers to be explicit • Reflect current clinical practice and use appropriate comparators • Be based on the best quality data available • Cover the appropriate time period • Include sensitivity analysis to explore uncertainty of data inputs and model structure • Be transparent and reproducible • Have internal and external validity

41. The realities of research funding • A large proportion of funding for economic evaluation is attached to trials • Three options available to analysts: • Satisfy expectation of standard trial-based economic evaluation and risk misleading results • Refuse to collaborate • Pragmatic collaboration: • Seek opportunities to use modelling to inform trial design • Ensure sufficient budget for analysis which includes synthesis and modelling

42. A pragmatic way forward?

43. Suggested checklist for assessing economic evaluations • 1) Was question well-defined? • 2) Were alternatives clearly described? • 3) What evidence on effectiveness was used? • 4) Were resources measured and valued fully & appropriately? • 5) Was discounting necessary and was it performed? • 6) Were incremental costs and outcomes analysed? • 7) Was an adequate sensitivity analysis performed? • 8) Are the results adequate to inform purchasing? • 9) Are the conclusions justified? • 10) Are the results applicable to the local population? • Source: Drummond MF, O’Brien, B, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. 2nd edition. Oxford: Oxford University Press, 1997. • Drummond MF, Jefferson T, et al. Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ 1996; 313:275-83.