Disclosures:

1. Markov ModelingTraining in Clinical ResearchDCEA Lecture 6 UCSF Department of Epidemiologyand Biostatistics March 11, 2010Aaron B. Caughey, MD, PhDabcmd@berkeley.edu

2. Disclosures: • Tandem Technology – Prenatal Diagnosis Company • To no other commercial relationships

3. Good Books • Nudge – Richard Thaler, Cass Sunstein - Nice summary of basic behavioral econ tenets • Checklist Manifesto – Atul Gawande - Nice background / explanation of NEJM paper

4. Objectives: • To understand the definition and uses of a Markov simulation or disease state model • To understand steps to design a Markov model

5. Background • Many diseases progress through stages or states • Consider DM, CRD, Cardiac Dz, etc. • physiologic abnormality • mild then moderate clinical disease • complications • end-stage • Ongoing risk of shifting between stages • over months or years • Various outcomes and risks in different states

6. Background • Up to now - portrayed clinical outcomes with: • short-term outcomes • constant lifetime probabilities • Sometimes more appropriate to model as: • diseases in stages • movement between stages. • Markov disease state simulation

7. Outline 1. What is a Markov simulation? 2. When should I do a Markov simulation? 3. Steps in a Markov simulation

8. 1. What is a Markov simulation? Portray progression of a disease over time • Divide the disease into discrete “states” • Need to specifically define time interval • Then specify initial distribution of: • risks of progression per time interval • Assign utilities and costs to each state and transition • Conduct simulation with defined end-point • E.g. a number of years/cycles or death

9. 2. When should I do a Markov simulation? • Probabilities/utilities change over time • e.g., risk of stroke (or disutility) increases with age • Face validity • Conceptualized as having discrete, progressive states • Data availability • Data on risk of disease progression/intervention effectiveness more readily available for short time periods • Multiple opportunities for intervention • Portray effects of interventions occurring at multiple stages in disease progression

10. Aneurysm CEA Example 1. In older population all-cause mortality competes with the risk of SAH, and increases as the cohort ages. 2. SAH risk data are available for short time periods only, easily translated to annual risk and not as easily to lifetime risk.

11. Course of HIV, impact of increased early HAART Conducted with Markov for 3 reasons: • Data on HIV progression and treatment effectiveness focus on disease state transitions. • Face validity: clinicians, epidemiologists, others think of HIV disease in stages: infection, worsening CD4 and viral load, pre-AIDS disease, AIDS, and death. 3. HAART can be used in different stages of disease – in fact, the effect of HAART timing is the issue being assessed.

12. OB Gyn Examples - Markov • HNPCC+ • Prophylactic hysterectomy vs. surveillance • IVF • What is the optimal number of embryos to transfer? • Postpartum contraception • Cost effectiveness / economic consequences

13. ExamplesFrom the Class • Hepatic transplant?

14. Diseases for which Markov may add little • Key chance nodes have a short time frame • Quick resolution/stabilization of condition • Short term data not available, only lifetime • Examples: • acute curable infections • management of acute events (MIs, strokes) • cancers (if prognosis captured with a few branches) • immunizations for non-epidemic childhood infections (e.g., hemophilus influenza)

15. 3. Steps in a Markov - Overview • Similar to standard model building • A. Structure the simulation • Portray disease states and transitions • Determine end stages • B. Obtain data for the transition probabilities • C. Implement the model: • Building • Calibrating • Debugging • Sensitivity analysis / Monte Carlo Simulations

16. A. Structure the Simulation Portray disease states – fundamentals • Include all important states of the disease: • often stages of severity, • e.g., renal disease in diabetes: severity of renal compromise (normal, micro-, macroalbuminuria, end-stage renal disease) • sometimes recurrent events • e.g., recurrence/remission. • also often health states induced by therapy • (e.g., side-effects)

17. Defining Disease States - Practical Issues Precisely what states? • Discrete shifts mark boundaries • changed health status, e.g., hypertension to stroke • arbitrary/convention, e.g., micro/macro albuminuria • Working definitions in the field • Data exist on progression • Balance simplicity and completeness • Interventions being studied • Usually need absorbing state (e.g., death)

18. Defining Disease States – Aneurysm Example Aneurysm: long-term outcomes calculated by modeling movement among four states: • Healthy • Mild disability (due to surgery or SAH) • Moderate-severe disability (ditto) • Death

19. 10,000 women with HNPCC Age 30 Prophylactic Surgery (TAH/BSO) Annual Testing (USN, CA125, EMB, Exam) Annual Exams (No other testing) HNPCC - Methods

20. Prophylactic Surgery (TAH/BSO) Ovarian Cancer Die from Surgery Survive Surgery Stage I Stage II Stage III Stage IV Endometrial Hyperplasia/Cancer Die from Surgery Survive Surgery Stage I Stage II Stage III Stage IV Synchronous Ovarian & Endometrial CA No Cancer on Pathology HNPCC - Methods

21. Annual exams Interval Ovarian Cancer Interval Endometrial Dz Interval Ov & Endom CA Cont annual surveillance HNPCC - Methods

22. Annual exams Interval Ovarian Cancer Interval Endometrial Dz Interval Ov & Endom CA Cont annual surveillance HNPCC - Methods

23. Annual Testing Endometrial Biopsy (+) TAH/BSO Die from Surgery Survive Surgery Ovarian Cancer Endometrial Hyperplasia/Cancer Synchronous Ovarian & Endometrial Cancer No Cance on Pathologyr EMB nl but Abn Endometrial USN D&C Endometrial pathology TAH/BSO Die from Surgery Survive Surgery Ovarian Cancer Endometrial Hyperplasia/Cancer Synchronous Ovarian & Endometrial Cancer No Cancer on Pathology Otherwise return to annual surveillance Elevated CA125, abnormal ultrasound, OR persistently abnormal CA125 TAH/BSO Die from Surgery Survive Surgery Ovarian Cancer Endometrial Disease Synchronous Ovarian & Endometrial Dz No Cancer on Pathology Otherwise return to annual surveillance HNPCC - Methods

24. Annual Testing Endometrial Biopsy (+) TAH/BSO Die from Surgery Survive Surgery Ovarian Cancer Endometrial Hyperplasia/Cancer Look at Model ! Synchronous Ovarian & Endometrial Cancer No Cance on Pathologyr EMB nl but Abn Endometrial USN D&C Endometrial pathology TAH/BSO Die from Surgery Survive Surgery Ovarian Cancer Endometrial Hyperplasia/Cancer Synchronous Ovarian & Endometrial Cancer No Cancer on Pathology Otherwise return to annual surveillance Elevated CA125, abnormal ultrasound, OR persistently abnormal CA125 TAH/BSO Die from Surgery Survive Surgery Ovarian Cancer Endometrial Disease Synchronous Ovarian & Endometrial Dz No Cancer on Pathology Otherwise return to annual surveillance HNPCC - Methods

25. Portraying transitions Possible transitions between states • Single “forward” transitions (i.e., from state 1 to 2, 2 to 3, 3 to death) • Forward jumps (e.g., from 1 to 3, see HIV example) • Rarely: backward transitions (e.g., from 3 to 2) • more realistic to add state (e.g., 3 in remission); • state achieved via a sicker state ≠ state achieved via a healthier state. • Death, in Markov simulations, due to disease or other causes

26. Risk of progression • Risk per unit time (i.e., per Markov cycle) in “source” state e.g., For individuals with microalbuminuria, 5% annual risk of progressing to macroalbuminuria • Time-period risk can evolve e.g., annual risk of mortality increases with age

27. Effectiveness of Interventions Usually represented as reduction in the risk of progression e.g., ACE-inhibitors decrease risk of micro- to macroalbuminuria by 70%

28. Disease state outcomes Each disease state assigned utility (and cost) per cycle • Utility: If annual cycles, might be the portion of a QALY gained by being in that state for that year • Costs: direct, total, etc. • Keep track of utilities and costs accumulated in each state in each cycle  cumulative totals available at end

29. Simulation structure • Track movement between states over time. • Portray individual or group (e.g., 1000) • End with specified duration, or when per cycle utilities below threshold (e.g., 0.001 QALY)

30. Graphic techniques • Simple flow diagrameffective for basic Markov states and transitions; limited transition probabilities possible without clutter.

31. Multi-cycle bubble diagram(Fig 1 Naimark) • Clear and more information -- evolving state distributions and cumulative outcomes • Not often used in published Markov analyses, probably because unwieldy with more than 3-4 states

32. “Markov subtree” (Naimark Fig 2) • Markov with infinity symbol (∞) instead of chance node; states with branches; transitions with boxes at the end of each sub-branch. • If complex, as in example, multiple subtrees needed. • Excellent at documenting structure, but requires understanding trees and has no natural way to report transition probabilities.

33. Transition matrix • Efficiently summarizes states and transitions • Corresponds to structure used to analyze Markov (pre fast computers)

34. Multi-column table • Allows more information (e.g., effectiveness) with some loss in organizational efficiency. HIV disease:States match CDC definitions + common clinical distinctions.

36. B. Data for transition probabilities • Precise extraction and adaptation of published (or custom) data • Plus usual data for CEA

37. For the aneurysm Markov: • annual probability of aneurysm rupture (SAH) from a prospective cohort, assumed constant over time • one-time risks of death and disability from surgery / SAH from various studies. • annual age-adjusted risk of death all causes from life tables, studies of individuals with disabilities.

38. For renal disease in diabetes: • probability of progression and death from natural history studies (annualized) • effectiveness (reduction in progression) from trials

39. When data go missing If detailed state-to-state transitions unavailable … • data on larger jumps establish “benchmarks” • benchmarks used to assign values to intervening transition probabilities (“calibration”)

40. HNPCC Example • Annual risk of cancer not available • Decade of life specific risk available • Assumed at baseline – uniform distribution • Varied in SA – increasing distribution • Ran simulation and checked for similarity • calibration

41. C. Implement the model • Building • Calibrating • Debugging • Sensitivity Analysis / Monte Carlo Simulations

42. Building the model • Standard protocols in decision analysis software • Spreadsheet: custom programmed in tables • Successful model-building: • careful planning of states and transitions • programming from simple to complex, initially only a few transitions • check results repeatedly to confirm plausibility

43. Calibration • If empirical benchmarks available, especially if more trustworthy than transition data -- calibration process: • goal = transition probabilities that produce results consistent with real-world data • time-consuming - working backwards • E.g. preterm labor and preterm birth

44. Debugging / Quality Control • Markov models complex, rarely “transparent” • Essential to monitor the accuracy of model outputs • Check outcomes at extremes of ranges • E.g. Check incremental outcomes with efficacy=0 • One-way SAs for effect / plausibility • Markov trace: • distribution by state for each cycle • Shows evolution of disease progression over time c • Examine for reasonable patterns

45. Reporting of results • Similar to that for any CEA – expected values for each arm and the net differences between arms

46. HNPCC Example • Assuming 10,000 women with HNPCC at age 30: • 10,000 undergo prophylactic TAH/BSO • 5 women will be diagnosed with ovarian cancer at surgery • 6 women will be diagnosed with endometrial cancer at surgery • Life expectancy 79.98 • 10,000 undergo annual screening • 370 women will be diagnosed with of ovarian cancer • 1,840 will be diagnosed with endometrial cancer • Life expectancy 79.31 -0.67 • 10,000 undergo annual exams • 830 women will be diagnosed with ovarian cancer • 4,870 will be diagnosed with endometrial cancer • Life expectancy 77.41 -2.5 Chen LM, et al. Obstet Gynecol, 2007

47. HNPCC Example • Assuming 10,000 women with HNPCC at age 30: Chen LM, et al. Obstet Gynecol, 2007

48. IVF Example Little SE, et al. Obstet Gynecol, 2006

49. IVF Example Little SE, et al. Obstet Gynecol, 2006

50. IVF Example Incremental Cost-Utility of 2 vs. 1 Embryo Transfers – Varying the maximum number of cycles Little SE, et al. Obstet Gynecol, 2006