Paul G. Shekelle, MD, PhD June 2008

1. Case Study: Comparative Effectiveness Reviews at the Southern California EPC: Off-label uses for Atypical Antipsychotics and Therapies for Low Bone Density Paul G. Shekelle, MD, PhD June 2008

2. Issues • Increasing scope of project • Prior systematic reviews – how to use • Adverse events analyses – what to include • New studies published after completion of analyses

3. Scope of project – Low Bone Density • Project initially covered: bisphosphonates, SERMs, calcitonin, and testosterone. • Calcium, Vitamin D – only when compared to the drugs listed above • Estrogen not in original scope • Within class and between class comparisons, plus vs placebo

4. Changes to scope • Discussions with the technical expert panel, AHRQ, and then other stakeholders resulted in frequent changes to scope: • Estrogen: first it was out, then it was in but we could use an existing SR on the topic, but this ended up not being what we needed, then asked to include all reviews on estrogen, then add in ‘select’ trials of estrogen (WHI, HERS) ….. • Multiple contract modifications have been issued, each requiring a new budget and justification • Changes in scope also required frequent changes in the analytic plan

5. Prior systematic reviews – Low Bone Density • Originally this CER was envisioned as a “review of reviews” to which we would add all RCTs published after the search dates • However, there were multiple systematic reviews on various classes of drugs. Each had different search dates and different inclusion criteria, resulting in different findings. This created a great deal of difficulty, particularly as stakeholders would point to the review that best supported their favored conclusion. • Another major challenge was trying to sort out the populations included in the reviews, since our charge was to review the evidence separately for high risk and other risk women, and the existing SRs did not always track this well • Only at the end of the process did we conclude it would have been better to have completely started from scratch.

6. New Guidance on Using Prior Reviews • Whitlock E and colleagues, Ann Intern Med 2008;148:776-82 (May 20 issue) relates experiences from several EPCs • Main topics are: • Locating existing SRs, focusing on places where the highest quality reviews reside • Assess relevance in terms of PICOTS • Assess methodologic quality – Oxman, AMSTAR • How to use? In total? In part? Just to reference mine? • Transparency of what was done.

7. Methodological steps when using existing systematic reviews (SRs) Whitlock, E. P. et. al. Ann Intern Med 2008;148:776-782 Whitlock, E. P. et. al. Ann Intern Med 2008;148:776-782

8. Adverse events analyses –what evidence to include? • One key issue not previously discussed is whether to include evidence only from studies of the use of the drug for the indications in the CER, or data from other uses of the drug • This raised its ugly head in both of our CERs • “Off label” uses of atypical antipsychotics: PTSD, OCD, Tourette’s, Bipolar Depression, Dementia • What to do about the larger body of data on AEs in the schizophrenic population? • Including the schizophrenic data explodes the literature base • Excluding it leaves an evidence base where weight gain is not proven and limited data about EPS

9. Adverse events analyses –what evidence to include? • This problem also came up in the LBD CER • Estrogen – use only side effects seen in studies of use for osteoporosis, or all studies? • Jaw necrosis and use of bisphosphonates • Jaw necrosis has not been reported in studies of bisphosphonates used for low bone density, only in treatment of cancer patients and only in IV preparations. • This means the risk of jaw necrosis was initially left out of this CER • This wasn’t acceptable to users, so we ended up including case reports for this side effect

10. Late-breaking studies • Studies are often published while CER is under final review • This unhappy event occurred with both our CERs – when the reports were already final, and just days from being released, there were major RCTs published in the NEJM • Option 1: Re-conduct meta-analyses & adverse events analyses to include new study • Takes a lot of extra time. For LBD, during this time yet another study was published – on Afib • Option 2: Create addendum • Option 3: Ignore – makes stakeholders angry

11. CERs – Lessons Learned • In addition to the previously identified problems of few head-to-head studies and heterogeneity among studies, additional troublesome problems have been: • Stakeholders requesting changes of scope en route • The belief that prior systematic reviews will make our task easier • What to do about evidence of AEs from studies of use of the drugs in other health conditions • Just like when it seems you’re done: you’re not!