SHQ Improvement Project: Enhancing Health Actuarial Services

1. WSHIP SHQ Improvement Project November 8, 2007

2. Reden & Anders • Has over 16 years of experience providing health actuarial services, medical management strategy, and analytic consulting nationally to health plans, insurers, employers and government entities • National practice – can access full breadth of subject matter experts within R&A across all offices, and can utilize other Ingenix experts as needed • Software products that address diverse needs, including those related to underwriting and actuarial, claims editing, fraud and abuse, payment systems, and high-risk case identification

3. Project Team Experts in large multi-payer databases, predictive modeling and health status surveys • Michael Cousins, Ph.D. – Engagement lead • Economics of disease and medical management programs, predictive modeling • Pay-for-performance analytics • Mary Hegemann, F.S.A., M.A.A.A. – Actuarial certification lead • State high risk health plan pricing, financial projections, and plan analysis • Rating, underwriting, benefit analysis of insured and uninsured • Actuarial certifications for small group practices, Medicare MA and Part D bids • Bret Jensen, A.S.A. – Database manager and analyst lead • Risk profiling tools, including CDPS, MRX, DxCG, ACG • Large, multi-payer database management and analysis • Jeffrey Goldstein, MD, MS, FACHE – Clinical and peer review • Patient safety and practice management • Change management expert, implement strategies to improve clinical outcomes • Regulatory compliance, corporate compliance and accreditation • Michael Manocchia, Ph.D. – Health status survey design lead • Health care and wellness data capture and intake • Focus groups and cognitive testing • Academic and business health status survey research • Sarah Benson, MBA – Project manager

4. Peer Review and Consultative Team Experts in high-risk insurance pools • Jim Drennan, F.S.A., M.A.A.A. – Senior advisor and peer review • State high risk plan pricing, financial projections, and plan analysis • Rating, underwriting, benefit analysis of insured and uninsured • Earl Hoffman, F.S.A., M.A.A.A. – Senior advisor and peer review • State high risk health plan pricing, financial projections, and plan analysis • Rating, underwriting, benefit analysis of insured and uninsured

5. Purpose • Prepare the Board to receive the Tool Committee’s SHQ recommendation in December • How is the new form similar and dissimilar to the current version? • How is the new form better than the current version? • How will the consumer’s experience be changed?

6. Agenda Topics • Re-certification and re-design process overview • Analytic approach for current and new SHQ • Survey usability improvement and re-design • Timeline and next steps • Additional questions

7. Project Goals • Evaluate, revise, and certify that SHQ reasonably identifies the predicted top 8%, in accordance with RCW 48.41.060 • Incorporate input from key stakeholders • Conduct modeling benchmark study • Simplify and improve the SHQ paper and web forms • Ensure smooth transition from the current SHQ

8. Overall Approach Parallel tracks that converge November 9th Conditions and Points for Top 8% Form for Applicants Nov. 9th

9. SHQ Re-Certification Overall Approach Form for Applicants • Gather input from stakeholders • Market-based qualitative and quantitative research • Conduct expert survey review • Conduct cognitive interviews • Pilot SHQ paper and web surveys • Update SHQ conditions, language, format, and usability Conditions and Points for Top 8% • Gather input from stakeholders • Collect, validate, and build claims and modeling databases • Conduct modeling and benchmark study • Update SHQ conditions • Calculate point values and determine rejection thresholds • Certify the results • Verify scoring and online system results

10. Two Selected Add-ons • Assess relative accuracy of current predictive model (ie, CDPS) that is used to benchmark prediction of top 8% • Compare CDPS to Ingenix industry-leading predictive models • Substantially improve SHQ’s format and usability • Improve usability and completeness of paper and electronic forms

11. November 9th Deliverables for Tool Committee • Provide report to Tool Committee of the WSHIP Board • Methods comparisons • Summary of results • Recommendations • SHQ Form • Points and thresholds • Transition plan • Data request for next certification • Provide the Certification Memorandum to the Tool Committee

12. Timeline and Next Steps 12/10 11/9 11/26 12/3 Today Interim Reviews by Tool Committee (3) SHQ Form and Certification Report to WSHIP Staff (Then, 2 week review for stakeholders) Input from Stakeholders (Then, planning 11/26 with WSHIP staff) Recommendation from Tool Committee (Then, submit to WSHIP Board) Review by WSHIP Board

13. Agenda Topics • Re-certification and re-design process overview • Analytic approach for current and new SHQ • Comparison of approaches • Predictive modeling benchmark study • Survey usability improvement and re-design • Timeline and next steps • Additional questions

14. Analytic Methods for Current SHQ • Prior Certification (Milliman) • Used Milliman’s Small Group Underwriting Manual • Mapped Medical Claims to Questionnaire • No Detailed Rx Claims • Questionnaire Score for each Person • Claims Data Modeled by CDPS • Risk Score for each Person • Comparison of Questionnaire Score and Risk Score for each Person • Manually adjusted SHQ points if deemed appropriate • Adjusted Risk Scores per Person were determined • Top 8% were identified

15. Analytic Methods for New SHQ • New Certification (Reden & Anders) • Grouped medical claims data from individual market (includes carriers and WSHIP) into clinical categories called Episode Treatment Groups • >800 ETGs • Example: “Joint degeneration, localized, with surgery - knee & lower leg” • Created predictive model using ETGs to predict Year 2 costs • Adjusted for prescription drugs • Adjusted for under-reported claims (eg, mental health) • Risk weights were determined for each ETG • ETG risk scores per person were determined from the data • Top 8% were identified

16. Predictive Modeling Comparison • Why assess accuracy of CDPS predictive model for this certification? • We wanted to use the CDPS predictive model to select conditions for the SHQ (ie, use the WA individual market data to tell us which conditions to include on the SHQ) • CDPS predictive model used for several years by WSHIP as “the standard”, yet its accuracy wasn’t known and its performance hadn’t been compared to other models • How did we assess CDPS predictive model? • Used medical claims data from individual market to predict Year 2 costs using CDPS, Impact Pro, and using Impact Pro’s ETGs

17. Predictive Modeling Comparison • How did the different model tools compare? • Accuracy at predicting top 8% was comparable between all the models • Why wasn’t there a difference? • We think it is because we are approaching the upper limit of how well administrative claims data can be used to predict future costs. • And, some data fields, such as pharmacy, place of service, type of service, provider ID, and bill type, were not available for this certification

18. Predictive Modeling Comparison • Since there wasn’t a difference, why did we select the ETG model and not use the CDPS model to create the SHQ questions? • CDPS model has about 100 conditions, while the ETG model can have up to 800 conditions * -- since the modeling accuracy is similar, based on input from the Tool Committee, it was agreed we should use the model with the more detailed conditions so applicants can make more precise responses (which will increase overall accuracy) * As will be shown below, we started with about 800 conditions and found that 228 of these conditions should be on SHQ to predict the top 8%

19. Predictive Modeling Comparison: Summary • CDPS model has similar accuracy as other models (but, this good performance level was a moot point for previous certifications since the CDPS model was not used to select conditions – we think it was only used to adjust some of the points) • Choice of model for SHQ conditions • ETGs are discrete enough to serve as the basis for questions on the SHQ • CDPS indicators are too aggregated to meet this need • Consolidated the 800 ETG conditions into 228 ETG conditions that had utility at predicting top 8%

20. Current v New SHQ: Numbers Looking just at these numbers, the underlying analytics may not seem very different – but the new SHQ is based on a very different approach Note: The rejection point threshold for the new SHQ was mathematically adjusted to equal the current SHQ to help minimize potential confusion in the marketplace. Despite this, the underlying methodology for the new SHQ, including the conditions and points, is quite different from the current SHQ.

21. Differences in Analytic Approach • Benefits • Use of actual claims data, specific to Washington, specific to the individual market to identify the conditions and scores * • Use of widely-used clinical groupings (ETGs) • Scores account for co-morbidities • Drawbacks • Loss of convenience from using a small group underwriting manual In addition to analytic improvements, there are also improvements to the actual web/paper form – these will be described in the next section. * Points for a few rare conditions were determined by study results in published scientific medical journals

22. Agenda Topics • Re-certification and re-design process overview • Analytic approach for current and new SHQ • Survey usability improvement and re-design • Timeline and next steps • Additional questions

23. Survey Improvement and Re-design • Improve usability and completeness of paper and electronic forms

24. Revision to Current SHQ • Improve usability/completeness of paper/web forms General Approach • Revised study design to include expert and secondary data analysis • Use survey expert review to critique current form • Analyze feedback from Carrier Workgroup regarding SHQ problems encountered at the consumer level • Evaluate appropriateness of new items not from claims – HRA based items of significance • Field test (e.g., “cognitive interviews”) and revise new SHQ as necessary • Run a paper and web-pilot • Analyze new SHQ data from data quality, reliability, and validity standpoints

25. Survey Improvement and Re-DesignMethodology Overview • Market-based quantitative and qualitative research • Survey carriers, agent/brokers and consumer advocates • Qualitatively review carrier workgroup notes • Completed and presented August 30, 2007 • Conduct expert survey review • Completed and presented August 30, 2007 • Cognitive interviews in Seattle • Completed and presented on October 17, 2007 • Pilot SHQ paper and web survey • Completed and presented on October 17, 2007 • Create final SHQ form (integrate all conditions and improvements) • To be submitted November 9, 2007

26. Expert Review • Three external expert reviewers offered many insights to improve the presentation of the SHQ and pointed out methodological flaws in the survey: • Michael Massagli, Ph.D. (www.patientslikeme.com) • Virginia Casey, Ph.D. (Harvard School of Public Health) • Kate Morrow, Ph.D. (Brown University Medical School) • Incorporated findings into the design • Expert reviewers design issues taken into account • Modeling work – eliminate items due to lack of statistical significance • Incorporated comments by carriers and agents • bold when condition has a limitation in form • Working on format, directions, graphics on web and paper form – including site pal.

27. Cognitive Interviewing Team • Kate Morrow, Ph.D. – Qualitative Researcher, Cognitive Interviewer and Consultant to R&A • Brown Medical School, Clinical psychologist • Working in the infectious disease field (HIV and Hep-C work) • Conducted survey development in the past • Relevant qualitative research work • Michael Manocchia, Ph.D. – Health status survey design lead • Health care and wellness data capture and intake • Focus groups and cognitive testing • Academic and business health status survey research • GMR Transcription, Inc. • Provided full transcription services from audio interview files • Chosen due to medical and health research transcription experience

28. Cognitive Interview Conclusions • Overall, presentation of new SHQ form accepted • Recognition of conditions based on past experience and not experience on the form • Past confusion in filling out the form led to rejection or multiple submissions in the past • Directions to fill out the form can be tweaked, but are OK with respondents • Exceptions still problematic • The YES – NO type of condition box needs to be drawn out and coded with the condition table • General acceptance of the new form, by a majority of research subjects with tweaks to the paper SHQ to be made

29. SHQ Survey Review Carriers, Agents/Brokers and Non-Carrier parties • SHQ carrier, agent/broker and non-carrier survey developed • 10 Question survey • Person identifiers • Indicate their interaction with SHQ and consumers • Problem scores on • Understanding directions • Understanding medical conditions • Understanding where to check appropriate boxes • One key change • General ratings on understanding and ease in filling out the SHQ • Sent to 5 carriers, 20 agents and 2 non-carriers • 80% return from carriers and 90% return from agents

30. Survey Pilot Conclusions Results from the new SHQ indicate • Instructions were rated favorably • Exceptions as questions were completed by almost all participants • YES – NO condition box was completed correctly most of the time – otherwise completely ignored until condition respondent had was also checked. • If condition was checked, respondents completed other warranted information 90% of the time • New behavioral questions were answered appropriately • Overall, ratings were very favorable of the new form

31. Current v New SHQ: Usability

32. Differences in Usability • Benefits • Use of actual claims data, specific to Washington, specific to the individual market to identify the conditions and scores • Use of widely-used clinical groupings (ETGs) • Scores account for co-morbidities • Consumer input from "target market" (people denied coverage) • Improvements based on scientific cognitive testing • Consumer input for legibility, ease of comprehension, etc. • Drawbacks • Dependent on complete and accurate data from carriers

33. Questionnaire insert new screen shots

34. Agenda Topics • Re-certification and re-design process overview • Analytic approach for current and new SHQ • Survey usability improvement and re-design • Timeline and next steps • Additional questions

35. Overall Approach Parallel tracks that converge November 9th Conditions and Points for Top 8% Form for Applicants Nov. 9th

36. Timeline and Next Steps 12/10 11/9 11/26 12/3 Today Interim Reviews by Tool Committee (3) SHQ Form and Certification Report to WSHIP Staff (Then, 2 week review for stakeholders) Input from Stakeholders (Then, planning 11/26 with WSHIP staff) Recommendation from Tool Committee (Then, submit to WSHIP Board) Review by WSHIP Board

37. Additional Discussion