Introduction To Evaluating Process And Outcomes In Research Design

1. Introduction To Evaluating Process And Outcomes In Research Design Sarita Bhalotra Donald Shepard August 18, 2004

2. Human Services Evaluation 1960s • Johnson’s “Great Society” proliferation of experimental and pragmatic H.S. programs • Results in need for evaluation

3. Human Services Evaluation 1970s • Experimental approach of programs inadequate to understand human services programs • Traditional approach of evaluation over-reliant on bio-medical model

4. Human Services Evaluation 1980s • Post modernist approach emphasizes • practical problem-solving orientation • to program evaluation • “Outcomes” are personal or organizational • changes or benefits as a result of activity, • intervention, or service

5. Lifestyle Modification Program Demonstration: Evaluation • 1. Outcomes Evaluation • a. Clinical • b. Cost/Utilization • c. Cost-effectiveness • 2. a. MBMI and LA/PMRI • b. Program Sites • c. Del Marva • d. CMS

6. PROGRAM THEORY FOR MEDICARE LIFESTYLE MODIFICATION DEMONSTRATION Population Age > 65 years Clinically high-risk for progression of cardiovascular disease Additional Services Nutrition Exercise Stress Management Group Support Health Status Reduced Cardiovascular Events Improved Q.O.L. Reversal of Cardiovascular Disease (Cost-saving/cost-neutral) Appropriate Effective Efficient Resources Additional Medicare Funds Additional Demonstration Site funds In-Kind Contributions

7. ContextLMPD is testing two lifestyle interventions nationwide:1. Mind/Body Medical Institute’s Cardiac Wellness Program2. Dr. Dean Ornish Program for Reversing Heart Disease Research QuestionWhat organizational characteristics explain the number of Medicare beneficiaries who participate in the Medicare LMPD? Specific Aims • Determine the critical stakeholders and their roles in the Medicare LMPD • Establish the critical steps and their sequencing needed for the successful participation of Medicare beneficiaries in the Medicare LMPD. • Analyze the characteristics of, and interactions between and among stakeholders that impacts the critical steps and the extent to which these affect the participation of Medicare beneficiaries in the Medicare LMPD. • Develop a program model of the structure, processes, and intermediate outcomes of a successful organization in terms of Medicare beneficiary participation.

8. Hypotheses • Successful sites will have identified and cooperated with critical stakeholders, especially referral sources, in the planning phase. • Successful sites will have designed an effective senior management team, and selected and train motivated staff. • Successful sites will have developed and implemented participant-focused marketing, recruitment, enrollment and retention techniques. • Successful sites will have developed and implemented procedures for tracking and maintaining relationships with stakeholders

9. LMPD Evaluation Institutionalization Demonstration Success Parameters Enrollment Quality Cost

10. Human Resources Leadership Clinical Managerial Process Evaluation: Program Theory for Sites Participation Adequate Sustained Institutionalization Cultural Financial Marketing Provider Relations HQ Assistance Information Systems Facilities Funding/Investment Revenues from Payers

11. Cumulative Enrollment in Lifestyle demonstration by Program

12. Medicare Lifestyle Modification DemonstrationEnrollment Review and Participation Status

13. Did Not Meet Criteria Other Barriers Enrollment Barriers For Patients Contacted by Nurse Recruiter, First Ten Months (excluding those in Process), n = 1, 387 Note: Information was updated on slightly different dates. Minor discrepancies in totals reflect differences in dates of data collection.

14. Program A, 4 sites 10% 9.1% 9% 8% 7% 6% 5% 3.8% 4% 3% 1.7% 1.5% 2% 0.9% 1% 0% Initially Interested And Initiated Application Continued Interest and Submitted To CMS for Medicare Part B Eligibility Still Interested And Submitted To Delmarva For Clinical Eligibility Remain Interested And Obtaining Clinical Information Approved By Delmarva And Ready to Join Next Cohort Results of Nurse Recruiter, First 10.5 Months (n = 1706)

15. Nurse Recruiter Outcomes, First 10.5 Months (n=1706

16. Medicare Patients Program Referring Host Facility Cardiac Rehab Program Staff Providers Marketing Recruitment Enrollment Retention Selected Stakeholder Groups and Key Steps Leading to Participation(Specific Aims 1 and 2)

17. Barriers to Enrollment Across Programs • Participants • Out of pocket cost • Time of day (e.g., ends after dark) • Time demands • Rigorous diet; too time-consuming in preparation • Program is too stringent to follow • Lifestyle changes are not supported by environment • Extensive documentation needed for enrollment • Time consuming screening required • Program staff • General mailing is expensive • Limited time available for some targeted recruitment despite its value (e.g. inpatient nurses) • No time for other targeted recruitment (e.g., via cath lab) • Documentation is labor intensive • No benefits to program staff apparent from documentation

18. Barriers to Enrollment Across Programs • Host Facility • Not convinced of clinical efficacy • Financial loss leader • Takes away from other programs (e.g. cardiac rehab) • Impinges on turf of other physicians • Highly politicized in some cases (e.g. “champion’s cause”)

19. Barriers to Enrollment Across Programs • Community physicians/ Referral sources • Not convinced of efficacy of programs • Not sure patients will comply even if efficacious • Easier to prescribe drugs, other medical or surgical intervention • Concerned about loss of revenue/patients • Do not want to refer patients who they think will fail • Frustrated when referred patients face long delays or are not found to be eligible • Frustrated by amount of paperwork required to document a patient’s eligibility • Concerned by the time of the doctors and nurses to explain the program to a patient • Information available is incomplete and outdated

20. Key Activities, Interactions, and Characteristics of Successful Organizations (Specific Aims 3 and 4) • Marketing is multi-modal, but targeted especially to referring physicians • Marketing anticipates physician resistance, and is convincing of the clinical efficacy, feasibility, and lack of threat to current practice • Institutional supports financially, culturally, and clinically • Eligibility criteria do not pose either a substantive or logistic barrier • Reimbursement is by all payors or majority • Program is a substitution for or complementary with traditional cardiac rehab • Program Leadership is conducive to trust and credibility • HQ Leadership is conducive to trust and credibility

21. Examples From Site Visits On a scale of 1 to 5, from ineffective to effective.

22. Major Issues The target community opposes the change being advocated The adoption costs exceed tangible benefits Effective structural and motivational systems designed to engage and reward are needed at every level Consumer needs, preferences and lifestyles are paramount • Conclusions • Enrollment in LMPD, as in other preventive programs, is a challenge • Environmental scans help identify barriers and ways to overcome them • Interventions, such as the nurse recruiter, are achieving partial success • Lifestyle Modification enrollment remains below program expectations