The VR ROI Project: Recent Results for Virginia and Maryland

1. The VR ROI Project: Recent Results for Virginia and Maryland 11thAnnual Summit on Performance Management Excellence Oklahoma City, OK September 5, 2018

2. Session Objectives • To describe the VR ROI project’s approach to examining the impact of VR on long-term employment outcomes and estimating the ROI of state VR programs • To present results of recent ROI analyses of data from Virginia and Maryland • To discuss how such an approach can be used to inform VR policy and program decision-making

3. Overview of VR ROI Project • Funded by NIDILRR • 2010 FIP involved 4 state VR agencies: both VA agencies, MD, and OK • Objective: develop and test a valid, rigorous model for assessing ROI at state agency level • 2014 DRRP expanded to 8 state agencies across 6 states: VA, MD, DE, KY, NC, TX • Refine and test the ROI model with more heterogeneous set of state agencies • Explore feasibility of a VR-ROI Calculator • Using data on 3 cohorts: VR applicants in SFY 2000, 2007, 2012

4. Overview of Our Approachto Estimating ROI for State VR agencies 4

5. Brief Overview of Our Approach • VR ROI Project has developed a set of tools that describes the economic value of VR services while providing information for program improvement • We want to be as accurate as possible and at the same time as meaningful to you all as possible. • Key VR staff at Maryland DORS & Virginia DARS have been essential to the project development and continual refinements • Results are intended to be helpful in consideration of service delivery and policy development but are not ‘the one answer’ or to be used without context.

6. Key Features of Our Approach (1 of 7) • Uses longitudinal wage data from quarterly employment records. • Spans several years of pre-VR earnings as well as five or more years following receipt of VR services. • Estimates VR’s impact from when services begin, not when they end (i.e., using applicant cohorts rather than closure cohorts). • Separately examines the impact of VR for individuals with different kinds of disabling conditions. • These results focus on 2007 estimates for individuals with a Mental Illness (MI), Physical Impairment (PI), or Cognitive Impairment (CI, including both intellectual disability and learning disabilities)

7. Key Features(2 of 7):Pre-VR Employment Pre-VR Employment Rates by Disability Type: 2007 Applicants to Virginia General

8. Key Features(3 of 7):Pre-VR Earnings if Employed Mean Pre-VR Earnings (if Employed) by Disability Type: 2007 Virginia General Applicants

9. Key Features of Our Approach (4 of 7) • Estimates the impacts of specific types of VR services rather than a “generic” VR service (see next slide) • Thus, explicitly recognizes that VR is an individualized program and that different VR participants receive different kinds of services. • Uses state-of-the-science statistical controls to ensure that outcomes are the result of VR rather than other factors. • Estimates employment and earnings impacts as well as service costs at the individual level.

10. Key Features (5 of 7):Service Types ― DTERMPS • Service Categories & Mapping • Diagnosis & evaluation: services for assessing eligibility and developing IPE, medical diagnostics • Training: career & tech training, JRT, OJT, GED, vocational, etc. • Education: post-secondary education • Restorative: medical/healthcare services • Maintenance: transportation, clothing, vehicle/home modifications, rent, etc. • Placement: employment services, WWRC vocational support services, etc. • Job Supports: job coaching, supported employment

11. Key Features(6 of 7):Service Providers • Services might also have differential impacts depending upon their source • By agency counselors – client-specific information is limited in AWARE for this cohort • Purchased from external vendors – available through AWARE by case and service category • Provided by state-operated rehabilitation facility (WWRC in Virginia, WTC in Maryland) • We capture the following impacts relative to someone who did not receive the service from either source. For each DTERMPS category, estimate the impacts on employment and earnings when provided by: • an outside vendor but not WWRC/WTC, • WWRC/WTC but not an outside vendor, or • both sources.

12. Key Features (7 of 7):Ensure that VR Impacts are Due to VR Rather than Other Factors • Model include about 25 control variables • Demographics • Disability-related • Economic conditions, both local and national • Local proportion of employment in (a) federal government and (b) a different state • Model includes several state-of-the-science statistical features • One makes use of variation in the mix of services across counselors and across field offices

13. Overview of Cohortsand Context for Results

14. Overview of Cohorts:Applicants for VR Services in SFY2007 • Separate analyses, by agency, for individuals with: • MI: Mental Illness • 27% of DARS cohort, 41% of DORS cohort • PI: Physical Disabilities • 32% of DARS cohort, 40% of DORS cohort • CI: Cognitive Impairment (ID and LD) • 24% of DARS cohort, 23% of DORS cohort • An individual could be in multiple cohorts. The cohorts used in estimation include • 59% of the 10,849applicants to Virginia DARS • 75%of the 9,018applicants to Maryland DORS

15. Some Context for the Results • The Great Recession of 2008 and beyond • Our employment & earnings data goes through 2012 – so 5 years of post-application data • Transition-age youths range from none to 25% of a cohort’s sample. Prior labor market experience provides a baseline in our model but youths have little, if any, prior experience. • Because most categories were closed during this period, Maryland operated with lengthy waiting lists. • Data are not available to estimate non-market impacts of VR (e.g., levels of independent living, community integration, self-efficacy)

16. Order of Results Presentationfor VA DARS and MD DORS • Present VA DARS first and then MD DORS • Labor market effects of service receipt: DTERMPS • For each of the disability types: MI, PI, CI • ROR (Rate of Return) estimates • For each of the disability types: MI, PI, CI

17. 2007 Cohort Estimates:Virginia DARS

18. Effects of Service Receipt on Employment and Earningsby DTERMPS and Disability

19. Individuals Diagnosed with a Mental Illness (MI):Long-Run (> 2 years after app.) Labor Market Impacts VA DARS: Received Purchased Services Only • Notes for these 3 slides: • Although WWRC also provides services to DARS clients, this chart only shows service impacts when the service was purchased. • Additional charts have been developed that show service impacts by DTERMPS when the service is provided (a) by WWRC (but not purchased) or (b) from both sources.

20. Individuals Diagnosed with a Physical Impairment (PI):Long-Run (> 2 years after app.) Labor Market Impacts VA DARS: Received Purchased Services Only

21. Individuals Diagnosed with a Cognitive Impairment (CI):Long-Run (> 2years after app.) Labor Market Impacts VA DARS: Received Purchased Services Only

22. Annualized Rates of Return(ROR) for VA DARS By Disability Group

23. ROR: Combining Benefits (Labor Market Impacts) with VR Cost Estimates • “A rate of return is the gain or loss on an investment over a specified time period, expressed as a percentage of the investment's cost” (Investopedia) • We measure the “gain” as the dollar value of the labor market impact results across all DTERMPS service categories, both purchased and from WWRC • We attempt to estimate the full “cost” of VR, including • Purchased services • WWRC charges for services • ACP costs (administrative, counseling, and placement services) • A positive ROR indicates that employment and earnings “gains” more than covered VR “costs.” • A zero ROR indicates that “gains” just offset “costs.” • The long-run annual return is about 1% in money market accounts and 10% in the U.S. stock market.

24. Annualized Rates-of-Returnby Disability Group for VA DARS

25. 2007 Cohort Estimates:Maryland DORS

26. Effects of Service Receipt on Employment and Earningsby DTERMPS and Disability

27. Individuals Diagnosed with a Mental Illness (MI):Long-Run (> 2 years after app.) Labor Market Impacts MD DORS: Received Purchased Services Only • Notes for these 3 slides: • Although WTC also provides services to DORS clients, this chart only shows service impacts when the service was purchased. • Additional charts have been developed that show service impacts by DTERMPS when the service is provided (a) by WTC (but not purchased) or (b) from both sources.

28. Individuals Diagnosed with a Physical Impairment (PI):Long-Run (> 2 years after app.) Labor Market Impacts MD DORS: Received Purchased Services Only

29. Individuals Diagnosed with a Cognitive Impairment (CI):Long-Run (> 2years after app.) Labor Market Impacts MD DORS: Received Purchased Services Only

30. Annualized Rates of Return(ROR) for MD DORS By Disability Group

31. Annualized Rates-of-Returnby Disability Group for MD DORS

32. Reflections and Discussion from Maryland DORSand Virginia DARS

33. Reflections from Maryland DORS(1 of 3) • “Results are intended to be helpful in consideration of service delivery and policy development but are not ‘the one answer’ or to be used without context.” • Consider Diagnostics • Negative Return on Investment for 2007 Applicant Cohort? • DORS emphasis on counselor professionalism and use of available diagnostics resulted in service purchased in the most difficult cases. • Authorizations for medical records ($15) and broken appointment fees could also negatively be reflected in the ROI • ROI for diagnostics could be affected by those who did not make it to IPE

34. Reflections from Maryland DORS(2of 3) • During 12 quarters prior to application: • Employment participation and wages (for those employed) decreased for those with physical impairments (PI). • Employment participation increased and wages decreased (for those employed) for the Cognitive Impairment (CI) and Mental Impairment (MI) disability groups. • Transition Youth with IEPs (often CI) receiving real work experience before applying to DORS • DDA and BHA providers referring consumers for intensive job coaching support after they have achieved and/or lost employment or had wages decreased.

35. Reflections from Maryland DORS(3of 3) • Labor Market Participation by individuals participating in WTC training: • Strong for all three groups (CI, MI, and PI) • Employment propensity and earnings substantially higher when service exclusively provided by WTC • Improved IPE and authorization service category coding: • Head start on internal controls documentation for RSA 911 reporting • Increased consistency in service coding across the Agency

36. Discussion • Comments on / reactions to these results from: • Maryland (John) • Virginia (Rick) [In other words, what do these results mean to us… and what don’t they mean?] • Other thoughts / reactions from Virginia & Maryland: • How you have used or might use this information; • Impact on your agency of involvement in the VR ROI project; or • Other?

37. Virginia DARS:Results for PERT,a WWRC Program SFY2000 Applicants to Virginia DARS

38. VR – ROI Feature # 7:ROI of a Collaborative Transition Program PERT Impact on Finding a Job and Income • PERT (Post-secondary Education/Rehabilitation Transition) Program • Comprehensive career and independent living skills assessments at WWRC for high school students with disabilities who are selected by local school divisions • Community-based team implementation of assessment findings • Participants may receive additional VR services following PERT participation