Performance Measurement & Reporting

1. Performance Measurement & Reporting For Employment And Training Programs

2. Agenda • Quick review of Agenda and Ground Rules for the Session • This morning’s session: • Foundation for Performance • Revisions to existing reporting requirements • Data sources • Calculating outcomes on the performance measures

3. Ground Rules for Today & Tomorrow • Cell phones on mute • Questions, Answers and Parking Lot issues– Flip Charts • The session is designed to provide information on the definitions of the common measures as outlined in TEGL 28-04 and on the revised reporting instructions. We cannot debate whether or not common measures are “good or bad.”

4. A Foundation for Performance Management

5. Improving Performance Results • Reporting performance is a fundamental element of improving customer services and good public administration • Performance information should be easily understood by all customers, stakeholders, and operators of the workforce investment system • Establishing common measures and standardizing customer data collection improves the comparability and understanding of performance results • Ensuring the accuracy and timeliness of performance results is necessary for demonstrating system integrity and value

6. Establishing Common Measures • President’s Management Agenda • Affects six agencies (29 programs) • Department of Labor • Department of Education • Department of Health and Human Svcs. • Department of Veterans Affairs • Department of the Interior • Department of Housing and Urban Development • TEGL 28-04 issued on April 15, 2005 • Rescinds TEGL 15-03 • Policy applies to DOL funded programs only • July 1, 2005, implementation for W-P, VETS and WIA; October 1, 2005, implementation for TAA

7. The Common Measures • Adult measures: • Entered employment • Employment retention • Earnings increase • Youth and lifelong learning measures: • Placement in employment or education • Attainment of a degree or certificate • Literacy or numeracy gains • Designations of adult or youth are spelled out in each program’s eligibility requirements

8. Benefits of Common Measures • Focus on the core purposes of the workforce system; employment for adults and skill gain for youth • Break down barriers to integration resulting from different definitions, data and reports for each workforce program • Resolve questions raised by GAO and other oversight agencies regarding the consistency and reliability of data • Reduce confusion among our customers and stakeholders who want to know about results

9. The Intersection of Common Measures and Revised Reporting Requirements forWagner-Peyser, VETS, TAA & WIA Programs

10. Roll-Out Of Common Measures Common measure outcomes will be collected through revised reports Common measure definitions replace existing program-specific performance measure definitions where possible Efficiency measures to be computed at Federal level only

11. Wagner-PeyserPerformance Measures • Adult Common Performance Measures Job seeker entered employment rate Adult entered employment rate Job seeker employment retention at six months Adult employment retention rate Adult earnings increase Job seeker customer satisfaction score Employer customer satisfaction score Wagner-Peyser and VETS Program MeasuresRoll-Out Of Common Measures

12. Adult earnings increase Adult earnings increase Wagner-Peyser and VETS Program MeasuresRoll-Out Of Common Measures Wagner-PeyserPerformance Measures • Adult Common Performance Measures Job seeker entered employment rate Adult entered employment rate Adult entered employment rate Job seeker employment retention at six months Adult employment retention rate Adult employment retention rate Job seeker customer satisfaction score PY05 will be the baseline year for these measures. Goals will be negotiated beginning with PY06. Employer customer satisfaction score

13. TAA Performance Measures • Adult Common Performance Measures Entered employment rate Adult entered employment rate Employment retention rate at six months Adult employment retention rate Adult earnings increase Earnings replacement rate TAA Performance MeasuresRoll-Out Of Common Measures

14. TAA Performance MeasuresRoll-Out Of Common Measures TAAPerformance Measures • Adult Common Performance Measures Adult entered employment rate Entered employment rate Adult entered employment rate Employment retention rate at six months Adult employment retention rate Adult employment retention rate Adult earnings increase Earnings replacement rate Adult earnings increase States are responsible for achieving National Goals for the Trade program.

15. WIA AdultPerformance Measures • Adult Common Performance Measures Entered employment rate Adult entered employment rate Employment retention rate at six months Adult employment retention rate Adult earnings increase Earnings gain Employment and credential rate WIA Adult Program MeasuresRoll-Out Of Common Measures

16. WIA Adult Program Measures Roll-Out Of Common Measures WIA AdultPerformance Measures • Adult Common Performance Measures Adult entered employment rate Entered employment rate Adult entered employment rate Employment retention rate at six months Adult employment retention rate Adult employment retention rate Adult earnings increase Adult earnings increase Earnings gain Employment and credential rate Note: States have negotiated performance goals for entered employment, retention, earnings gain and credential for PY05 and PY06.

17. WIA Dislocated WorkerPerformance Measures • Adult Common Performance Measures Entered employment rate Adult entered employment rate Employment retention rate at six months Adult employment retention rate Adult earnings increase Earnings replacement Employment and credential rate WIA Dislocated Worker Program MeasuresRoll-Out Of Common Measures

18. WIA Dislocated Worker Program MeasuresRoll-Out Of Common Measures WIA Dislocated WorkerPerformance Measures • Adult Common Performance Measures Adult entered employment rate Entered employment rate Adult entered employment rate Employment retention rate at six months Adult employment retention rate Adult employment retention rate Adult earnings increase Adult earnings increase Earnings replacement Employment and credential rate Note: States have negotiated performance goals for entered employment, retention, earnings gain and credential measures for PY05 and PY06.

19. WIA Youth Program MeasuresRoll-Out Of Common Measures Youth Common Performance Measures • Placement in employment or education/ advanced training (PY05) • Attainment of a degree or certificate (PY05) • Literacy and numeracy gains (PY05/06) States have not negotiated performance goals for the youth common measures, however states will be expected to begin reporting on these measures in PY05 and PY06. States have negotiated performance goals for WIA programs according to statutory measures for PY 2005 and PY 2006 Younger Youth Skill attainment rate Diploma attainment rate Retention rate Older Youth Entered employment Employment retention Earnings change Employment and credential rate

20. Revised Reporting Requirements • ETA received emergency approval in April 2005 to incorporate common measures into the WIA, W-P, VETS, and TAA program reporting systems • Remember, where possible the definitions for common performance measures are replacing “old” definitions • These changes are effective on July 1, 2005 for WIA W-P and VETS programs, and October 1, 2005, for the TAA program • The Department published three 60-day public comment Information Collection Requests in the Federal Register as part of the process to obtain full approval of these revisions. The public comment period ends September 12, 2005. • Comments should be sent to: Dr. Esther R. Johnson, Administrator Office of Performance and Technology Department of Labor/Employment and Training Administration 200 Constitution Avenue NW Room S5206 Washington, DC 20210 • Electronic copies should be sent to ETAperforms@dol.gov

21. Revised Reporting Requirements What the revised reporting accomplishes • Facilitates the collection and reporting of information, including the common performance measures • Where possible, standardizes reporting elements across programs • Continues to communicate program performance results and holds states accountable for federal funds

22. Revisions to Reporting Requirements WIA Performance Reporting System • Quarterly Report • Collects aggregate counts of self-service participants, the number of participants in training, and youth (by age and school status) • Collects current WIA youth measures plus common measures • Separate reporting of common measures for Adults, Dislocated Workers, and NEG participants • Rolling four-quarter reporting methodology to align with W-P employment services Reporting System • Eliminated customer satisfaction reporting on a quarterly basis • Annual Report • Collects similar aggregate counts of participants and exiters as the Quarterly Report • New report table to collect youth common measures • WIASRD • Collects additional data on participant services and outcomes to calculate common measures • States do not submit WIASRD files on participants who receive only self-services and informational activities • States must report co-enrollment of individuals in Wagner-Peyser or TAA programs

23. Revisions to Reporting Requirements Wagner-Peyser Reporting System (W-P & VETS) • Aligns performance reporting cohorts with WIA reporting cohorts • Eliminates all customer satisfaction reporting • Collects aggregate counts on participants, exiters, and those receiving workforce information services • Collects aggregate outcomes for the adult common measures • Collects aggregate data on services to Homeless Veterans (VETS 200 C report only) • Eliminates reporting on each category of veteran (e.g., campaign, disabled) by age grouping Trade Act Participant Report (TAPR) • Collects data elements to calculate common measures • Collects information on participant services as authorized under the TAA Reform Act of 2002

24. Important: Reporting Reminder!! • Reporting for WIA, W-P, and VETS for PY 2004 (which ended June 30, 2005) has not changed and does not reflect common performance measures • No change to 4th quarter ES 9002 or VETS 200 reports, due in August • No change to 4th quarter WIA 9090, due in August, or the WIA Annual or WIASRD, due in October • Reporting for TAA for FY 2005 (which will end September 30, 2005) has not changed and does not reflect common performance measures • No change to 4th quarter TAPR, due in November

25. Important: Data Validation Update • ETA is upgrading data validation and reporting software to reflect changes in the WIA reporting system • Revised import file format will be available in early August 2005 • Based on revised WIASRD specifications • Modified edit checks/coding procedures • Conversion rules from old-to-new WIASRD format • Projected release of revised WIA data validation and reporting software is Mid-October 2005 • All data validation and reporting documentation will be available on the web at http://www.doleta.gov/performance

26. Key Concepts

27. Program Participation • A participant is a person who is determined eligible to participate in a program and receives a service funded by the program in a physical location • The date of participation is the date of the first service after eligibility determination • Individuals who receive services that are accessed by virtual means outside of the program’s facility need not, but may, be treated as participants ___________________

28. Program Participation • Generally, all participants will be taken into account when computing performance outcomes under the common measures • Two types of participants may be excluded from the performance calculations: • Participants who do not voluntarily disclose valid Social Security numbers • Participants who are excluded from the calculations by law ____________________

29. “Participant” – Wagner-Peyser Act • Wagner-Peyser employment services • Universal Access • No formal determination of eligibility is required • DVOP/LVER services • veterans, eligible persons, transitioning service members who meet eligibility criteria

30. Clarification of “Self-directed Job Search” Exclusion • Application to W-P employment services: • Exclusion is limited to the act of conducting a job search, job bank browsing or modifying job search requirements. • Some self-directed job search activities should be considered reportable service (e.g. building resumes, accessing workforce information, and enrolling in online job search mini courses.)

31. Minimum Data Needed from Self-Service Participants • At a minimum, data needed from self-service participants: • social security number (SSN) • employment status at the point ofparticipation. • Upon request of a customer’s SSN, the customer must also be offered the opportunity to provide Equal Employment/demographic information.

32. Program Exit • Exit occurs when . . . • Participant does not receive a service or partner service for 90 consecutive calendar days; • Participant has no planned gap in service; and • Participant is not scheduled for future services • Date of exit is applied retroactively to the last day of service funded by the program or partner program • Must have the capability to track partner services in order to extend the date of exit • No More Hard Exit

33. Program Exit • Days included in any planned gap in services do not count when determining the exit date • Delay before the beginning of training • Health condition or providing care for a family member with a health condition • Temporary move from the area • All gaps in service must be documented

34. “Exiter” – Wagner-Peyser Act Significant change from former Wagner-Peyser performance measures, which calculated performance outcomes based on registration quarter.

35. Exclusions from all common measures: Persons institutionalized (exceptions are detained Responsible Reintegration of Youthful Offenders and disabled individuals residing in institutions) Persons unable to continue due to health/medical or family care reasons Deceased Reservists called to active duty Relocated to a mandated residential program (youth only) To be excluded from performance calculations, the condition(s) must be expected to last 90 days or more Global Exclusions

36. Data Sources

37. Outcome Data Collection • Unemployment Insurance wage records • Includes state wage records and additional sources such as WRIS and FEDES • The first source for employment and retention measures • The only source for earnings increase • Supplemental data may be used to determine employment and retention

38. Outcome Data Collection • Administrative records are the source for outcomes that are not employment- based • Administrative data may be obtained through automated data exchanges • Administrative record data (except automated data exchanges) and data collection methods must be documented and subject to audit

39. Calculating OutcomesAdult Measures

40. Summary of Major Policy Changes

41. Of those not employed at participation: Number employed in quarter after exit Number of exiters Computing Outcomes AdultEntered Employment • Exclude adults employed at the time of participation (including those individuals who have taken a job after being laid-off or dislocated) • Include adults employed at the time of participation who are on layoff notice • Wage records or supplemental data may be used to determine employment in the quarter after exit • Supplemental data must be documented • Employment at participation is based on information from the individual – not wage records

42. Excluded YES NO Numerator YES Employed in quarter after exit? YES Denominator NO AdultEntered Employment Employed at participation?

43. Of those employed in the quarter after exit: Number employed in both the 2nd and 3rd quarters after exit Number of exiters Computing Outcomes AdultEmployment Retention • Based on those employed in quarter after exit • Wage records and supplemental data are acceptable data sources • Supplemental data must be documented • Employment in the 1st, 2nd and 3rd quarters after exit does not have to be with the same employer • Must be employed in both the 2ndand 3rd quarters after exit to be retained

44. Excluded NO YES Numerator YES Employed in 2nd and 3rd qtr. after exit? YES Denominator NO AdultEmployment Retention Employed in quarter after exit?

45. Of those employed in the quarter after exit: - Earnings in 2nd and 3rd quarter after exit Earnings in 2nd and 3rd quarter prior to participation Number exiters Computing Outcomes AdultEarnings Increase • Only wage records may be used for determining pre-program and post- program earnings • Includes same population as the adult retention measure • Likely to be negative for dislocated workers • False, erroneous and misleading earnings data may be excluded by the grantee

46. YES YES NO AdultEarnings Increase NO Excluded Employed in qtr. after exit? YES YES Supplemental data used in Q1? NO Supplemental data used in Q3? Included

47. Calculating OutcomesYouthMeasures

48. Youth Measures Requirements Reporting for PY 05: • 7 current WIA youth measures (4 OY and 3 YY) – negotiate on these for accountability • First 2 youth common measures (placement and certificate) • Literacy/Numeracy common measure encouraged, but not required (required in PY 06) • Literacy/Numeracy only include new youth enrolling after July 1 of the year in which the State implements it • No negotiation on common measures in PY 05

49. Youth and Lifelong Learning Common Measures • Will include all youth, no distinction between younger/older 1) Placement in Employment, Education, or Training 2) Attainment of a Degree/Certificate 3) Literacy/Numeracy Gains

50. Summary of Major Policy Changes