DATA VALIDATION

1. DATA VALIDATION NORTH CAROLINA’S EXPERIENCE

2. Program Year 2002 Data Validation • We contracted with our former Finance Director (retired) to complete PY2002 data validation • There were no electronic records provided to the performance or MIS staff after the project was completed • No records were sent to DOL • A sparse state report was provided and was disseminated to the Local Areas NC Division of Employment & Training

3. Program Year 2003 Data Validation • Challenges: • No raw data from prior year • No state policies or procedures • No practical experience • No training (beyond DOL workshops) • No staff NC Division of Employment & Training

4. Program Year 2003 Data Validation • First Steps: • Convene a Data Validation Committee to establish policies, procedures and training materials (August 2004) • Committee Membership: • DET Staff • MIS Super Users • Case Managers NC Division of Employment & Training

5. Data Validation Committee • DET Staff • MIS Director, Planners (2), Field Staff (Monitor), Technical Writer, Performance Manager • MIS Super Users (7) • MIS Super Users are MIS and performance specialists in each Local Area • Case Managers (6) • For adult, dislocated worker and youth programs NC Division of Employment & Training

6. MIS Super Users • MIS Super Users are specialists in each Local Area responsible for: • Attaining a high-level proficiency with the WorkforcePlus™ system and functioning as the Local Area’s expert user of the participant MIS • Attending and actively participating in all MIS User’s Group meetings • Training Local Area end users in the proper use of WorkforcePlus™ • Providing ongoing technical assistance and remedial training on WorkforcePlus™ issues NC Division of Employment & Training

7. MIS Super Users • Providing feedback to DET concerning MIS issues • Conducting continual review of Local Area data, monitoring the quality of data entry and the proficiency of users and promptly addressing identified problem areas • Understanding the relationship between the Local Area’s performance goals, program design, and program outcomes and ensuring the proper recording of program outcomes to maximize performance levels • Attaining proficiency in extracting performance and management reports from WorkforcePlus™ NC Division of Employment & Training

8. Data Validation Committee • First Steps: • Train DV Committee members • Agree on the WorkforcePlus™ source documentation that will be acceptable for NC data validation and amend DOL source list to be NC specific • Determine what state policies and procedures need to be put in place specific to data validation NC Division of Employment & Training

9. Data Validation Committee • Source Documentation • Developed a revised DV source document chart, deleting references in DOL’s chart that were not applicable to NC and adding specific WorkforcePlus™ (MIS) references for data entry screens and printouts that could be used for DV documentation • A WorkforcePlus™ ‘definitions and instructions’ document that had been drafted by our Super Users group was adopted as state policy NC Division of Employment & Training

10. Data Validation Committee • State Policies and Procedures • Three Local Area Issuances were developed and released in December 2004 • Entering Activities and Dates for First WIA Activity • Clarified service requirements for initial WIA registration in NC • WIA Application Time Limit • WIA Data Element Validation and Participant File Structure • Formalized a case file structure and content requirements across all Local Areas NC Division of Employment & Training

11. Data Validation Policies • Entering Activities and Dates for First WIA Activity (Local Area Issuance 2004-15) • Established specific employment activities as the initial service to be provided for every WIA participant • Provides statewide consistency in setting the WIA registration date • The “start date” must equal the “creation date” in WorkforcePlus™ for the initial activities • For Adults and Dislocated Workers, the initial enrollment must be “case management” • For Youth, the initial enrollment must be “comprehensive guidance and counseling” NC Division of Employment & Training

12. Data Validation Policies • WIA Application Time Limit (Local Area Issuance 2004-15) • Reinforces that individuals must be eligible at the time of enrollment into WIA, not just at application • Resolves concerns we had regarding a lengthy time lapse between eligibility determination and enrollment into the WIA program. • Requires a new application to be completed if more than 60 days have lapsed since the application date/eligibility date NC Division of Employment & Training

13. Data Validation Policies • WIA Data Element Validation and Participant File Structure (Local Areas Issuance 2004-17) • Intended to improve compliance with data validation requirements • Requires Local Areas to develop a formal file structure policy (or amend an existing file structure policy) for all participant files • Requires each file to have a file checklist at the front of the file indicating where supporting documentation can be found in the file • Requires each DV data element to be labeled within the file NC Division of Employment & Training

14. Data Validation Policies • File Structure (continued) • Each DV data element has a ‘field code’ consisting of a cohort, category, and DOL’s reference number • Cohorts: • A (adult), DW (dislocated worker), OY (older youth) or YY (younger youth) • Category: • E – Eligibility/Intake/Application • A – Employment Activities • XP – Exit and Post-Program Activities • DOL Reference Number: corresponds to the elements’ reference numbers in the DOL WIA Data Validation Handbook NC Division of Employment & Training

15. Data Validation Policies • File Structure (continued) • Field codes must be placed by each data elements in the file checklist(s) and the field codes added to the source documentation in the files • The policy also requires that certain documents/screen prints also be placed in each file: • Signed and dated application • 1st activity enrollment screen • Case profiles • Case/activity notes • Supplemental data screens (where applicable) • Leave Program screen • Employment Plan screen NC Division of Employment & Training

16. Data Validation Policies • File Structure (continued) • All files for participants active in WIA as of July 1, 2004 are required to be in the file structure format and contain all source documentation for data validation • Some Local Areas had file structure ‘parties’ where the case managers met and assembled participants’ files in the new structure • Feedback from the case managers has been overwhelmingly positive • Most Local Areas use sectioned file folders, although it is not required by the state NC Division of Employment & Training

17. Data Validation Committee • Training • Developed a data validation training outline and materials • Created two training manuals • For local case managers • For state data validation staff • Initial DV training for MIS Super Users and select case managers was held over two days in February 2005 • Super Users in each Local Area were responsible for training their case managers • Quarterly MIS Users Groups meetings provide a forum for continued dialogue between the state and Local Area staff NC Division of Employment & Training

18. Data Validation Committee • Additional Training • Case notes may be used in North Carolina to validate select data elements • During Program Year 2003 data validation, it was found that some of the case notes in the files were marginal, at best • An RFP was released to solicit bids for the provision of training for data validation specific “Writing Effective Case Notes” • We provided 16 training sessions and trained 643 case managers at the Workforce Development Training center • Local Areas also arranged for additional “Writing Effective Case Notes” training NC Division of Employment & Training

19. Data Validation Committee • Additional Training, continued • There are plans to offer ongoing data validation training to end users and case managers (after the revised reporting requirements are released by DOL) • Data validation training will probably be combined with common measures training • One day sessions with half of the day devoted to common measures and the other half devoted to data validation • Case Managers tend to want training specific to the individuals they serve (i.e., youth separate from adult and dislocated workers) NC Division of Employment & Training

20. Staffing • DET has used 4 time-limited/temporary staff to complete data validation for the last 2 years • Pay is between $11.18 - $16.62 per hour • No fringe benefits • Receive reimbursement for mileage, meals and lodging • There are two data validation teams – East and West • Both teams go as a group to Local Areas with more than 100 files in the sample to minimize disruption to the Local Area staff NC Division of Employment & Training

21. Staff Training • The DV staff receive a week of training on WIA programs, the NC participant database, and federal and state DV policies and procedures • 25 participant files are ‘validated’ in house to allow staff to become familiar with the information they will be seeing in participant file folders and to practice validating the elements • Once the files are validated, additional instruction and clarification are provided where needed NC Division of Employment & Training

22. Staff Training • All validators and the manager go to the first Local Area • A select group of files from each cohort at the Local Area (adult, dislocated worker, older youth, and younger youth) is reviewed by all 5 individuals • DV results for all of these select files are compared and discussed • The file(s) are re-reviewed by the group to determine if a questioned data element is supported by source documentation NC Division of Employment & Training

23. Actual Data Validation • North Carolina has an official sample and an unofficial sample • Every Local Area has a data validation visit • The data validation schedule is arranged by the manager • The validators are required to call Local Area contacts the week prior to the visit to confirm and to inform them when they will arrive (day and time) • Validators hold entrance and exit conferences with the Local Area staff • Validators complete hard copy worksheets at the Local Area. • For every data element that is failed, the validator has to document why the element was failed NC Division of Employment & Training

24. Actual Data Validation • Upon return to the state office: • Each validator writes an individual report about his/her general findings at the Local Area and summarizes recommendations for the Local Area • Validators enter DV findings into the DOL software • Validators create a comment sheet for each Local Area containing the individuals’ names, data elements failed and why the elements failed (by cohort) • State staff compiles reports for each Local Area • Numerical report • Narrative report (including comments on all failed data elements NC Division of Employment & Training

25. Actual Data Validation • After the completion of all data validation for the state • Individual reports are completed for each Local Area • DOL “WIA Data Element Validation Summary and Analytical Report” • DET Report • Narrative report (including comments on all failed data elements NC Division of Employment & Training

26. Example 1 of Local Area Report LAXXX - Dislocated Worker (DOL Report) NC Division of Employment & Training

27. Example 2 of Local Area Report LAXXX - Dislocated Worker (DET Report) NC Division of Employment & Training

28. Reports • DOL’s DV report may tell them what they want to know, but it is confusing to the Local Areas • How many of you know (without looking) what ‘Overall Error Rate’ and ‘Reported Data Error Rate’ mean? • How many of your Local Areas’ staff understand the meaning of the error rates? • The DET DV report puts information in perspective for the Local Area: how many elements were selected for validation and how many passed or failed? NC Division of Employment & Training

29. What Have We Learned? • Local Areas want training, structure and guidance • For the most part, case managers like the file structure requirements (once they got their files in order with all requisite information) • Program Year 2004 data looks better than prior years’ data • We anticipate that as the exit date range for DV catches up with our DV policies, our data validation performance will improve significantly. NC Division of Employment & Training

30. NC VALIDATION EXPERIENCE QUESTIONS? NC Division of Employment & Training

31. Contacts and References • Martha Cranford NC Division of Employment & Training 313 Chapanoke Road, Suite 120 4316 Mail Service Center Raleigh, NC 27699-4316 Phone: 919-329-5290 Fax: 919-661-4770 mcranford@nccommerce.com • NC DET State Policies www.ncdet.com • Case Notes Training Mary Ann Lawrence (Center for Workforce Learning) MALaw528@aol.com www.mylearningchain.com NC Division of Employment & Training