WIA PERFORMANCE MEASURES AND STANDARDS:
Download
1 / 28

Christopher T. King Ray Marshall Center for the Study of Human Resources - PowerPoint PPT Presentation


  • 82 Views
  • Uploaded on

WIA PERFORMANCE MEASURES AND STANDARDS: The WIASRD, Common Measures and Standards Negotiation Challenges. Christopher T. King Ray Marshall Center for the Study of Human Resources University of Texas, Austin [email protected] 512/471-2186 David W. Stevens The Jacob France Institute

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Christopher T. King Ray Marshall Center for the Study of Human Resources' - zaltana-torres


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
WIA PERFORMANCE MEASURES AND STANDARDS:The WIASRD, Common Measures andStandards Negotiation Challenges

Christopher T. King

Ray Marshall Center for the Study of Human Resources

University of Texas, Austin

[email protected]

512/471-2186

David W. Stevens

The Jacob France Institute

University of Baltimore

[email protected]

410/837-4729

April 22, 2003


Briefing topics
BRIEFING TOPICS

1. Highlights from PY 2000 program outcome information in the

WIASRD files from the seven ADARE Project states,

focusing on the quality of the data elements.

2. Negotiated, actual and actual minus negotiated difference in

PY 2000 performance data for the seven ADARE Project states.

3. Observations about the proposed common measures.

4. WIA performance standards negotiation challenges and

opportunities (including pros and cons of regression modeling).

5. Other challenges that will follow reauthorization.


Employed in quarter after exit quarter
EMPLOYED IN QUARTER AFTER EXIT QUARTER

The data element code choices are: yes, no and not yet available

Georgia, Illinois and Missouri did not use the not yet available code.

The four ADARE Project states that used the not yet availablecode

used it the following percent of the time:

  • Florida 44 percent

  • Maryland 73 percent

  • Texas 23 percent

  • Washington 50 percent


Use and source of supplemental data
USE AND SOURCE OF SUPPLEMENTAL DATA

The data element code choices are: used case management files and

record sharing/matching

  • Florida, Missouri and Washington did not report anyuse of

    supplemental data sources.

  • Georgia reported only three instances of supplemental data use.

  • Texas reported using supplemental data one percent of the time.

  • Illinois and Maryland reported using supplemental data three

    percent of the time.


Occupational code of any job held since exit
OCCUPATIONAL CODEof any job held since exit

This information is to be reported if the individual is reported as employed

in the quarter after exit.

The information can be based on information derived from case management

files, follow-up services or other sources.

It is not necessary to wait until information on employed in quarter after exit

is available.

  • Florida, Georgia and Maryland used only the nine-digit DOT code.

  • Illinois and Texas used only the five-digit OES code.

  • Washington used both the DOT and OES coding taxonomies.

  • Missouri used the five-digit or six-digit O*Net98 code.


Entered training related employment
ENTERED TRAINING RELATED EMPLOYMENT

Two-thirds of the yes or no entries for this data element were recorded as

a yes.

The range of affirmative entries was from a low of 29 percent for

Maryland to a high of 94 percent for Florida.

  • The reported method used by Florida, Maryland, Texas and Washington

    to determine training related employment was ‘other appropriate method’.

  • The reported method used most often by Georgia, Illinois and Missouri

    was ‘a comparison of the occupational codes of the training activity

    and the job’, but each of these three states also used ‘a comparison of

    the industry of employment with the occupation of training using

    an appropriate crosswalk’.


Entered nontraditional employment
ENTERED NONTRADITIONAL EMPLOYMENT

The nontraditional employment designation can be based on either

local or national data.

  • Six percent of the yes or no entries for this data element were reported

    as a yes.

  • The range of affirmative entries among the seven ADARE Project states

    was from a low of one percent to a high of fifteen percent.

  • Texas did not report yes or no entries for this data element.


Type of recognized educational occupational certificate credential diploma or degree attained
TYPE OF RECOGNIZED EDUCATIONAL/OCCUPATIONALCERTIFICATE, CREDENTIAL, DIPLOMA OR DEGREEATTAINED

  • Seven codes are provided. States and localities have flexibility in

    choosing the methods used to collect data documenting this data

    element.

  • Each of the seven ADARE Project states reported award of some

    credentials in each of the six type of credential categories.


Py 2000 core measures of performance seven adare project states
PY 2000 CORE MEASURES OF PERFORMANCESEVEN ADARE PROJECT STATES

The four Adult and Dislocated Worker performance measures are covered.

  • Entered employment rate.

  • Employment and credential rate.

  • Retention rate.

  • Earnings change

    Each of the four charts that follow ‘flies in’ PY 2000 negotiated, actual

    and actual minus negotiated performance measure values for the

    seven ADARE Project states.


Questions to ask when looking at the charts that follow
QUESTIONS TO ASK WHEN LOOKING AT THECHARTS THAT FOLLOW

  • Do I know enough about the criteria for specifying each negotiated

    performance measure value to interpret the observed differences

    in these negotiated values among the seven ADARE Project states?

  • Do I know enough about the data sources that were used to calculate

    the actual performance measure values to interpret the actual minus

    negotiateddifferences in these values among the seven ADARE Project

    states?

  • What management and/or policy conclusions can I reach based on

    my answers to the previous two questions?

  • Can I be confident in making incentive awards and imposing sanctions

    based on actual minus negotiated value differences?



Program year 2000 july 2000 june 2001 employment and credential rate
Program Year 2000 (July 2000-June 2001): Employment And Credential Rate




Revisiting the questions asked having looked at the charts
REVISITING THE QUESTIONS ASKEDHAVING LOOKED AT THECHARTS

  • Do I know enough about the criteria for specifying each negotiated

    performance measure value to interpret the observed differences

    in these negotiated values among the seven ADARE Project states?

  • Do I know enough about the data sources that were used to calculate

    the actual performance measure values to interpret the actual minus

    negotiateddifferences in these values among the seven ADARE Project

    states?

  • What management and/or policy conclusions can I reach based on

    my answers to the previous two questions?

  • Can I be confident in making incentive awards and imposing sanctions

    based on actual minus negotiated value differences?


Common measure issues performance measure quality
COMMON MEASURE ISSUESPerformance Measure Quality

ENTERED EMPLOYMENT RATE

  • Registration date

  • Employed or not employed at registration

  • Exit date

  • Entered employment by the end of the first quarter after exit

    ISSUES

  • Staff decision whether and when to register a customer

  • Quality of ‘employed or not employed at registration’ data element

  • Unintended consequences of this measure

  • Staff decision when to enter or allow automatic entry of exit date

  • Use of supplemental data sources


Common measure issues performance measure quality1
COMMON MEASURE ISSUESPerformance Measure Quality

EMPLOYMENT RETENTION RATE

  • Employed first quarter after exit (regardless of employment status

    at time of registration)

  • Employed second quarter after exit

  • Employed third quarter after exit

    ISSUES

  • Stakeholder interest in this measure

  • Drill-down questions that will be asked

  • Use of supplemental data sources

  • Timeliness of availability for intended uses


Common measure issues performance measure quality2
COMMON MEASURE ISSUESPerformance Measure Quality

EARNINGS INCREASE

  • Earnings in second quarter prior to registration

  • Employed in first quarter after exit

  • Earnings in first quarter after exit

  • Earnings in third quarter after exit

    ISSUES

  • Stakeholder interest in this measure

  • Drill-down questions that will be asked

  • Number of ‘pays’ in each reference quarter

  • Use of supplemental data sources

  • Timeliness of availability for intended uses


Common measure issues performance measure quality3
COMMON MEASURE ISSUESPerformance Measure Quality

EFFICIENCY

  • The dollar amount specification to serve as the numerator

  • The number of participants figure to serve as the denominator

    ISSUES

  • Stakeholder interest in this measure

  • Drill-down questions that will be asked

  • Quality of data elements

  • Unintended consequences


Common measure issues performance measure quality4
COMMON MEASURE ISSUESPerformance Measure Quality

PLACEMENT IN EMPLOYMENT OR EDUCATION

  • Registration date

  • Enrolled in secondary education at registration

  • Exit date

  • Not enrolled in post-secondary education at registration

  • Not employed at registration

  • Enrolled in secondary education at exit

  • Employed in first quarter after exit

  • In military service in first quarter after exit

  • Enrolled in post-secondary education in first quarter after exit

  • Enrolled in advanced training/occupational skills training in

    first quarter after exit

    CONTINUED……


Common measure issues performance measure quality5
COMMON MEASURE ISSUESPerformance Measure Quality

PLACEMENT IN EMPLOYMENT OR EDUCATION

CONTINUED….

ISSUES

  • Stakeholder interest in this measure

  • Drill-down questions that will be asked

  • Quality/uniformity of data definitions and sources

  • Cost of data collection

  • Access to education records

  • Timeliness of data availability for intended uses

  • Unintended consequence—proliferation of

    credentials


Common measure issues performance measure quality6
COMMON MEASURE ISSUESPerformance Measure Quality

ATTAINMENT OF A DEGREE OR CERTIFICATE

  • Registration date

  • Enrolled in education

  • Exit date

  • Attained a diploma, GED, or certificate by the end of the third

    quarter after exit

    ISSUES

  • Stakeholder interest in this measure

  • Drill-down questions that will be asked

  • Access to education records

  • Quality/uniformity of data definitions and sources

  • Timeliness of data availability

  • Unintended consequences


Common measure issues performance measure quality7
COMMON MEASURE ISSUESPerformance Measure Quality

LITERACY OR NUMERACY GAINS

?


Common measure issues performance measure quality8
COMMON MEASURE ISSUESPerformance Measure Quality

FIVE ISSUES ARE OF PARTICULAR IMPORTANCE AND CONCERN:

  • The accuracy and probable unintended consequences associated

    with the employed or not employed at registration data element

  • The integrity and value-added of supplemental data use

  • Selection of denominator and numerator definitions for the

    proposed efficiency measure

  • The complexity and value-added of the placement in employment

    or education measure

  • Expected unintended consequences associated with the

    attainment of a degree of certificate measure


Performance standard issues
PERFORMANCE STANDARD ISSUES

THREE TOPICS ARE OF PARTICULAR IMPORTANCE:

State and Local Workforce Area Benchmarking

  • The Census Bureau LEHD Program as a potential source of local

    demographic and economic activity information for discretionary

    use in negotiation of state and local performance standards

  • Benchmarking of own performance over time

  • Benchmarking against other ‘similar’ states or Local Workforce Areas

  • Return to regression modeling? Pros and cons

    CONTINUED….


Performance standard issues1
PERFORMANCE STANDARD ISSUES

Challenges Associated with Pursuing Continuous Improvement

  • Integrity of state and local management information systems

    over time

  • Continuity of data source availability and content over time

  • Expected unintended consequences


Performance standard issues2
PERFORMANCE STANDARD ISSUES

Vulnerability to Unintended State and Local Actions

  • Discretionary opportunities to define selection in criteria,

    assignment to service components criteria (including whether

    and when to use partner services) and timing of exit criteria

  • Investment in staff development can reduce the frequency

    of some of the unwanted behaviors that will otherwise

    follow introduction of the common measures


Other challenges
OTHER CHALLENGES

  • Occupations in demand

  • Required registration of some customers

  • Stakeholder interest in number of customers served


ad