1 / 24

Evaluation of ESF intervention for individual employability

Evaluation of ESF intervention for individual employability. Evaluation of Programmes and Policies for Human Resources Development Area Rome, Corso D’Italia, 33. 1. National Placement Working Group’s Activities. Contents

myron
Download Presentation

Evaluation of ESF intervention for individual employability

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation of ESF intervention for individual employability Evaluation of Programmes and Policies for Human Resources Development Area Rome, Corso D’Italia, 33

  2. 1. National Placement Working Group’s Activities Contents In section 1 we look through the activities of the ESF National Placement Working Group (hereafter NPG), over a period of more than 10 years. The activity of NPG lies on the evalutation of training effects on individual employability. Isfol coordinates the NPG, which members are the regional/national Esf Managing Authorities and their designed referents. Evaluation of Programmes and Policies for Human Resources Development Area Rome, Corso D’Italia, 33

  3. In brief • In Italy, training policies are under the scope of regional’s competences (jurisdiction). • Till earlier EU programming period (2008), training projects were mainly co-funded by ESF. • In this current programming period, 2007-2013, the ESF is still an important financial instrument on training policies but, in particular for regions of Ob.2, it has reduced its financial weight

  4. Evaluation of ESF intervention for individual employability • A coordination over regional evalutation activieties is needed to allow getting a consistent national synthesis 2. More than it’s used to be, at present evaluation activity should be “tied” to the policy whatever it is the financial source.

  5. The PNG’s activities: chronology 5. Exploitingadministrative database forevaluationpurposes 4. Robustnessof the surveys (from a statisticalpointofview) 3. Modelsfor the impact evaluation (net effects) 2. Harmonizingregional’s surveys: the questionnaire 1. Data on training projects and beneficiaries: the applicationformforpre-enrolment

  6. 1. Data on training projects and beneficiaries: the application form for pre-enrolment “ […]The best solution to the evaluation problem lies in improving the quality of the data on which evaluations are conducted and not in the development of formal econometric methods to circumvent inadequate data.”(Heckman, Lalonde e Smith, 1999)

  7. 1. Data on training projects and beneficiaries: the application form for pre-enrolment • Isfolcoordinatestwonationalworkinggroupthat are stronglytied up eachother National Monitoring Working Group (NMG) Placement National Working Group (NPG) 1. Training projects and beneficiaries database • 2. Harmonizing regional’s surveys: the questionnaire • 3. Models for the impact evaluation (net effects) • 4. Robustness of the surveys (from a statistical point of view) • 5 ….

  8. 2. Harmonizing regional’s surveys: the questionnaire PNG’s earlier activities went to a shared “core“ questionnaire: a set of common questions that each regional questionnaire should incorporate. This, in turn, entails: • a national synthesis of the main results; • the definition of a set of common result indicators such as the “gross employment rate of training interventions”

  9. 2. Harmonizing regional’s surveys: the questionnaire • To reach these objectivies it’s necessary to settle the same “time-lag”, beginnig from the conclusion of the training course (T), which the employability of the beneficiaries will be referred at. Choice had been twelve months after conclusion of training (T+12). • Once defined the reference time “T+12” one should take care of the interview’s timing. For a trade-off rises up: • As far as the lag becomes greater, the measurement errors increase for individuals could not have memory of their own condition at time T+12 (lower reliability of answer); • As far as the lag increases, important data, such as telephone numbers, becomes unusable for a greater number of beneficiaries; • As far as the lag becomes greater, it’s going to be relevant getting more informations about the currently condition of beneficiary.

  10. 3. Models for the impact evaluation (net effects) • Here we distinguish: • Impact evaluation on job creation (net job) • Impact evaluation for individual employability (net employment rate or net effects) • Working document n°6(CE, 2007) “Measuring structural funds employment effects” (1)NET JOBS = GROSS JOBS X (1– DEADWEIGHT) X (1 – DISPLACEMENT AND SUBSTITUTION)X (1 + SUPPLIER MULTIPLIER + INCOME MULTIPLIER) Net jobs represent the overall jobs created as the sum of both direct (deadweight) and indirect (displacement, substitution, supplier and income multiplier) effects. • Our analysis on impact evaluation is restricted to the direct effect: (2)NET EMPLOYMENT RATE = GROSS EMPLOYMENT RATE X (1– DEADWEIGHT) Within econometric models, expression (2) is measured as a difference between the averages employment rates fitted for people treated (beneficiaries of training intervention) and for people belonged to the control group (not treated) which approximate the counterfactual situation. Clearly these models rely upon both to the consistency of the control group and, from (1), to the assumption that indirect effects are disregarded.

  11. 3. Models for the impact evaluation (net effects) Impact evaluation of training policies (but the same could be said for other active labor market policies) on individual employability deals with demands like this: What’s the effect (sign and measure) on the probability to get a job produced by the partecipation to a training intervention? To provide an answer to this/these question(s), the researchers have to adequately approximate the counterfactual situation (not osservable) = What if

  12. 3. Models for the impact evaluation (net effects) Remark: • Evaluation on employability effects should not be constrained to the availability of an impact evaluation • Surveys on people trained could satisfy several purposes related to the employability scope which don’t need of counterfactual

  13. 3. Models for the impact evaluation (net effects) In the last programming period (2000-2006) Isfol has realised in Objective 3 an impact evaluation (effectiveness of training interventions on the probability to find a job). The study: • was realised with data come from regional surveys. Those surveys were conducted during the years 2003-2005 on training projects completed between the years 2001-2003 • was restricted to a subset of regions, interventions and beneficiaries, because of the informations available needed for the econometric model specification (we refer to the instrumental variables used to tackle selction bias problem); • utilised Istat Labor Force Survey panel data (twelve months) to build up an external control group for the counterfactual estimates.

  14. 3. Models for the impact evaluation (net effects) Difficulty to have an internal control group (or consistent/effective external control group) and the lack of effective instrumental variables to face the selection bias problem, are among the main causes that, in the past, precluded to conduct impact evaluation. Once more, wehavetoremark the importanceof ex-ante data availability (projects and beneficiaries)

  15. 4. Robustnessof the surveys (from a statisticalpointofview) • Often in the previous slides we’ve utilized the word “survey”. We’ve punctualized that each region chooses if, when and how realise surveys on training beneficiaries. The NPG guidilines keep attention about the statistical foundations the surveys ought to meet, in each step: • Planning. • Sample design (in case of sample survey). • Interviews. • Results analysis. • Planning • Drawing the evaluation purposes. This yields effects on both sample design and questionnarie structure. For example, a preliminary study should highlights the informations not available that must be “recovered” through the questionnarie during the interview.

  16. 4. Robustnessof the surveys (from a statisticalpointofview) 2. Sample design Keeping attention over all the aspects linked to the sample strategy: sample numerosity, sampling errors, sample estimator, and so on. • 3. Interviews • Doing actions able to “prevent” or “minimize” the non-sampling errors (measurement errors due to missing data). • Take care of beneficiaries list (survey frame), in particular telephone number and postal/home address; • Send to the beneficiaries (at least the persons sampled) a letter, which gives them information of the survey (goals, assurances as regard the respect of laws on statistical treatment of personal data, and so on) • Pilot survey to test both the questionnaire and CATI • On going evaluation about the interview process.

  17. 4. Robustnessof the surveys (from a statisticalpointofview) 4. Results analysis Inference • Other methods finalized to detect the individual labor market condition (as an example there are the follow-up questionnaires), appears to be not suitable for evaluation scope: • The structure of questionnaires is not adequate to meet evaluation requirements • Data doesn’t meet reliability criteria.

  18. 5. Exploitingadministrative database forevaluationpurposes • During the last meeting of NPG, discussion themes have been concerned with the opportunity of exploiting administrative database for evaluation goals. The expression “Administrative Placement” have been coined. Database (Labor informative system) entails, for each individual here registered, to get overall informations on job career. • Several points should be stressed: • Exploiting administrative data sources could indeed reduce survey costs; • Administrative database could make it possible to get a reliable and consistent external control group for impact evaluation; • Administrative database could be utilized to update telephone numbers and/or postal addresses.

  19. 5. Exploitingadministrative database forevaluationpurposes • Administrative Placement spread all these positive effects whenever it’s thought together with, not as a substitute of, placement survey. • Further, it must be verified the quantitative and qualitative measure of beneficiaries that are not covered by the administrative database.

  20. 2. Centralised Placement surveys Contents In section 2 we look through the coexistence of regional and centralised placement surveys. The latter are caracterised by a single sample design and the territorial coverage embeds more than one region, possibly they could involve the whole national territory. Here, the coordination role of NPG is even more necessary to avoid useless and harmful overlaps. Exploiting administrative database for evaluation purposes

  21. CentralisedPlacementsurveys • ISFOL has conducted two “placement” surveys during the early ESF programming period. Both refer to regions of Objective 1: • Training interventions for individual employability funded through the ESF Regional Operational Program • Masters and PhDs funded through the National Operational Program “Ricerca Scientifica, sviluppo tecnologico, alta formazione” (MIUR).

  22. CentralisedPlacementsurveys • Centralised surveys came from a sharing process between region involved (ESF Regional Managing Authorities)and Isfol. • Centralised surveys ensure estimates with regional statistical significance • Each region will have returned the microdata of the survey referring to its training interventions

  23. CentralisedPlacementsurveys At present, Isfolisconducting a nationalsurvey on indivualemployabilityeffectsof training interventionscofundedby ESF. National surveyhasbeenplannedwithin the NPG. Regionsprovidedto the Isfolresearchergroup the interventions and beneficiariesdatasets. The surveyconcerns training projectscompletedbetween the years 2008 and 2009, irrespectiveof the programmingperiod (2000-2006 or 2007-2013). Though the national survey likes covering the whole national territory, some specific evaluation topics (impact evaluation) can be confined to a lower number of regions…

  24. Thanks for your attention Toti Enrico Isfol – Evaluation of Programmes and Policies for Human Resources Development Area Tel. +39 06 44590688 e-maile.toti@isfol.it

More Related