European cohesion policy european social fund monitoring and evaluation in 2014 2020
Download
1 / 130

European Cohesion Policy – European Social Fund Monitoring and Evaluation in 2014-2020 - PowerPoint PPT Presentation


  • 146 Views
  • Uploaded on

European Cohesion Policy – European Social Fund Monitoring and Evaluation in 2014-2020. Effie Meletiou Impact Assessment and Evaluation DG Employment, Social Affairs and Inclusion Nicosia, 26 November 2013. Outline. Monitoring: highlights of regulatory requirements

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' European Cohesion Policy – European Social Fund Monitoring and Evaluation in 2014-2020' - mallory-spencer


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
European cohesion policy european social fund monitoring and evaluation in 2014 2020

European Cohesion Policy – European Social Fund Monitoring and Evaluationin 2014-2020

Effie Meletiou

Impact Assessment and Evaluation

DG Employment, Social Affairs and Inclusion

Nicosia, 26 November 2013


Outline
Outline

  • Monitoring: highlights of regulatory requirements

  • Monitoring: regulatory requirements on indicators

  • Monitoring: programme specific indicators

  • Evaluation


Monitoring highlights of regulatory requirements with comments

Monitoring:

Highlights of regulatory requirements

(with comments)


Monitoring committee art 41
Monitoring committee (Art. 41)

  • Set up within 3 months of adopting the OP

  • Single MC can cover more than one programme

  • MC draws up its own rules of procedure with reference to institutional, legal and financial framework of the MS concerned


Composition of mc art 42
Composition of MC(Art. 42)

  • Decided by MS provided that it is composed of relevant MS authorities, IBs and partners referred to in Art. 5.

  • Representatives of these partners shall be delegated through transparent processes.

  • Each MC member may have a voting right

  • The list of members shall be published


Functions of the mc art 43
Functions of the MC (Art. 43)

MC shall meet at least once a year

Review implementation of OP(s)

Shall be informed of progress of achieving targets and milestones in the performance framework and results of qualitative analyses

Shall examine all issues affecting performance of the programme, including the conclusions of the performance review

Shall be consulted on OP modifications and give opinion, if it considers it appropriate

May make observations regarding OP implementation and evaluations, including actions related to reduction of admin burden on beneficiaries

Regional Policy

Cohesion Policy


Functions of the mc art 100
Functions of the MC (Art. 100)

Shall examine:

Issues affecting performance

Implementation of evaluation plan and follow-up given to evaluation findings,

Implementation to communication plan, JAPs & financial instruments

Progress in fulfilling applicable ex ante conditionalities, where not fulfilled at the time of submission of OP or PA

Shall examine and approve:

Evaluation plan (covering one or several OPs) and any modification thereof

Communication plan

Modification of OP

Regional Policy

Cohesion Policy


Important changes in regulations implementing and delegated acts
Important changes in Regulations, Implementing and Delegated Acts

  • Enhanced focus on results

  • Increased importance of monitoring and evaluation

  • Even stronger need for clear intervention logic

  • Close link with Europe 2020 strategy


Clear intervention logic
Clear intervention logic Acts

  • Composed of the hierarchy of programme objectives, actions, expected outputs and results

  • Important for all phases: programme design, implementation, monitoring and evaluation

    • E.g. choice of programme-specific indicators, time-planning of evaluations


Clear intervention logic1
Clear intervention logic Acts

  • Challenges and needs

    • Country-specific recommendations

    • National Reform Programme

    • European semester analyses

  • Consistent translation into thematic objectives and investment priorities

Funding priorities

Linkage to Europe 2020 Strategy



Programme architecture
Programme architecture Acts

Thematic objective

Priority axis

Investment priority 1

Investment priority 2

Specific objective 1

Specific objective 2

Specific objective 1

Specific objective 2


Programming 1
Programming (1) Acts

  • Identification of development needs

  • Selection of thematic objectives and investment priorities as set out in the CPR and Fund-specific rules

  • Definition of at least one specific objective per investment priorityto target the latter appropriately in the specific national or regional context

  • Definition ofresultindicators linked to specific objectives


Programme architecture1
Programme architecture Acts

Thematic objective

Priority axis

Investment priority 1

Investment priority 2

Specific objective 1

Specific objective 2

Specific objective 1

Specific objective 2


Specific objectives Acts

Art. 2 CPR - Definitions

"specific objective' means the aim to which an investment priority or Union priority contributes in a specific national or regional context through actions or measures undertaken within such a priority"


Importance of specific objectives Acts

Art. 17 CPR – ex ante conditionalities

'applicable ex ante conditionality' means a concrete and precisely pre-defined critical factor, which is a prerequisite for and has a direct and genuine link to, and direct impact on, the effective and efficient achievement of a specific objective for an investment priority or a Union priority


Importance of specific objectives Acts

Art. 87(2)(b) CPR – Operational Programme

For each priority axis:

"the investment priorities and corresponding specific objectives"

"in order to strengthen the result-orientation of the programming, the expected results for the specific objectives, and the corresponding result indicators, with a baseline value and a target value, where appropriate quantified in accordance with the Fund-specific rules"

"a description of the type and examples of actions to be supported under each investment priority and their expected contribution to the specific objectives"


Importance of specific objectives Acts

Annex II – performance framework

"Milestones are intermediate targets, directly linked to the achievement of the specific objective of a priority, where appropriate, expressing the intended progress towards the targets set for the end of the period"

Milestones and targets shall be "consistent with the nature and character of the specific objectives of the priority"


Equal opportunities Acts

Art. 7 Equal opportunities

"The Member States and the Commission shall ensure that equality between men and women and the integration of gender perspective are taken into account and promoted throughout the preparation and implementation of programmes, in relation to monitoring, reporting and evaluation."


Programming 2
Programming (2) Acts

  • Onlyafterthe development needs, objectives and the result sought have been clarified, should one consider the types of actions to be supported, choosing (a mix of actions) which best achieves the objectives defined

  • Output indicators should be logically linked to and reflect the types of actions planned. Outputs generated by actions should also contribute logically towards the results that one aims to achieve.


Programming 3
Programming (3) Acts

  • Completing the drafting of the intervention logic

  • Reflection and adjustment of intervention logic

  • Testing the intervention logic – ex ante evaluation

  • Reflection and adjustment of intervention logic


General remarks on indicators
General remarks on indicators Acts

  • Common and where relevant programme-specific

  • Financial, outputand result

  • No impact indicators (difference with 2007-2013)

  • Reported annually andelectronically by MA as structured data as part of the AIR, broken down by investment priority

  • Reported as annual data, not cumulatively (difference with 2007-2013)

  • Relate to partially or fully implemented operations (definition in Art. 2 draft CPR)


General remarks on indicators1
General remarks on indicators Acts

  • Recommendation: monitoring data entered into system throughout the year by beneficiaries/ bodies in charge of entering monitoring data

  • Clear name, unequivocal and easy to understand definition, measurement unit (provided for common indicators)


Monitoring information system Acts

The system must record and store data on individual participants in a way that permits the managing authorities to perform the tasks related to monitoring and evaluation in conformity with the requirements set out in Art. 49 and Annex XX CPR and Articles 5 and 15(iv)(4) and (6), Annex I and II of the ESF Regulation

Tasks include:

  • undertake impact evaluations

  • Be able to contact participants after they have left the support

  • draw a representative sample of participants


Micro data
Micro-data Acts

  • CPR (Art. 114(2)(d)) sets out a legal obligation for the managing authorities to establish a system that records and stores individual participant data in computerised form

    • Micro-data of participants should be collected and stored

    • Micro-data are observation data collected on an individual object, i.e. a participation record

    • Observation data (characteristics and results) collected by indicators

    • Micro data allow MS to create output/result statistics and to match different observation data

    • May be complemented by unique personal identifiers


Data protection Acts

The data processing arrangements must be in line with the provisions of Data protection Directive 95/46, in particular Articles 7 and 8 thereof.

Indicators require the collection of two categories of data on individuals:

  • Personal data – indicator marked with *

  • Personal sensitive data – indicator marked with **


Increased importance of monitoring delegated and implementation acts
Increased importance of monitoring - ActsDelegated and Implementation Acts

  • Model for the OP (IA)

  • Performance framework: arrangements to ensure a consistent approach for determining milestones and targets and for assessing the attainment of the milestones and targets (IA)

  • Performance framework: financial corrections criteria (DA)

  • Data to be recorded and stored in computerised form (DA)

  • Model for the annual and final implementation report (IA)

  • Model for the progress report (IA)


1 model for the operational programme ia
1. Model for the Operational Programme (IA) Acts

  • Performance framework for the priority axis

    • Broken down by Fund

    • Broken down by category of region

    • Key implementation steps, financial indicators, output indicators and where appropriate result indicators

    • Indicators are set at IP level, but the indicators for the performance framework have to be aggregated Indicators and implementation steps must be representative for the priority axis

    • Milestones for 2018 and targets for 2023


2. Performance framework (IA) Acts: arrangements to ensure a consistent approach for determining milestones and targets and for assessing their attainment

Main elements of the Implementing Act:

  • Arrangements for documentation of the establishment of milestones and targets

  • Basic requirements for different types of indicators

  • Arrangements for determining milestones and targets

  • Arrangements for the verification of the attainment of milestones and targets


2. Performance framework (IA) Acts: arrangements to ensure a consistent approach for determining milestones and targets and for assessing their attainment2.1. Arrangements for documentation of the establishment of milestones and targets

  • Methodologies and criteria toselect indicatorsfor PF ensuring milestones and targets comply with criteria of Annex II, § 3 CPR

  • Use of data/evidence and calculation method to estimate the value of milestones and targets

  • Rationale for the selection of output indicators, including explanation on share of the financial allocation represented by the operations which will produce the outputs + method to calculate the share -must represent more than 50% of financial allocation to the priority

  • Info on how methodology to ensure consistency in the PF has been applied in accordance with the provisions of the Partnership Agreement

  • Rational of selection of result indicators and key implementation steps


2. Performance framework (IA) Acts: arrangements to ensure a consistent approach for determining milestones and targets and for assessing their attainment2.2. Requirements for indicators and key implementation steps

  • Milestones and targets to be set at level of the priority

  • In case of multi-fund/multi-category of regions priority axes: breakdown by Fund and by category of region

  • For financial indicators: M&T refer to total amount of eligible expenditure entered into accounting system of certifying authority…

  • For ESF output indicators M&T refer to achieved value for fully or partially implemented operations

  • Key implementation steps refer to an important stage in delivery of a priority, with verified completion, and expressed as number or percentage.

  • Result indicators used where appropriate, closely linked to interventions


2. Performance framework (IA) Acts: arrangements to ensure a consistent approach for determining milestones and targets and for assessing their attainment2.4. Arrangements for verification of the attainment of milestones and targets

  • To be assessed at priority axis level

  • In case of multi-fund/multi-category of regions priority axes: assessment by Fund and by category of region

  • Achievement of milestone/target: all indicators of the performance framework of the priority axis have attained at least 85% of milestone (2018) or target (2023) value

  • Serious failure to achieve M&T:

    • Max 2 indicators/priority axis: any of the output or financial indicators failed to attain at least 65% of the milestone/target value

    • More than 2 indicators/priority: at least 2 of the output or financial indicators failed to attain at least 65% of the milestone/target value


3 performance framework da financial corrections criteria
3. Performance framework (DA) Acts: financial corrections criteria

Conditions for applying financial correctionsat end of programming period on basis of final implementation report:

  • Serious failure to achieve targets liked to financial indicators or output indicators

    • Max 2 indicators/priority: any of the indicators failed to attain at least 65% of target value

    • More than 2 indicators/priority: at least 2 of the indicators failed to attain at least 65% of target value

  • Serious failure is due to clearly identified implementation weakness

  • Commission previously communicated to MAthe clearly identified implementation weakness

  • MS failed to take necessary corrective action

  • No socio-economic or environmental factors, no significant changes in the economic or environmental conditions in a MS…seriously affecting implementation of priorities concerned


Ec reporting art 46bis
EC reporting (Art. 46bis) Acts

Reporting by the EC and debate on the ESI Funds

Three types of report: summary report and strategic reports and annual progress reports

Summary report:

Summary report based on AIRs, including summary of evaluations

Starting in 2016, to Council, EP, ECOSOC, CdR

Strategic report:

In 2017 and 2019, based on progress reports

Annual Progress Reports:

Starting in 2018, every two years

To spring summit, ESI Funds contribution to Europe 2020

Regional Policy

Cohesion Policy


Monitoring regulatory requirements on indicators

Monitoring Acts:

Regulatory requirements on

INDICATORS


List of common ESF and YEI indicators Acts

  • Structure of Annex I and II

  • Annex I - common ESF indicators

    • (1) Common output indicators on participants

    • (2) Common output indicators on entities

    • (3) Common immediate result indicators

    • (4) Common longer-term result indicators

  • Annex II - Youth Employment Initiative indicators

    • (1) Immediate result indicators

    • (2) Longer-term result indicators


Annex I Acts

Common indicators


Output indicators on people Acts

Employmentstatus

  • unemployed, including long-term unemployed*

  • long-term unemployed*

  • inactive*

  • inactive, not in education or training*

  • employed, includingself-employed*


Output indicators on people Acts

Age

  • below 25 years*

  • above 54 years*

  • participants above 54 years who are unemployed, including long-term unemployed, or inactive not in education or training *

    Education

  • with primary (ISCED 1) or lower secondary education (ISCED 2)*

  • with upper secondary (ISCED 3) or post-secondary education (ISCED 4)*

  • with tertiary education (ISCED 5 to 8)*


Participants above 54 years who are unemployed, including long-term unemployed, or inactive not in education or training

Above 54 years


Output indicators on people long-term unemployed, or inactive not in education or training

Disadvantaged participants

  • participants who live in jobless households*

  • participants who live in jobless households with dependent children*

  • participants who live in a single adult household with dependent children*

  • migrants, people with a foreign background, minorities (including marginalised communities such as the Roma)**

  • disabled**

  • other disadvantaged**


Participants who live in jobless households* long-term unemployed, or inactive not in education or training

All household members either unemployed or inactive

Household – housekeeping/social unit:

• having common arrangements;

• sharing household expenses or daily needs;

• in a shared common residence.

-> one person living alone / group of people, not necessarily related

-> living at the same address

Excluded:

- Households composed solely of students.

- Collective / institutional households (hospitals, old people’s homes, residential homes, prisons, military barracks, religious institutions, boarding houses and workers’ hostels, etc.)


Participants who live in jobless households long-term unemployed, or inactive not in education or training with dependent children *

Subindicator of

Participants who live in jobless households*

Dependent children

  • all children under 17 years of age and

  • persons between 17-24 years of age who are economically dependent on their parents

    -> Participant can be any household member (parent, dependent child, other household member)


Participants who live in a single adult household with dependent children*

NO subindicator to "jobless household", but

same definitions regarding

  • household

  • dependent children


Output indicators on people dependent children*

Data to be submitted in

June 2017

Disadvantaged participants

  • homeless or affected by housing exclusion*

  • from rural areas*

    The data on participants under the above two indicators are to be provided in the AIR as specified in Article 44(3) of Regulation (EU) No [...] CPR (i.e. in 2017). They are to be collected based on a representative sample of participants within each investment priority.

    Data for participants about rural areas are to becollectedat LAU 2 (local administrative unit, former NUTS 5).


Homeless or affected by housing exclusion dependent children*

National definition

or

ETHOS (European Typology of Homelessness and Housing Exclusion) definition – classification of four living circumstances :

1. Rooflessness (living rough / emergency accommodation),

2. Houselessness (in accommodation for the homeless, in women's shelters, in accommodation for immigrants, people due to be released from institutions and people receiving long-term support due to homelessness),

3. Insecure accommodation (insecure tenancies, under threat of eviction or violence),

4. Inadequate housing (unfit housing, non-conventional dwellings)

-> Reference "Confronting Homelessness in the European Union"


From rural areas dependent children*

  • Rural areas are to be understood as thinly populated areas according to the Degree of urbanisation (DEGURBA category 3) classification.

  • Thinly-populated areas means that more than 50 % of the population lives in rural grid cells.

  • The data shall be collected at the Local Administrative Unit level of LAU 2 (local administration/communes).The DE-GURBA category 3 shall be established according to http://ec.europa.eu/eurostat/ramon/miscellaneous/index.cfm?TargetUrl=DSP_DEGURBAheader "for reference year 2012".


From rural areas dependent children*


Output indicators on entities dependent children*

  • number of projects fully or partially implemented by social partners or non-governmental organisations

  • number of projects dedicated to sustainable participation and progress of women in employment;

  • number of projects targeting public administrations or public services at national, regional or local level

  • number of supported micro, small and medium-sized enterprises (including cooperative enterprises, enterprises of the social economy)


Number of projects dedicated at sustainable participation and progress of women in employment

Project with the aim of increasing the sustainable participation and progress of women in employment, thus combating the feminisation of poverty, reducing gender-based segregation

and combating gender stereotypes in the labour market and in education and training, promoting reconciliation of work and personal life for all and equal sharing of care responsibilities between men and women.

ESF Regulation, Art. 7, Promotion of equality between men and women


Immediate result indicators and progress of women in employment

  • inactive participants engaged in job searching upon leaving*

  • participants in education/training upon leaving*

  • participants gaining a qualification upon leaving*

  • participants in employment, including self-employment, upon leaving*

  • disadvantaged participants engaged in job searching, education/ training, gaining a qualification, or in employment, including self-employment, upon leaving**


Immediate result indicators
Immediate result indicators and progress of women in employment

Disadvantaged are:

  • participants who live in jobless households*

  • participants who live in jobless households with dependent children*

  • participants who live in a single adult household with dependent children*

  • migrants, people with a foreign background, minorities (including marginalised communities such as the Roma)**

  • disabled**

  • other disadvantaged**

  • homeless or housing exclusion

  • rural area


Disadvantaged participants in job searching, in education/training, gaining a qualification or in employment upon leaving

Different logic than other immediate result indicators

  • All disadvantaged groups to be reported together

  • All immediate results to be reported together


Longer-term result indicators education/training, gaining a qualification or in employment upon leaving

  • participants in employment, including self-employment, 6 months after leaving*

  • participants with an improved labour market situation 6 months after leaving*

  • participants above 54 years in employment, including self-employment, 6 months after leaving*

  • disadvantaged participants in employment, including self-employment, 6 months after leaving**

    These data are to be provided in the annual implementation reports as specified in Article 44(4) of Regulation (EU) No [CPR] (i.e. 2019 & 2023). They are to be collected based on a representative sample of participants within each investment priority.


  • Participants in employment, education/training, gaining a qualification or in employment upon leaving

  • including self-employment,

  • 6 months after leaving

Including self-employment

No separate reporting of self-employed 6 months after leaving


Targets for common result indicators education/training, gaining a qualification or in employment upon leaving

An output indicator as reference is needed.

OP/AIR templates and SFC2014 will give a drop-down menue. Only those combinations will be possible.


Annex II education/training, gaining a qualification or in employment upon leaving

YEI indicators



YEI immediate result indicators (I) indicators

  • unemployed participants who complete the YEI supported intervention*

  • unemployed participants who receive an offer of employment, continued education, apprenticeship or traineeship upon leaving*

  • unemployed participants who are in education/training, gaining a qualification, or in employment, including self-employment, upon leaving*


YEI immediate result indicators (II) indicators

  • long-term unemployed participants who complete the YEI supported intervention*

  • long-term unemployed participants who receive an offer of employment, continued education, apprenticeship or traineeship upon leaving*

  • long-term unemployed participants who are in education/training, gaining a qualification, or in employment, including self-employment, upon leaving*


YEI immediate result indicators (III) indicators

  • inactive participants not in education or training who complete the YEI supported intervention*

  • inactive participants not in education or training who receive an offer of employment, continued education, apprenticeship or traineeship upon leaving*

  • inactive participants not in education or training who are in education/training, gaining a qualification, or in employment upon leaving*


YEI immediate result indicators indicators

3 target groups

x

3 immediate results

=

9 immediate result indicators


YEI immediate result indicators indicators

Completion of intervention

Attendance according to schedule until the last day/last session of scheduled end.

-> No recording as immediate result

if irregular attendance/ drop out.


YEI immediate result indicators indicators

Offer of employment, continued education, apprenticeship or traineeship upon leaving

Voluntary conditional promise,

Indication of offeror's willingness to enter into agreement under specific terms with the participant

Acceptance result in binding agreement with legal commitment of both parties


YEI immediate result indicators indicators

Continued education

Enrolment in

  • formal education or

  • training programmes leading to recognised vocational qualification.


YEI immediate result indicators indicators

Apprenticeship

  • Training contract or formal agreement (occupation, duration, skills to be acquired, wage or allowance etc.)

  • Directly or via the education institution

  • Normally part of formal education and training at upper secondary level (ISCED 3)

  • Duration on average 3 years

  • Successful completion leads to nationally recognised qualification


YEI immediate result indicators indicators

Traineeships

  • Limited period of work practice spent at business, public bodies or non-profit institutions

  • Last a few weeks to a few months

  • Usually not considered to constitute employment contracts


YEI longer-term result indicators indicators

  • participants in continued education, training programmes leading to a qualification, an apprenticeship or a traineeship 6 months after leaving*

  • participants in employment, including self-employment, 6 months after leaving*

  • participants in self-employment 6 months after leaving*

    The data for longer-term result indicators are to be collected based on a representative sample of participants within each investment priority.


Priority axes covering more than one category of region
Priority axes covering more than one category of region indicators

Can be justified where where identical objectives and actions are pursued across all regions

Re-percussions:

  • Co-financing rate and financial management by priority axis and by category of region

  • Breakdown of performance framework and output indicators (and in case of ESF, also result indicators)by category of region


Baselines and targets
Baselines and targets indicators

  • Baselines for result indicators with a target

  • Cumulative target values for 2023 (for output and result indicators), if n+3 rule adopted

  • Targets are quantified for all output indicators (absolute numbers) and common result indicators (absolute numbers or shares/rates) and quantified or qualitative for programme-specific result indicators

  • Consult background papers for ESF target setting http://ec.europa.eu/esf/main.jsp?catId=67&langId=en&newsId=8174


Targets for common result indicators example
Targets for common result indicators indicators- example

  • Investment priority:

  • “Equality between men and women and reconciliation between work and private life”

  • Specific objective:

  • Increase participation of low skilled inactive or unemployed people with care responsibilities in the labour market

    Targets for common result indicators should be set in function of the data reported for common output indicators.


Result target is often expressed in % indicators

The % requires a reference value, i.e. the relevant output data.

CI Result: Participants gaining a qualification upon leaving

Targets:

50% gaining a qualification upon leaving WRONG!

50% of whom? Of all participants? No!

50% of the low skilled participants (ISCED 1 and 2)gaining a qualification upon leaving


CI indicatorsResult: Participants in employment, including self-employment, upon leaving

Targets

50% in employment, including self-employment, upon leaving WRONG!

50% of whom? Of all participants? No!

50% of the unemployedin employment, including self-employment, upon leaving

40% of the inactivein employment, including self-employment, upon leaving


Monitoring programme specific indicators

Monitoring indicators:

Programme specific indicators


Principles for sound indicators
Principles for sound indicators indicators

  • Integrated set of indicators consisting of CI and programme-specific indicators

    • With a limited set of targets. Not all indicators need a target. But the most important ones do.

  • Indicators should cover the main scope of a priority (i.e. target group, type of activity etc.)

    • Targets should also cover large parts of a priority

  • Indicators should be simple - sophisticated data require evaluation


Programme specific indicators recommendation
Programme-specific indicators indicators– recommendation

  • Use common indicators and their definitions when establishing programme-specific indicators – decrease of administrative burden


Programme specific longer term result indicators
Programme-specific longer-term result indicators indicators

  • Possibility to monitor longer-term results over a longer time span than 6 months after leaving

    If the same population/sample as for longer-term result indicators is covered – information on sustainability of results


Enhanced monitoring of results
Enhanced monitoring of results indicators

  • With access to database with micro-data on employment status, enhanced monitoring could be envisaged:

    • Employment / unemployment spell of participants can be monitored in a prolonged period of time after leaving project


Time series with employment results of participants
Time series with employment results of participants indicators

Participants in employment x months after leaving, N=1000

Number of participants

Common longer-term result indicator

Programme-specific longer-term result indicators

Number of months after participants left project


Example from belgium flanders

People that never fell back into unemployment indicators

Example from Belgium (Flanders)

After 6 months: about 1000 never worked

about 500 never worked

Source: presentation "Impact evaluation of PES action for the unemployed in Flanders" by Benedict Wauters (ESF Agency Flanders) and Steven Groenez (KU Leuven HIVA) givenat ESF Evaluation Partnership 13-6-2013


Programme specific result indicators
Programme-specific result indicators indicators

  • Need to be logically linked to outputs (directly supported participants or entities)

  • Global effects on a target group are to be assessed through evaluations

  • Data comes from monitoring or surveys, or from existing databases building on micro-data

    => Avoid indicators based on general statistics, such as (increase of) employment rate in a region (by x%)

YES: participants aged 20 or younger starting an apprenticeship

NO: share of young people aged 20 or younger in apprenticeship


Three approaches to set programme specific indicators
Three approaches to set indicatorsprogramme-specific indicators

1) By combining different common ESF indicators

2) By combining common ESF indicators with programme specific characteristics/ features

3) By settingnewindicatorsfocusingsolely on programmespecificcharacteristics/ features


  • 1) By combining different one-dimensional common ESF indicators

  • Advantage: Data have to be collected anyway. Less administrative burden linked to data collection

  • NB: Some common ESF indicators are based on such combination

    • Participants above 54 years who are unemployed, including long-term unemployed, or inactive not in education or training*

    • Disadvantaged participants engaged in job searching, education/ training, gaining a qualification, or in employment, including self-employment, upon leaving**

    • Participants above 54 years in employment, including self-employment, 6 months after leaving*

    • Disadvantaged participants in employment, including self-employment, 6 months after leaving**


Common output indicator: indicatorsParticipants above 54 years who are unemployed, including long-term unemployed, or inactive not in education or training*

Above 54 years

Inactive, not in education or training

Unemployed, including long-term unemployed


Common longer-term result indicator: indicatorsDisadvantaged participants in employment, including self-employment, 6 months after leaving**

Migrants, people with a foreign background, minorities (incl. marginalised communities such as the Roma)

Other disadvantaged

Disabled

Participants in employment, including self-employment, 6 months after leaving


Programme output indicator young low skilled inactive participants
Programme Output Indicator: indicatorsYoung low skilled inactive participants

Inactive, not in education or training

Below 25 years

With primary or lower secondary education


Programme immediate result indicator high skilled unemployed in employment upon leaving
Programme Immediate Result indicator: indicatorshigh skilled unemployed in employment upon leaving

Unemployed, including long-term unemployed

With tertiary education

Participants in employment upon leaving


Programme immediate result indicator low skilled older workers gaining a qualification upon leaving
Programme Immediate Result indicator: indicatorslow skilled older workers gaining a qualification upon leaving

Employed, including self-employed

Above 54 years

With primary or lower secondary education

Participants gaining a qualification upon leaving


2) By combining common ESF indicators with programme specific characteristics/ features

Advantage: indicators can take into account specific aspects of the programme.


Output indicator neets
Output indicator: NEETS specific characteristics/ features

Participants between 16 and 24 years

Inactive, not in education or training

Unemployed, including long-term unemployed


Output indicator disadvantaged pupils
Output indicator: specific characteristics/ featuresDisadvantaged pupils

Pupils below 16 years

Migrant and minorities

Other disadvantaged

Disabled


Output indicator young high skilled participants establishing a start up upon leaving
Output indicator: specific characteristics/ featuresYoung high-skilled participants establishing a start-up upon leaving

Below 25 years

With tertiary education

Start-ups

Participants in employment, including self-employment, upon leaving

or


Result indicator disadvantaged pupils in education or training upon leaving
Result indicator: specific characteristics/ featuresDisadvantaged pupils in education or training upon leaving

Pupils below 16 years

Migrant and minorities

Other disadvantaged

Disabled

In education or training upon leaving


Result indicator neets in education or training upon leaving
Result indicator: specific characteristics/ featuresNEETs in education or training upon leaving

Inactive, not in education or training

Participants between 16 and 24 years

Unemployed, including long-term unemployed

Participants in education/

training upon leaving


3) specific characteristics/ featuresBy setting new indicators focusing solely on programme specific characteristics/ features


This example shows possible subdivision of a common indicator into programme-specific indicators.


Steps in designing programme specific indicators
Steps in designing programme-specific indicators indicator into programme-specific indicators.

  • 'Deconstruct' the programme with view to the following categories:

    • Target groups (people & entities)

    • Type of planned activities

    • Themes

    • Projects

    • Type of expected results

  • Prioritise the content of each category in order to then identify indicators on the most important aspects

  • Develop indicators e.g. by combining characteristics/ features from the various categories

  • Review intervention logic to ensure that the most important aspects are covered by indicators


Example
Example indicator into programme-specific indicators.

  • Investment priority:

  • “Equality between men and women and reconciliation between work and private life”

  • Specific objective:

  • Increase participation of low skilled inactive or unemployed people with care responsibilities in the labour market


  • Inactive indicator into programme-specific indicators.not in education or training /unemployed/ LTU with care responsibilities with ISCED below 3


  • Participants gaining a qualification upon leaving

  • Low skilled (below ISCED 3) inactive or unemployed participants in employment who received support in their care responsibilities

  • Participants in employment, including self-employment, upon leaving


Inactive not in education or training with care responsibilities with isced below 3

LTU responsibilities with ISCED below 3with care responsibilities with ISCED below 3

Unemployedwith care responsibilities with ISCED below 3

Inactive not in education or training with care responsibilities with ISCED below 3

Inactive, not in education or training

LTU

Unemployed

With ISCED 1 or 2

Below ISCED 1

Participants in employment, including self-employment, upon leaving

With care responisibilitities

Gaining a qualification


Evaluation

Evaluation responsibilities with ISCED below 3


What is hopefully going on right now

What is hopefully going on right now responsibilities with ISCED below 3


Programme design responsibilities with ISCED below 3

Intervention logic

Ex-ante evaluation

Interactive process

Iterative process

Indicators

On-going evaluation

Implementation

Monitoring

Ex-post evaluation


The ex ante should check for a clear intervention logic
The ex-ante should check for a clear intervention logic responsibilities with ISCED below 3

  • Challenges and needs

    • Country-specific recommendations

    • National Reform Programme

    • European semester analyses

  • Consistent translation into thematic objectives and investment priorities

Funding priorities

Linkage to Europe 2020 Strategy


The ex ante should check for a clear intervention logic1
The ex-ante should check for a clear intervention logic responsibilities with ISCED below 3

  • Specific objectives (SO)

    • Precise definition (of change)

      • Should be more specific than the investment priority about target groups, problem area or structure, procedures, institutions, etc. it seeks to change

      • Should logically link to the proposed actions

    • Measurement (result indicators)

  • Types of actions

    • The most appropriate interventions to achieve SO (the right "action mix"?)

    • The choice based on an analysis of problems

    • Measurement of their outputs (output indicators)


The ex ante should check for a clear intervention logic2
The ex-ante should check for a clear intervention logic responsibilities with ISCED below 3

  • Clarifies the intervention logic

  • Demonstrates causal links between:

    • Types of actions -> outputs -> intended results / specific objective

    • Description of how planned actions will contribute to specific objectives is often lacking in first OP drafts reviewed.

  • Helps identifying for which results/outputs, not captured by common indicators, programme-specific indicators should be established

Output indicators

Result indicators


The ex ante checks for relevant and clear indicators
The ex-ante checks for relevant and clear indicators responsibilities with ISCED below 3

Ex-ante evaluators should consult carefully draft guidance on ESF Monitoring and Evaluation 2014-2020 and assess:

  • Relevance

    • Result indicators should capture a change in the situation of supported participants or entities

  • Clarity

    • Indicators should have a clear title and an unequivocal and easy to understand definition


Indicators with realistic targets and identified data sources
Indicators with realistic targets and identified data sources

Ex-ante evaluators should also assess:

  • Target values – are they realistic?

    • Consult background papers for ESF target setting http://ec.europa.eu/esf/main.jsp?catId=67&langId=en&newsId=8174

  • Data collection

    • Will data be collected and reported on time?

    • Have necessary arrangements been put in place to collect and store micro-data of participants?

    • Are existing databases used as data sources?

      (NB: reduction of admin. burden for beneficiaries)

    • Are procedures in place to ensure quality of data?



Evaluation during the programming period art 49
Evaluation during the programming period (Art. 49) sources

MA shall ensure that evaluations are carried out to assess effectiveness, efficiency and impact and that they are subject to appropriate follow-up

Evaluation plan covering one or several OPs drawn up by MA or MS

EC may carry out evaluations at its own initiative.

Regional Policy

Cohesion Policy


Evaluation plan art 104
Evaluation Plan (Art. 104) sources

Evaluation plan to be submitted to the MC no later than a year after the adoption of the programme(s)

By December 2021 MA shall submit to EC a report summarising evaluation findings and main outputs and results of programme, providing comments on the reported information

Regional Policy

Cohesion Policy


YEI Evaluation sources

Art. 15(iv)(6) ESF Reg.

Two evaluations required:

1. completed by end 2015

2. completed by end 2018

Scope: assess effectiveness, efficiency and impact

Be aware of scope of reporting requirement!


YEI Reporting sources

Art. 15(iv)(4) ESF Reg.

The report in 2016, 2019 and 2024 "shall set out and assess the quality of employment offers received by YEI participants, including the disadvantaged, those from marginalized communities and those leaving education without qualifications. The Report shall set out and assess their progress in continuing education, finding sustainable and decent jobs, or moving into apprenticeship or quality traineeship.


Summary report sources

Art. 104(2) CPR

"By 31 December 2022, managing authorities shall submit to the Commission, for each programme, a report summarising the findings of evaluations carried out during the programming period and the main outputs and results of the programme, providing comments on the reported information."


Ex post evaluation art 50 104
Ex-post evaluation (Art. 50 + 104) sources

The EC shall carry out the ex post evaluations in close cooperation with MS and MA

Examine effectiveness and efficiency of ESI Funds and their contribution to Europe 2020 strategy, taking account of the EU targets

By 31 December 2024 and for each ESI Fund, EC shall prepare a synthesis report outlining the main conclusions of ex-post evaluations

Regional Policy

Cohesion Policy


Contents of evaluation plan
Contents of evaluation plan sources

Subject

Purpose (reasoning/use), scope, specific objectives

Key evaluation questions

Data sources and key methods

Timing (link to "reasoning")

Budget

Partnership

Audience

Dissemination strategy

Regional Policy

Cohesion Policy


Impact evaluation expert assistance at your disposal

Impact evaluation: sources

Expert assistance at your disposal


Http crie jrc ec europa eu
http://crie.jrc.ec.europa.eu sources

  • Centre for Research on Impact Evaluation - CRIE


Crie what is it
CRIE sources: What is it?

  • Centre for Research on Impact Evaluation Joint DG EMPL-DG JRC initiative

  • Established in June 2013

  • Support to MS and DG EMPL

  • to set up necessary arrangements for carrying out

  • Counterfactual Impact Evaluations (CIE)

  • of ESF funded interventions


Wp2 crie support to ms
WP2 sourcesCRIE support to MS

  • CRIE supports MS to set up the necessary arrangements for carrying out Counterfactual Impact Evaluations (CIE) of DG EMPL funded interventions by:

  • WP2AOrganizing training workshops on impact evaluation methods

  • WP2BProviding tailor-made advice on methodological and data issues that arise when designing, implementing and evaluating an initiative

  • WP2CProvide support to MS for arranging CIEs (preparation and implementation phase)


Wp2a are training workshops needed
WP2A sources Are training workshops needed?

  • Specific ESF guidance on CIE now exists:

  • EC DG EMPL (2012),

  • “Design and commissioning of counterfactual

  • impact evaluations - A guide for ESF

  • Managing authorities”

  • Other

  • books and handbooks

  • on CIE are now available


Wp2a value added of training workshop
WP2A sourcesValue added of trainingworkshop

  • WP2A1. Problem-based learning methodology

  • WP2A2.Exemplar CIEs are used to illustratethe issues and challenges

  • WP2A3. Customized: Participants can propose examples of CIEs (they are working on or in the process of commissioning) for discussion at the workshop



ad