using evaluation research as a means for policy analysis in a new mission oriented policy context
Download
Skip this Video
Download Presentation
E. Amanatidou, UNIMAN / MIoIR I. Garefi, Atlantis Consulting S.A. P. Cunningham, UNIMAN / MIoIR

Loading in 2 Seconds...

play fullscreen
1 / 19

E. Amanatidou, UNIMAN / MIoIR I. Garefi, Atlantis Consulting S.A. P. Cunningham, UNIMAN / MIoIR - PowerPoint PPT Presentation


  • 83 Views
  • Uploaded on

‘Using evaluation research as a means for policy analysis in a ‘new’ mission-oriented policy context’. E. Amanatidou, UNIMAN / MIoIR I. Garefi, Atlantis Consulting S.A. P. Cunningham, UNIMAN / MIoIR A. Gök, UNIMAN / MIoIR 2012 EU–SPRI Conference, Karlsruhe, 12-13 June 2012. Outline.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' E. Amanatidou, UNIMAN / MIoIR I. Garefi, Atlantis Consulting S.A. P. Cunningham, UNIMAN / MIoIR' - nolan-bullock


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
using evaluation research as a means for policy analysis in a new mission oriented policy context

‘Using evaluation research as a means for policy analysis in a ‘new’ mission-orientedpolicy context’

E. Amanatidou, UNIMAN / MIoIR

I. Garefi, Atlantis Consulting S.A.

P. Cunningham, UNIMAN / MIoIR

A. Gök, UNIMAN / MIoIR

2012 EU–SPRI Conference, Karlsruhe, 12-13 June 2012

outline
Outline
  • The ‘new’ policy context
  • Special nature of grand challenges
  • Challenges for evaluation
  • Meeting the evaluation challenges
  • Usefulness of evaluations
  • Impact assessment
  • Behavioural additionality
  • The Structural Funds framework
  • Meeting the evaluation challenges - revisited
the new policy context
The ‘new’ policy context
  • Mission – oriented
  • Orientation to dealing with ‘grand challenges’
  • Not new as policy orientation but some important different features:
    • From national, military – industry lead projects in the ‘40s & ‘50s to more global, socially-driven endeavours oriented to tackle challenges impossible to solve by single entities or countries or rational planning approaches
special nature of grand challenges
Special nature of grand challenges

Need to break long-standing boundaries; need approaches that are:

  • Multi-disciplinary in science & technology
  • Addressing s&t as well as social innovations
  • Multi-level in governance
  • Cross departmental in policy
  • Multi-actor, multi-agency
  • Cross – sectoral
  • Applying longer-term horizons
challenges for evaluation 1 2
Challenges for evaluation (1/2)

Grand challenges Evaluation challenges

Multi-disciplinarity

S&t fragmentation

S&t and social innov.

Multiple impact types

Multi-level gover’nce

Multiple levels of ref.

Policy coordination

Policy silos

Multi-actor / agency

Broader set of stkh’s

Long-term approach

Impact identification

challenges for evaluation 2 2
Challenges for evaluation (2/2)

Evaluation challenges Challenges’ groups

S&t fragmentation

Need for policy learning at operational, policy and especially system level

Multiple impact types

Multiple levels of ref.

Policy silos

Wider set of impacts

Beyond S&T&E

Beyond inputs/outputs,

Behavioural change

Broader set of stkh’s

Impact identification

meeting the evaluation challenges
Meeting the evaluation challenges

Evaluation challenges

Need for policy learning at operational, policy and especially system level

Usefulness of evaluations

Impact assessment

Wider set of impacts

Beyond S&T&E

Beyond inputs/outputs,

Behavioural change

Behavioural additionality

usefulness of evaluations 1 2
Usefulness of evaluations (1/2)

INNO Appraisal results showed that:

  • Significant positive correlations with usefulness were identified for:
    • Use of open tendering process when commissioning an evaluation
    • Use of external evaluators
    • Summative over formative evaluations
    • Evaluations that examined goal attainment and effectiveness and policy/ strategy development
    • Evaluations that employed: case study analysis; participant surveys; interviews; focus groups/workshops and meetings; peer review
    • Evaluations that resulted in minor redesign or expansion/prolongation of the measure
    • Evaluations not conducted primarily for auditors/financial authorities
    • Evaluations whose reports were published in English
  • Evaluations of measures for science-industry cooperation and creation of start-ups/spin-offs significantly more useful
usefulness of evaluations 2 2
Usefulness of evaluations (2/2)*

Broadly, an evaluation may be considered useful if it:

  • delivers the Terms of Reference in a consistent manner
  • provides actionable recommendations
  • delivers value for money
  • delivers some degree of policy learning.
  • BUT evaluations not extensively used to mobilise the community (only 50% targeted beyond policy makers and programme managers)
  • usefulness is highly subjective and context specific

* Based on the INNO-APPRAISAL study

impact assessment
Impact assessment*
  • Impact assessment limited and simplistic in its approach
  • Assessment of economic impact most dominant
  • Assessment of new impact types rather uncommon
  • Demand for non-economic impacts and spill-over effects
  • What new impacts under the ‘new’ mission orientation? (beyond intended, beyond visible, that can last, that span across and beyond levels of references, that refer to multiple stakeholders, and cross different policy arenas)
  • New sets of criteria and indicators required
  • Known challenges in assessing social impacts (e.g. causal links, evidence, attribution problems, timing)

* Based on the INNO-APPRAISAL study

behavioural additionality 1 2
Behavioural additionality (1/2)*
  • Emphasis on learning, long-term horizon
  • Gaining importance (addressed in 50% of national evaluations)
  • Three types of use:
    • behaviour-focused way
    • Integrated ways
    • Instrumental way
  • BA evaluations broadly discussed and more often targeted towards the general public and towards users

 learning and mobilisation potential

  • A need to demonstrate the conceptual link between the behavioural change and (intended) innovation effect
  • The ‘black box’ remains

* Based on the INNO-APPRAISAL study

behavioural additionality 2 2
Behavioural additionality (2/2)*
  • If an evaluation is considered as an administrative exercise that was imposed by a supra-national sponsor, the policy learning and behavioural additionality it creates is limited. However, if it becomes part of the policy learning experience its usefulness and behavioural additionality increase.
  • Context and wider framework within which evaluation is implemented crucial for both usefulness and BA

* Based on the INNO-APPRAISAL study

the structural funds framework
The Structural Funds framework

If and how the SF framework affects the quality and usefulness of evaluations and thus the potential for BA*

  • SF evaluation requirements in relation to
  • Structures
  • Evaluation design
  • Evaluation execution
  • aim, type, and nature of evaluations
  • Evaluators’ selection
  • Publicity & Dissemination
  • Quality assurance

* Based on the INNO-APPRAISAL study (Case studies in Greece, Malta and Poland)

the structural funds framework1
The Structural Funds framework

If and how the SF framework affects the quality and usefulness of evaluations and thus the potential for BA*

  • Research hypotheses
  • SF requirements may lead to specific characteristics in delivery & practice of evaluation
  • SF requirements may lead to higher quality evaluations
  • High quality SF evaluations may have greater impact
  • SF regulations demand high standards on structures and processes that inevitable need some institutional learning and structure building

* Based on the INNO-APPRAISAL study (Case studies in Greece, Malta and Poland)

sf requirements may lead to specific characteristics in delivery practice of evaluation
SF requirements may lead to specific characteristics in delivery & practice of evaluation 
  • Use of external evaluators
  • Data Analysis and collection methods: not surprising differences
  • Evaluation topics: SF requirements do make a difference in guiding the evaluation topics to cover across the different evaluation types (ex-ante, interim, ex-post)
  • Clearer differences across the evaluation types within each group than across the two groups (SF and non-SF group)

 Hypothesis valid but also differences in features may relate more to evaluation type than SF or non-SF framew.

* Based on the INNO-APPRAISAL study (Case studies in Greece, Malta and Poland)

sf requirements may lead to higher quality evaluations x
SF requirements may lead to higher quality evaluations XHigh quality SF evaluations may have greater impact X
  • Compliance to quality standards is less in SF evaluations
  • Certain standards clearly ‘overlooked’: production of useful recommendations; discussion of results with government and stakeholders
  •  SF requirements do not necessarily lead to high quality appraisals
  • What clearly improves usefulness of recommendations is the evaluation design and application of quantitative methods
  • Increased discussions with government and wider stakeholders are more caused by high quality non SF eval.
  •  SF requirements for high quality do not necessarily lead to high impact in terms of usefulness and dissemination

* Based on the INNO-APPRAISAL study (Case studies in Greece, Malta and Poland)

slide17

SF regulations demand high standards on structures and processes that inevitable need some institutionallearning and structure building 

  • SF evaluations more as ‘internal’ exercises
  •  limited discussions with government and stakeholders
  • SF regulations too focused on financial aspects and correctness of implementation - lack of qualitative impact assessment or BA
      •  limited quality inputs to new programmes and schemes
  • limited usefulness of recommendations
  • high quality SF evaluations ≠ high impacts in terms of usefulness and dissemination to national stakeholders
  •  Typical application of procedures ≠ institutional and policy learning BUT significant capacity building

* Based on the INNO-APPRAISAL study (Case studies in Greece, Malta and Poland)

slide18

Meeting the evaluation challenges revisited

Usefulness: not extensive; relates more to operational issues

Need for policy learning at operational, policy and especially system level

Impact assessment: Several issues pending; new issues emerging

Wider set of impacts

Beyond S&T&E

Beyond inputs/outputs,

Behavioural change

BA: Still a ‘black box’

SF framework: typical application; not opportunity for learning

thank you for your attention

Thank you for your attention!

E. Amanatidou, UNIMAN / MIoIR, [email protected]

I. Garefi, Atlantis Consulting S.A., [email protected]

P. Cunningham, UNIMAN / MIoIR, [email protected]

A. Gök, UNIMAN / MIoIR, [email protected]

ad