1 / 29

Participatory Evaluation 7th May 2010 Budapest

ex-post and on-going evaluations of eu development programmes by the European commission and the visegrad countries. Participatory Evaluation 7th May 2010 Budapest. Agenda. DTI and our evaluation experiences Participatory evaluation in general

justina-day
Download Presentation

Participatory Evaluation 7th May 2010 Budapest

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. ex-post and on-going evaluations of eu development programmes by the European commission and the visegrad countries Participatory Evaluation 7th May 2010 Budapest

  2. Agenda • DTI and our evaluation experiences • Participatory evaluation in general • Case - Vocational Education and Training system • Case - VET reform 2000 ex post evaluation • Lessons learned

  3. Evaluation assignments • Leonardo programme evaluation • Observing change in VET – Leonardo II programme • Impact assessments and ex ante evaluations in the area of audiovisual and media policies (DG information Society) • Impact assessments and ex ante evaluations in the area of ICT deployment, innovation, public services and quality of life (DG information Society) • Cedefop project evaluations (DG Education & Culture) • Mid-term evaluation of LEADER+ in Denmark • Thematic ex post evaluation of Structural Fund support to enterprise development and innovation (DG Regio) • WING - study for the impact analysis of FP5 eGovernment, e-learning and e-business (DG information Society) • + many evaluations at national level in Denmark, Sweden and Norway.

  4. Overview of participatory methods UNDERLYING PRINCIPLES • embracing complexity • recognition of multiple realities • prioritising the realities of the disadvantaged • grassroots empowerment • from assessment to sustainable learning • relating learning to action TECHNIQUES: • Visual techniques (diagrams, ranking, mapping, etc) • Consensus conferences • Group and team dynamics methods (incl. Role play) • Photo and video productions & diaries • Case studies, semi structured interviewing, participant observation, etc.

  5. Participatory Evaluation CASE 1: Evaluation of VET system

  6. Political frame Skilling – theory, practice and assessment Institutional frame Organisational frame Activity Complex social VET system Colleges and company National bodies, qualification structure, courses, assessment bodies, inspection Law, ministerial notes, regulations

  7. Policy - the broader the scope – the more complex

  8. Changing and evaluating social systems • Changing and evaluating - directed simultaneously at all levels • Interactivity and consistency between the different layers are the main requirements for effective change • It is a continuous process of incremental development sometimes combined with ”earthquakes” • Social systems are embedded (for example VET sits between economy, education and social affairs • Reasons for change can be endogenous and exogenous • Multi layer, Multi actor and multi purpose character!

  9. ”I can’t change the direction of the wind- but I can set my sails so I am sure to reach my destination” • The notion of educational planning: implicitly suggest clearly defined objectives, mutually exclusive choices, undisputed causal relationships, predictable rationalities, and rational decision-makers. • BUT IS THAT THE REALITY • Increasing complexity- devolution of governance • Educational planning- overlapping episodes, multiple actors and divergent perspectives. It entails the processes through which issues are analysed and policies are generated, implemented, assessed and redesigned.

  10. A fluid perspective on policy evaluation • (i) Analysis of the existing situation. • (ii) The generation of policy options (participatory). • (iii) Evaluation of policy options (participatory). • (iv) Making the policy decision. • (v) Planning of policy implementation (participatory). • (vi) Policy impact assessment (participatory). • (vii) Subsequent policy cycles (part...).

  11. Participatory Evaluation CASE 2: Ex-post evaluation of VET policy reform.

  12. Ex post evaluation and participatory methods • Evaluation of VET policy reform 2000 • Focus on implementation of reform • … and key challenges observed Organisation Evaluation and quality Competences Theme Pedagogical toolbox Inter play

  13. Themes covered by VET Reform evaluation • Teacher teams and new organisation types (teams, subject groups, mentors, etc.) • Contact teacher scheme (contact teacher for each pupil) • Modulerisation and ”whole picture” focus • Pedagogy and learning styles • School collaboration (between schools – pupil plans, education offer, teacher collaboration, etc.)

  14. Participatory methods used • Self-evaluation of collaborative efforts between different actors • Explorative workshops to assess learning points from pilot activities across institutions • Reflective assessment of scaling possibilities in multi-actor workshops

  15. Results • Recommendations for scaling-up activities • Practical idea catalogue for dissemination and further implementation at all schools • Greater awareness among the actors in relation to their role, strategies for the continued implementation of the reform

  16. Learning points • Evaluation as support for policy implementation • Governance of innovation • Both Top-down and exploiting grass root initiative • Systematic use of ”pilots” • Participatory methods require a need for further action (almost always the case)

  17. Lessons learned from participatory evaluation • Take into account the political context and timing (e.g. Elections, economic crisis) • Aim for a truly inter-sectoral process (where relevant) • Conducting a participatory evaluation takes time and resources, but it is worthwhile! • Reach consensus on the definition of key concepts • Address concerns related to participatory processes • Institutional context and individual competences and interests • Strong leadership is central to the sustainability of the evaluation initiative • Conducting participatory evaluation in itself can be an empowering process

  18. Consensusconferences

  19. Purpose of ConsensusConferences • Enrich and expand the scope of traditional debate between experts, politicians and interested parties by communicating citizens’ views and attitudes on potentially controversial technologies. • Citizens’ views play an important part in society’s overall technological assessment, as many technologies have direct bearing on the daily lives of ordinary people. • Qualifies people’s attitudes - they are given all the information they require until they are ready to assess a given technology. • Citizens’ thoughts and recommendations come face to face with the arguments and attitudes of politicians and interested parties.

  20. Planninggroup (of experts) The tasks of this group include: • Preparing guidelines and ensuring wide accessibility of the introductory material for the citizens’ panel • Commenting on and approving the introductory material • Approving the composition of the citizens’ panel • Contributing in the selection of experts to the expert conference panel • Commenting on and approving the conference programme

  21. Selectioncriteria for the Citizens Panel and tasksallocated Selection • Must be representative • Open minded and interested in debating the issues in question • Represent a broad experience base in relation to the conference topic Tasks • put qualified questions to the expert panel and on the basis of their answers formulate the final document (conclusions and recommendations to politicians). • In addition to the 4-day conference, panel members must commit themselves to participating in 2 preparatory weekend sessions. • Initially reading an introductory paper (written by journalist) • Work with the process consultant managing the process to express attitudes and messages within the panel

  22. Suitability of consensusconferences Consensus conferences are suitable in connection with: • A topic of current social relevance • Regulation requiring public support The method is also suitable when: • There is a need for further public awareness and debate • There is a need to identify attitudes and objectives • There is a special need for public input • The policy-makers have to be able and willing to listen and take the results seriously as proposals from the public.

  23. Exampleswithin the framework of the Danish Board of Technology (DBT) A few examples of consensus conferences held by DBT: • How can we assign value to the environment? (2003) • Testing our genes (2002) • Road pricing (2001) • Electronic surveillance (2000) • Noise and Technology (2000) • Genetically modified food (1999) • Tele-work – near and far (1997) • The Consumption and Environment of the Future (1996) • The Future of Fishing (1996) • Gene Therapy (1995)

  24. Important future perspectives • Development of scenario workshops so that citizens can be involved earlier, at the stage of design and selection of criteria for developing technology. • Strengthening the power of consensus conferences to present the broad approach of citizen panels as just as legitimate a criterion in technology decision-making as the more narrow approach of experts. • Both methods have demonstrated great capacity for creating awareness of methodological innovations and diversification, and for building networks.

  25. ParticipatoryEvaluation Case 3 – Leader+ evaluation – sustainable regional development

  26. Participatoryevaluations • LEADER+ • Goodgovernance • Participation at all levels • Sustainability • Gendermainstreaming • Transparency • ”Multi-level” governance (EU, national, regional) • Focus: On-going policy processesratherthanresults • The aim: To develop an innovative processorientedevaluationmethod for sustainable rural development by includinglocalknowledge

  27. Success factors assessed through ex-post perspective • Pressure of problems • Project design that supports win-win coalitions • Short term success • Manageable structures and opportunities for linkages • Powerful interceders and partners • Learning aptitude and exchange of information • Transparency, process competence and flexibility • Participation • Regional promoters as policy entrepreneurs • Critical mass of labour time and money for the regional management • Competent regional management

  28. The evaluationcycle • The ongoing evaluation of the political processes at the regional level on the basis of the success factors (previous slide) • Empirical research of the characteristics of the success factors at the regional level. Identification of the requirements for policy advice • Delivery of intermediary results of the evaluation to the partners in the project and possible the expert or external consultant • On the basis of intermediary results, the consultant gives advice to the regional stakeholders in order to develop a method of regional management and self-evaluation. • If the intermediary results show important problems in a region, the consultant intervenes into the ongoing process with the intention of changing the behaviour of the actors • Evaluation of the impact of the policy advice on the policy process

  29. Keychallenges • Difficulties in measuring the value of thesequalitativeindicators to evaluate the politicalprocess. • Solution: • Anonymity • Severalqualitativesources • Examination of presscoveragecombinedwiththeoreticalevaluationmethod • Not all regional stakeholders and promotersarewilling to take the advice of the consultant • Hence the method is bestused in regions were the stakeholders and promoters have lessexperiencewithsustainable regional development.

More Related