New Frontiers? „New frontiers in Evaluation“ April 24th and 25th, Vienna

1. New Frontiers?„New frontiers in Evaluation“April 24th and 25th, Vienna Dorothea Sturn

2. founded September 2004 Out of four forerunning organisations Forschungsförderungsfonds für die gewerbliche Wirtschaft (FFF) Technologie Impulse Gesellschaft (TIG) Austrian Space Agency (ASA) Büro für internationale Technologiekooperationen (BIT) As a result of an evaluation?! „FFF should be merged into a broader innovation agency. The proposed merger with TIG, BIT and ASA appears to be a reasonable option for achieving this although other configuration would also be possible.“ (FFF/FWF evaluation) 2006 420 Mio € Fundig Budget 182 people employed The FFG (Austrian Research Promotion Agency)

3. Organigramm Revision Geschäftsführung Strategie Agentur für Luft- und Raumfahrt (ALR) Interne Services Basis- programme (BP) Struktur- programme (SP) Thematische Programme (TP) Europäische und internationale Programme (EIP)

4. Policy Makers Ignore results from evaluatorsOverworkeddecide intransparent Traditional world of ignorance ManagersIgnore all strategic arguments Feel botheredDefend projects Evaluators Ignore questions from policy Behave as scientists

5. Did anything change? • YES • Rise of formative and interactive approaches • Policy questions are taken into account • Evaluators give important inputs in decision-making • NO • ALL parties are overworked and busy, higher level of policy makers is absent • Other influences and interests are stronger • Arguments and results of evaluations are both complex and difficult to implement (please give me an evaluator with only one hand)

6. Examples • The linear model • Portfolio analysis and Goal overload • Additionality

7. The linear model • What evaluators tell us:„The linear model is a rare case” • What we do: • For each Euro we spent must be declared whether it is basic research or applied research

8. Why? Simple answer: Commission wants it More complex answer: Difficult distribution of funding among different interest groups Public opinion: who gets how much money out of the organisation

9. Portfolio Evaluation • What evaluators tell us:„Do not only look at a single programme – you have to optimise the whole portfolio“ • What we do: • We are introducing a new programme twice a year. As a result, we have an enormous amount of small and byzantine programmes with overlapping goals

10. Why? • It is nearly impossible not to include the usual catchwords e.g. SMEs, technology transfer, excellence • It is nearly impossible to let a programme peter out • It is nearly impossible not to jump on a new thrilling topic, if other countries do so • It is nearly impossible not to do something new, since newspapers more and more like the research topic

11. Additionality • What evaluators tell us:„Additionality is complex to assess” • What we ask: • Where we get the best value for our money?”

12. Is it possible to concentrate on measures showing high additionality? Policy maker: „I want to make it faster“ Expert: „It is the motor which is responsible for the speed“ Policy maker: „next generation car will consist of at least 80% motor“

13. How we can improve? • Evaluation should be: • Support for strategic management • A context dependent managerial activity (Philipe Larédo)

14. What evaluators can do: • Combine formative and summative parts • Integrate elements that contribute to the actual decision-making and policy design (without loosing independency) • Organise the evaluation process as a process of interaction • Be brave – even if it means biting the hand which feeds you – but don't be too cynical, understand that there are multiple and conflicting objectives, interests and stakeholder

15. What funding agencies / policy makers can do: • Increase the organisation's strategic intelligence, absorptive capacity and evaluation know-how at all levels (including CEO) • Monitoring and information system: Collect all data you use and use all data you collect • Contact evaluators at the beginning of a process of change • Use experts but do not abuse them • Let evaluation become part of an active learning process