Systems Evaluation An European Perspective

1. Systems EvaluationAn European Perspective Erik Arnold www.technopolis-group.com Vienna 25 April 2006

2. ‘Systems’ has become an important concept in R&D policy and evaluation • We rely on the ‘innovation systems’ heuristic in designing and implementing policy interventions • We increasingly justify interventions in systems failure, as well as market failure, terms • We need to integrate evidence about systems performance with intervention logics • We increasingly pose policy questions in terms of portfolios - most recently ‘policy mix’ - rather than individual interventions

3. While ‘innovation system’ sounds good, as currently conceptualised it tells us little more than that everything is connected to everything else Source: Erik Arnold and Stefan Kuhlmann

4. Whatever happened to systems theory? • General systems theory (cp Wiener, Bertalanffy…) seems to have a great future behind it, with many of the questions it raised now being tackled within disciplines (Ingelstam) • Recent testimony came in the form of an evaluation of VINNOVA’s Complex Technological Systems programme, which centred on the absence of a theoretical vector of systems understanding among applications domains • The ‘Limits to Growth’ episode illustrated the dangers of disconnecting mathematics from understanding and evidence - but helped set an enormously important agenda • Nonetheless, if we don’t attempt even a verbal description of how innovation systems hang together we • Won’t learn much about how to put together our understandings in a policy-useful way • Can’t rescue evaluation from its essentially non-cumulative, non-scientific lock-in to trying to answer impossible questions

5. Research and innovation policies are starting to overlap and to become more systemic Measures Measures • System strengthening • Within actors • Between actors • - Reducing bottlenecks Intra-organisational learning, capability development and performance improvement Development measures MAPs and network measures Multiple Inter-organisational learning, network development and strengthening Point or step change in organisational performance Activity promotion or subsidy measures Linkage or ‘bridging’ measures Single Actors Single Multiple Single Multiple Actors Actors

6. Parliament Level 1 High-level cross-cutting policy Government Policy council Level 2 Ministry mission-centred co-ordination Ministry of Industry Other Sectoral Ministries Ministry of Education Level 3 Detailed policy development, co-ordination Research Councils and Academies Technology & Innovation Agencies Support Programme Agencies Level 4 Research and innovation performers Programme Contractors Producers: Firms, farms, hospitals, etc R&D Institutes Key Universities Instructions, resources Advice Results Horizontal co-ordination and integration Funding systems and governance are complex, but moving towards the Finnish model. Horizontal co-ordination, distributed strategic intelligence and arenas are keys

7. Market failure - mostly about basic research Indivisibility Inappropriability Uncertainty Systems failure - mostly about inadequate performance Capability failures Institutional failures Network failures (including lock-in and transitional failures) Framework failures While we claim that systems failures are especially important. They can be hard to see if you only evaluate within programmes Funding rationales

8. 97 98 99 00 01 02 03 04 05 FP6 FP5 FP4 Scope of 5YA in 2004 5YA Scope of 5YA in 2000 5YA Five Year Assessment is a complex ritual, not easy to connect to the Realpolitik of Framework Programme design

9. Evaluating the FPs has not been easy • The size and complexity of the FPs mean that the familiar difficulties of evaluating RTD programmes are present in large measure • Assessing dead weight, time scales, choice of methods, inadequate models of relationships between R&D and other social variables, etc etc etc • Some issues appear particular to the FPs • Data access • ASIF on 5YAs prior to 2002: “Many of the recommendations drew not so much on an evaluation of past Framework activities but on the collective opinions and assessments of the panel members concerning the general structure and organisation of science,, technology and innovation in the EU” • Peer review extended, arguably, beyond its elastic limits in past 5YAs, but panels now being better supported by studies

10. Key problems are in planning, not evaluation. FP5 is a classic case of ‘the missing middle’. (FP6 is a bit better) • Goals that forbid little, defined more in terms of process than results • Unclear ‘vertical’ interrelationship among goals make the intervention logic (programme theory) hard to discern and evaluate • Lack of clarity in goals about relationship with the context leaves scope for ‘killer assumptions’ • Scale and scope issues therefore not addressed in relation to objectives • Interplay among activities, results and purposes not well understood • Consistency between activities and goals managed by criteria, rather than by planning

11. More comprehensive planning, overcoming the ‘missing middle’ problem would provide … • Higher-quality, testable logics leading to an improved probability of reaching policy goals • Evidence- and logic-based arguments to underpin the size of the budget needed by the FPs, shifting the balance of negotiation towards rationality and increasing the chances that the resources available for EU R&D policy are about the same size as the resources actually needed • Improved evaluability, with corresponding benefits for improved processes, organisational learning and accountability to the taxpayer

12. Evaluation needs to be a component in a better articulated system of strategic intelligence and planning FP and other ERIS-related policies Evaluation and Studies Overall Objectives A successful and scientifically strong European industry and high quality of life for citizens Analysis of system health Policy development Policy Purposes Policy goals of FP SPs Other policy goals, eg creating ERA Meso-level ‘bottleneck analysis’ + thematic evaluation Programme / Action Goals Expected results of SP 1 Expected results of SP2 Expected results of SP3, etc Evaluating programmes and portfolios Project 1.1.2 Project 1.1.3 Projects, Activities Project 1.1.1

13. Swedish energy research - a response to the 1973 oil crisis

14. Performers’ ratings of Swedish research capabilities

15. Performers’ ratings of Swedish industrial capabilities

16. Systems issues in Sweden • Inadequate mental models of R, D&D in the political and policy systems • Systemic weaknesses in the Swedish research performer system, which undermine the achievement of the significant energy, environment and social goals established in the legislation for the programme • Interference between the political and research funding systems, leading to lock-in to an undesirable balance of activities in the programme • Poor integration between the research funding and innovation systems, leading to an ineffective expenditure pattern • Inability of the Swedish research and innovation governance system to provide adequate co-ordination • Key obstacles built into framework conditions, which prevent the translation of policy goals into practice, even where the needed knowledge base has been established

17. Austria - institutional challenges 2002 … (wow!)

18. FFF conclusions … lock-in by the stakeholders • A brilliant performer of its 1960s job, that has not evolved as fast as it should • Failed to exploit its strategic intelligence mission • Deficit financing problematic • Brings substantial benefits to beneficiaries, but is too risk-averse • Especially good for smaller firms • Overtaken by developments elsewhere in the funding system

19. FWF conclusions … lock-in by the stakeholders • A brilliant performer of its 1960s job, that has not evolved as fast as it should • Niche player • Failed to exploit its strategic intelligence mission • Budget too small • Compared with the GUF • Needs to pay overheads • Strategic intelligence, internationalisation • Potential new roles: themes, Pasteur’s Quadrant • Fragmentation of instruments within a narrow role

20. The Realpolitische response (text from final report) to reforms already in train - with one of the evaluators thereafter accompanying the reform process • FFF should be merged into a broader innovation agency. The proposed merger with TIG, BIT and ASA appears to be a reasonable option for achieving this, although other configurations would also be possible • The Funds should be transformed into agencies and the power of their beneficiaries in the governance structures should be limited • We interpreted the Research Promotion Act of 1967 as a vote of ‘no confidence’ in the Austrian state’s ability to govern R&D agencies in a modern manner. To reverse that vote, the ministries and political level need to demonstrate that they can • Manage by objectives and properly delegate authority to agencies, without seeking to interfere in daily operations such as project assessment. This should include delegation of programme design as well as management • Maintain the ‘strategic intelligence' needed to do this • Professionalise leadership and personnel decisions in the agencies, so that appointments are made in fair and open competition • Develop reasonably standardised ways of instructing agencies, so that ministries can use different agencies to achieve different policy objectives

21. RCN NAVF NLVF NTNF NORAS NFFR BF KS IE NT MH MU NMF RSF RMF RHF RNF Systems issues in the RCN evaluation focused on governance and institutional structures that prevented the new organisation from doing the ‘crucial experiment’ of having a single council

22. Executive Board Government Director General Division Boards (6) Strategy, Admin Division Directors (6) Programme & Discipline Boards Ministries Divisional Staff It also provided testimony to the power of personalities. RCN’s three ‘steering levels’ were a battlefield in the early days but fighting stopped when the minister sacked the protagonists

23. Does a PART-like approach with standardised reporting help? • PART • Programme design and purpose (20%) • Strategic planning (10%) • Programme management (50%) • Programme Results/Accountability (50%) - where measurable, which they frequently aren’t • Looks very rational - so did GOSPLAN • Provides neither systems insight nor the comparative RoI information the finance ministries think they want • Gives policy makers no help with bottleneck analysis and systems improvement

24. And it rather passes by the operational purpose of evaluation The right thing Fix it Carry on The wrong thing Aaaaaargh!! Stop Done well Done badly

25. Analysis of system health Policy development Hypotheses about bottlenecks Evaluation results Meso-level ‘bottleneck analysis’ and evaluation Hypotheses about bottlenecks Evaluation results Evaluating programmes and portfolios Evaluate at different levels. Don’t try to do everything at once. Needs increased strategic intelligence in and for funder organisations

26. Evaluation has a political logic • Forget a wholly rationalistic approach to systems evaluation • Politics matter - and the more of the system you evaluate, the more the politics matter • People matter • Windows of opportunity come and go - useful evaluation is time-bound • ‘Administrative shaping’ of evaluation Terms of Reference is a fact of life • We all want to speak truth to power - the question is how much truth to speak • The overriding criterion is not absolute truth (whatever that is) but what is likely to be useful • The MSc student question - this is not the same as saying that evaluators should be compliant - that’s a suicide mission • Evaluation is, finally, a contribution to a debate, nothing more • (People whose egos are too large to cope with that should do research) • We would be better served by a more articulated IS theory