Verena Groß, Bernhard Lageman, Michael Rothgang (RWI Essen)

1. Session C: Portfolio Evaluation Conceptual Challenges of Evaluating the Impacts of a Sizable Technology Programme Portfolio: The Promotion of Industrial Collective Research (IGF) Verena Groß, Bernhard Lageman, Michael Rothgang (RWI Essen)

2. Content A Portfolio Evaluation B About IGF C Some Questions Concerning the IGF Program Design D Evaluating IGF: the Challenge E Evaluation Design

3. Portfolio evaluation aims at assessing rationality, programme-specific institutional solutions, effects and efficiency of complex programmes Two central dimensions: Analyzing the structures, procedures, effects of different components of the programme Comparative analysis of the programme with respect to similar programmes Methodological solutions? Evaluation research offers a lot of relevant quantitative and qualitative solutions Evaluation studies do not lead always to convincing results Portfolio evaluation remains a very challenging task A Portfolio Evaluation

4. Old programme: IGF – state-funded collective industrial research: a technology programme introduced in the early 50th of the German Ministry of Economics The German Federation of Industrial Research Associations "Otto von Guericke" (AIF) is commissioned with the programme execution B About IGF B.1 Old programme - modern features

5. Modern features: development of industry-wide research networks projects should mainly promote SMEs precompetitive research and results available for every firm few ex ante restrictions on the topics of the research projects project ideas are developed bottom up B About IGF B.1 Old programme - modern features

6. total budget since the early fifties 2 billion Euro (2005 prices) nearly 15,000 collective research projects actual annual budget ca. 90 million €, which is to be increased up to 120 million € in 2010 600 to 700 new collective research projects per year average project cost about 138,000 €, project costs reaching from 50,000 to 350,000 € normal project period is two years B About IGF B.2 Costs and projects

7. 103 research associations (in 1954 on 17) in most sectors and technology fields that network: enterprises (~ 50,000 SMEs) ~ 700 research institutes Examples: Association for Cooperative Research for Hot Dip Galvanizing Research Association of the German Food Industry Society for Chemical Engineering and Biotechnology B About IGF B.3 Industrial Research Associations, Institutes and Enterprises

8. B About IGF B.4 Project Genesis Research Institutes Enterprises Phase 1: Identifying research problems and development of research ideas Phase 2: Discussion of competing research ideas and elaboration of proposals within research asscociations Industrial Research Associations Phase 3: Presentation and official submission of proposals to AIF Phase 4: Evaluation of proposals and funding recommendation to the Ministery Phase 5: Final decision about funding by the Ministery

9. C Two Questions Concerning IGF Programme Design C.1 Question 1: „Bottom-up“ vs. „Top-down Approach“ (I) A) The traditional programme “philosophy” and institutional construction • Bottom-up: the IGF decide themselves about relevant technological fields and research projects according to the principle of self-organization B) Alternative approach: • Top-down: the “relevant” technological fields could be defined by expert bodies working on behalf of the Ministry What is the optimal solution: A or B or a mixture of both?

10. C Two Questions Concerning IGF Programme Design C.2 Question 1: „Bottom-up“ vs. „Top-down Approach“ (II) Clear strengths of the bottom-up approach • Good chances for identifying real needs of SMEs in branches where industrial research associations are active • The university institutes are bound to look for active and passive SME support for their research • Incentives for promoting transfer activities are immanent to IGF structures

11. C Two Questions Concerning IGF Programme Design C.3 Question 1: „Bottom-up“ vs. „Top-down Approach“ (III) “Bottom-up” in practice: project portfolio • Sectoral project distribution: large part of project funding for “old” industries (estimated 40% over the last ten years) • Strong weight of traditional technologies • Up to the 90th no high-tech-projects, since 2002 high-tech-projects make about 10 to 15 % of the total budget “Bottom-up” in practice: institutional structures • The challenge of combining well functioning institutional structures with flexibility

12. CTwo Questions Concerning IGF Programme Design C.4 Question 1: „Bottom-up“ vs. „Top-down Approach“ (IV) New developments in IGF: • New research associations were established in the 90s, specialized in high-tech and cross-sectional technological fields • A portion (presently about 15 %) of financial means for high-tech projects is reserved • The traditional key of distributing funds to the research associations is going to be abolished

13. The logic behind IGF: Limited financial, material and personal resource base of SMEs leads them to “underinvestment” in R&D IGF as compensation for “structural weaknesses” of SMEs by finding solutions for technical problems which arise in SME production processes by inspiring SMEs to engage in R&D activities themselves by creating innovation and research networks between SMEs C Two Questions Concerning IGF Programme Design C.5 Question 2: Target Group SMEs (I)

14. SME participation revisited: The IGF definition of “SMEs” is rather broad (turnover < 125 Mio. €, < 500 employees) compared to EU definition An increasing part of dependent SMEs Securing an adequate share of SMEs in IGF bodies and project advisory boards seems to be a problem The question whether IGF projects strengthen the innovative capacity of SMEs has to be convincingly answered The basic concept is to questioned: What is the target group? - Isolated, single SMEs or SMEs in industrial innovation systems? C Two Questions Concerning IGF Programme Design C.6 Question 2: Target Group SMEs (II)

15. CTwo Questions Concerning IGF Programme Design C.7 Conclusion Evaluation task: Do the new regulations improve IGF performance by • strengthening the part of modern highly developed technologies in the project portfolio? • better enabling SMEs to take actively part in the IGF system?

16. D Evaluation IGF: the challenge D.1 Identifying economic project results Complications • Project results are by definition (and in practice!) a public good, that means usable by everybody • Proof of concrete implementation activities of IGF project results in single firms only is of limited value for evaluation • No firm which uses IGF results has to render an account of this • Results become part of the general body of technological knowledge, they may not be identifiable as IGF results by users • A time lag between closing date of IGF projects and practical uses reaching from 3 to 10 years

17. D Evaluation IGF: the challenge D.2 Identifying the “relevance” of projects for SMEs Complications • The idea that projects are “relevant” because they have been carried out under the IGF regulations is to be questioned • Bottom-up has to be weighted against the factual selectivity of the system • articulation of SME needs depends on the presence and power of research associations and • the engagement of research institutes which are specialized in the respective technological field • Comparing economic relevance of different innovations

18. E Our Evaluation Design E.1 Levels of Analysis and Components Micro Level: • Investigating genesis, realization and application of research results on the project level Meso Level: • Analyzing the activities of IGF actors and bodies: • industrial research associations • the role of AiF and its central bodies • research institutes which participate in IGF projects • IGF projects in the framework of sectoral innovation systems Macro Level: • Comparing IGF with other technology programmes oriented towards SMEs • Analysing the IGF project portfolio in the framework of the new and old programme design

19. E Our Evaluation Design E.2 Instruments (I) Interviews: • Expert interviews with: • Representatives of (all) industrial research associations (over 4 years) • Persons in charge of single projects, selected by chance • Independent experts evaluating project proposals and results • Representatives of central IGF bodies Surveys: • Different enterprise surveys: • General survey of industrial firms with respect to their knowledge on and estimation of IGF • Project oriented surveys with respect to estimated relevance of the projects and the use of project results

20. E Our Evaluation Design E.3 Instruments (II) Case studies of the role of IGF in sectoral innovation systems: • The selected sectors represent: • traditional (but developed) technologies like food and furniture industries • far developed technologies but well established industrial products (like mechanical engineering industries and the automotive sector) • high-technology fields or high-end cross-sectoral technologies Quantitative analysis: • Econometric Analysis of IGF programme data and the results of our surveys • Comparing the participation of SMEs in different technology programmes, as far as relevant data are accessible

21. Thank you for your attention!