Evaluation of large Scale Industrial projects

1. Evaluation of large Scale Industrial projects Evaluation Units’ Open Days Rome, 3/06/2006 Tito Bianchi Public Investment Evaluation Unit (UVAL) Department of Development Policies

2. INDEX OF THE PRESENTATION 1. Evaluation mandate 2. Features of the policy to be evaluated 3. Evaluation Method 4. First results

3. 1. Evaluation mandate I: Debate • Mounting criticism of subsidies to private firm investment • Declining resources for industrial subsidies • Negotiated/contractual approach to large scale industrial promotion disputes resources to “objective-automatic” subsidies • Drafting of new 2007-2013 CSF

4. Evaluation mandate II: institutional request for Evaluation In 2003 CIPE - Ministerial Committee for economic planning - debates over the effectiveness of subsidies to firms… … asks UVAL and the network of evaluation Units to conduct an evaluation of incentives for industrial promotion “to base future decision-making” (Decree N.16/03) …UVAL chooses to evaluate Contratti di Programma (Negotiated contracts w/large firms)

5. 2. Features of the policy to be evaluated Contratto di Programma (CdP): Bi-lateral agreement Central Govt  Large industrial firm Subsidizes Firm’s Investment Plan Commits to: Employment levels Research activities Destination of equipment

6. Funded by domestic fund for lagging regions Started 1987, continuing to present Originally funded only large scale industrial firms.. .. Recently extended to agri-food and tourism ....Extended to consortia of smaller firms; 2. Policy Features cont’d: “Contratti di programma” (CdPs)

7. Policy Features: Number, size and sector CdPs approved at december 2005 In present generation, Food-processing chains and tourism consortia are the majority. 40 such contracts are only approved by CIPE, not yet signed.

8. Lack of a transparency of rules along the pipeline of ex-ante evaluation, approval Timing uncertain, proposed CdPs are never rejected Selection criteria specified allow large discretion Policy features: selection process unclear

9. Regions have started to fund similar projects, claim this policy tool is in their responsibility Attraction of foreign firms remains a priority in the new draft CSF Eastern EU outcompetes Italy in terms of labor costs and higher State Aid Ceilings What should be the future of CdPs “negotiated-discretionary” industrial subsidies?

10. 3. Evaluation Method: Technical Steering Committee A 8-member steering committee appointed itself including representatives of UVAL, the Network of evaluation Units, the Ev. Unit of the Ministry of Industry, 1 expert of industrial policy. …it was put in charge of designing evaluation activities, conducting the evaluation, reporting to CIPE

11. 3. Evaluation Method: Specifying research questions • Impact Evaluation was specified as: • Additionality; would the private investment have been done w/out subsidies? Of the same size, in the same place? • Was there a sustained economic benefit for the area receiveing the investment? • What was the national level impact (on the industrial sector)? • Relative impact of CdPs involving consortia vs. Individual large firms.

12. 3. Evaluation Method: choice of an Intermediate option Number of Cases In depth Case-Study 0 .. 5 ... 10 ... hundreds... thousands Quantitative study Comparative case-based evaluation

13. 3. Evaluation Method: components of the evaluation project • Main study: comparative study of 9 cases of CdPs, based on data, documents and interviews • Additional case-study (FIAT) – outlier largest and most studied – based only on secondary sources • Data analisys of recent CdPs funding consortia, testing internal coherence of projects

14. 3. Evaluation Method: choice of cases for the main study, pragmatic segmentation • We picked among those (~40) that had finished the investment plan (first 2 genr.) • Included two Consortia, the first to be financed • Covering 7 regions of the South • Eight diferent Industrial sectors: Electronics (2), Textiles, Shoes, Auto-parts, Food-processing, Oil refining, Jewelry, Chemicals

16. 3. Evaluation Method: contracting evaluation experts • hired 9 experts in the evaluation of industrial policies through a competitive process: • Publication on DPS website of call for bids • Experts fees set at € 20.000 for each study • Candidate experts were evaluated based on CV and on a two-page proposal • Each selected to evaluate specific identified CdP • Common meetings have discussed preliminary evaluation findings, shared methodology

17. 3. Evaluation Method: Comparative Analisys • We did not impose paragraph structure but asked to answer the selected questions • Asked to calculate minimum set of indicators: firm, contract, and territory. • Encouraged to pursue non-considered research hypotheses, calculate additional indicators • Currently steering group is writing comparative part

18. CdPs fare better than the reputation they have Research activies of firms often discontinued, they were unrealistic or non serious Manufacturing plants can take root in the area, but never had strong generation effects Employment goals, especially the purpose to mitigate localized employment crises, are weakening factors 4. Preliminary Findings

19. Skills for the evaluation of industrial policies are not abundant Windows of opportunity to evaluate policies arise, have to be timely exploited Managing a comparative multiple-study evaluation of this kind has advantages and disadvantages 4. Lessons about evaluation

20. 4. Lessons for Evaluation: ++ and -- of comparative case-study method + _ • Involves larger number of independent intelligences • The process itself, not only the final product, contributes to a shared view • Virtuous competition b/ween experts • Risk to Lose control of timing • Pursuit of consensus b/ween different views may reduce the sharpness of the method, of the message