Presentation Outline

1. Shanghai International Program for Development Evaluation Training Oct. 22– Nov.2, 2007, Shanghai, China Country-led Joint EvaluationDutch ORET/MILIEV Programme in ChinaChen ZhaoyingNCSTE , ChinaHan Jun NCSTE, ChinaHans Slot IOB, The Netherlands

2. Presentation Outline ■A brief note on Country-led Joint Evaluation ■The Evaluation initiation, governance structure, methodology, implementation and dissemination ■ Issues for discussion

3. Joint Evaluation Definition DAC Glossary defines joint evaluation as: An evaluation to which different donors and/or partners participate. Four categories of joint evaluation ■ Donor + Donor Donor + Partner Country ■ Multi-Donor + Multi-Partner ■ Partner + Partner

4. Joint Evaluation Overview ■Partnership in development leads to partnership in evaluation. Joint Evaluation has been on the international development agenda since the early 1990s. ■Country-led Joint Evaluation of ORET/MILIEV Programme in China is regarded by DAC Network as one of the important joint evaluation cases since 1990.

5. Country-led Evaluation Overview of CLE ■Ownership in development leads to ownership of evaluation. ■ The World Bank, UNDP, the DAC and some donor governments have been developing approaches to promote CLEs ■ The majority of evaluations of development aid programmes are still donor evaluations ■ What is needed are evaluations from the perspective of the partner country. CLEs will provide that.

6. The Country-led JointEvaluation in China Dutch ORET/MILIEV Program in China ■About the title of the evaluation ■ The recipient country leads the process, supported by the donor and both sides are jointly responsible for the evaluation The evaluation covers the period 1991-2003. A total project portfolio of 84 projects were approved. 40 projects were finished and 44 were ongoing. Of these 44, 17 completed the transaction but still had to produce the final report. The grant amount of the transaction is about € 200 million, the total transaction amount of the evaluated portfolio € 470 million.

7. How is the Evaluation Initiated? Dutch ORET/MILIEV Program in China ■After the experiences of joint evaluation of aid development, NCSTE would like to have a more pronounced role in the development evaluation and IOB would like to be involved in a recipient-led joint evaluation. ■The joint evaluation was co-initiated by IOB and NCSTE.

9. The Evaluation Aims to Dutch ORET/MILIEV Program in China Assess to what extent the programme (through its activities / projects) has fulfilled the policy objectives, needs and priorities of the Netherlands and China Verify whether the funds have been appropriately and efficiently used Provide information for both the Chinese and the Dutch that could be used to improve the programme, as well as for policy formulation

10. Evaluation Criteria Dutch ORET/MILIEV Program in China ■The relevance of the Programme ■The efficiency of the Programme ■The effectiveness of the Programme ■The impact of the projects implemented under the Programme.

11. How is the evaluation organized? Governance Structure An equal governance structure was established for this particular type of evaluation. ■The Steering Committee (SC) Make decisions on the evaluation ■Team Leaders (TL) Organise the fieldwork and other tasks ■The Reference Group (RG) Provide advice and support

12. How is the evaluation conducted? Four steps of the joint evaluation ■ Design/TOR ■ Setting up & organization ■ Implementation & Final Draft Report ■ Report dissemination/Following up

13. How is the evaluation conducted? Evaluation Design/TOR TOR are even more important for the success of the joint evaluation ■TOR describe the overall evaluation and establish the initial agreements prior to the work plan ■TOR process establishes the basic guidelines so everyone understands the context of the evaluation ■WhetherTOR is the product of collaborative process is an important indicator for assessing the degree of cooperation in joint evaluation. ■IOB and NCSTE jointly draft the TOR.

14. How is the evaluation conducted? Evaluation Design/TOR Identifying theKey Issues in Writing TOR ■Focus on key issues to be addressed by the evaluation ■Avoid too many issues. It is better to have an evaluation that examines a few issues in-depth

15. How is the evaluation conducted? Key Issues In TOR Policy Relevance ■In 1999 the Chinese government launched the Western Development Strategy. Has the number of projects approved in this region changed since 1999? ■Are the projects listed as priority projects in the local development plans? ■Are the projects in line with the Netherlands development policy? ■To what degree are government officials, on-lending banks, suppliers, window companies and end users well informed about the objectives of the programme?

16. How is the evaluation conducted? Key Issues In TOR Efficiency ■ Have the projects achieved their expected outputs on schedule and within budget? ■ Have the projects experienced delays during implementation? If so, what were the main causes of these delays? ■ Has the condition that 60% or more of the transaction should be sourced in the Netherlands resulted in higher prices? ■ Are the procedures for project application, appraisal and approval considered reasonable by the end users, the organisations that receive ORET/MILIEV funds?

17. How is the evaluation conducted? Key Issues In TOR Effectiveness ■ Has the programme facilitated access for Dutch suppliers to China’s local and/or national markets? ■ Have the projects created skilled job opportunities in China? ■ Is the equipment still operational and maintained after the completion of the projects? ■ In the opinion of the end users, are the projects sustainable in terms of maintenance, after-sales services and the availability of spare parts?

18. How is the evaluation conducted? Key Issues In TOR Impact ■ Are the projects being replicated elsewhere? ■ Have the projects had a negative impact on the environment, the poverty situation and/or on the situation of women? ■ Has the programme resulted in an increase in trade between China and the Netherlands? ■ In the opinion of the end users, what have been the impacts of the project activities?

19. How Is the Evaluation Conducted? Evaluation Methodology ■Desk Study: Cover all the 84 projects ■Field Visits: 35 projects were chosen, 19 finished and 16 ongoing projects ■Questionnaires: 68 were returned from end users and 60 from suppliers ■Stakeholder Workshops: 17 provincial-level workshops were held during the field visits. More than 300 stakeholders were invited to participate in the workshops ■Cross-verified & In-depth Analysis:The information in the fact sheets and the evidence collected from the field visits were cross-verified to support in-depth analysis; the database of initial evidence for the 35 visited projects has been built

21. The Responses from Stakeholders ■The responses from Chinese stakeholders ■The responses from Dutch stakeholders ■ The responses from other development agencies ■ The policy reaction in China and the Netherlands

22. The Dissemination of the Evaluation ■ The dissemination has been given attention from the beginning. ■The final evaluation reports published in both Chinese and English ■ The dissemination workshop in Beijing China, March 29, 2007 ■Communicate this evaluation case at Sixth Meeting of the DAC Network on Development Evaluation, 27-28 June 2007, Paris.

24. Issues for Discussion ■The key benefits of the evaluation ■The limitation of the evaluation ■ Potential challenges of the country-led joint evaluation ■Making the decision to undertake a joint evaluation

25. The key benefits ofthe Country-led Joint Evaluation ■ Offer great opportunities for mutual capacity development ■ Increase ownership, the evaluation reflects better the voice of the partner countries ■ More relevant to the needs of both sides ■ Sharing the burden of work of evaluation

26. The limitation ofthe Country-led Joint Evaluation Institutional limitation in China ■ A national evaluation system has not been set up ■ No one in-charge of the official policy reaction ■ Lack of special budget for evaluation

27. The Potential challengesof the Country-led Joint Evaluation ■ The biggest challenge comes from different evaluation cultures and evaluation systems ■ Ensure that the evaluation meets the needs of both countries ■ Build consensus to design and conduct evaluation (such as: Identifying the key Issues in writing TOR, choice of field study cases) ■ Evaluate the impact of the program

28. Environmental issues in Dutch ORET/MILIEV evaluation 1-Introduction: ■ -The objective of the Dutch ORET/MILIEV Programme: development and environment related, but export transactions are mostly focused and stressed (equipment export; mix-credit + grant). ■-Before the approval, at the appraisal stage, all projects are requested to go through environmental impact assessment with the basic principle of Not to harm the environment - positive or neutral would be OK. The result of environmental ex-anti test: almost half for positive; half for neutral. ■-Among the portfolio, only 18 projects are focused on environment issues or environment orientated (21% of 84 total projects). Therefore, ORET/MILIEV programme is not so strong on environment issues.

29. Environmental issues in Dutch ORET/MILIEV evaluation 1-Introduction: ■ -Among the 18 projects,7 environment orientated projects were selected as cases, and visited on site during the field missions. These 7 projects are related to waster water treatment, solid waste treatment, environmental master planning, integrated mangrove management, banknote destroying system etc. ■ -Difference of outcome or impact evaluation between the environment focused projects and other projects-direct or indirect, intended or unintended impact effect. Because of the characteristics and objectives, the environment orientated projects for sure will have, to some extent, direct and intended environmental contribution indicators as well as the impact. While other projects are not necessary having influence on environment. Their impact, if there are, might be in an indirect and unintended way, but it is more difficult to measure the impact of these kinds of projects.

30. Environmental issues in Dutch ORET/MILIEV evaluation 2-Limitations and difficulties to assess environmental impact: ■ -The environmental issues are so complicated that it’s rather difficult to measure their outcome and impact than normal projects. Environmental impact evaluation needs appropriate and useful approaches. Although some environmental questions are discussed and analysed, especially in relevance, effectiveness and impact, environment sustainability are not related or concerned in this evaluation. At TOR and evaluation procedure, no specific indicators were designed for environmental issues. We use the same evaluation criteria for all projects. ■-A measurable impact needs available information and comprehensive statistic data- due to the absence of suitable project-level data and related information, it has been impossible to assess the environmental issues at sector/programme level in this evaluation. Especially, it’s not easy to get environmental statistic data in China, national wide and locally. Also it’s time and money consuming if you want to collect these complicated data. On the other hands, Chinese monitoring system is rather weak.

31. Environmental issues in Dutch ORET/MILIEV evaluation 2-Limitations and difficulties to assess environmental impact: ■-Quantitative and qualitative evaluation-it is easier to make a judgment of positive, neutral or negative impact for a project in a qualitative way, but it is difficult to measure impact in a quantitative way. At present, some countries like the Netherlands and some international organizations like World Bank have undertaken several research and impact evaluation on poverty reduction. It seems more quantitative indicators are included. But how about the environmental impact evaluation? Shall we have more quantitative impact indicators?

32. Environmental issues in Dutch ORET/MILIEV evaluation 2-Limitations and difficulties to assess environmental impact: ■ -Attributions - improvements in water and air quality can be the output or outcome of a project, but are there other factors that contribute to the improvement? How many people have been affected by such changes? And to what extent? Even among the 7 environmental orientated projects, it’s easier to assess waste water treatment project because its goal and outcome indicators are clear and designed at the very beginning.For example, we can get these kinds of data like how many cubic meters of waste water treated daily? How many tons of sewage reduced when discharging into nearby rivers? How many local people benefited from this project? But others are difficult to analyze their contributions to environment.

33. Environmental issues in Dutch ORET/MILIEV evaluation 3- Some findings related to 7 environment orientated projects: ■ -Overview -The 7 environment orientated projects have achieved (or highly achieved) the objectives and having positive impact on environment. For the whole project portfolio, only one project was found having negative impact on environment. Negative impact -The Refined Potato Starch Processing Project in Hebei Province. According to the contract, the end user must build a waste water treating facility, but the factory failed to do so and discharged the waste water directly into a reservoir. Except for water pollution, this project have very positive impact on social, economic and poverty alleviation. Debate: is it a good project if the end user will install the necessary waste water treatment equipment in future?

34. Environmental issues in Dutch ORET/MILIEV evaluation 3- Some findings related to 7 environment orientated projects: ■-Demonstration - 4 of the 7 projects were assessed having demonstration effects. Demonstration usually presents good practice or successful experience. For example, the demonstration of the advanced environmental technology-Dutch Carrousel-2000 technique in waste water treatment plant. Over 300 people visited Beijing Shunyi Waste Water Treatment Plant (WWTP) for learning and training. And also, the management of mangrove forest in Guangdong Province demonstrated to local farmers community on how to protect the forest and how to manage biodiversity issues, etc.

35. Environmental issues in Dutch ORET/MILIEV evaluation 3- Some findings related to 7 environment orientated projects: ■-Replication - Chinese stakeholders as well as the evaluation team hold the strong opinion that ‘replication’ should be an important indicator when measuring impact; 3 pilot projects were found having been replicated, either in other part of China or in Asian countries. When destroying old or worn banknote, in order to avoid air pollution by burning and water pollution by re-processing, The People’s Bank of China introduce 19 sets of banknote disintegration and briqueting machines. Later on, the Bank ordered additional destroying systems from the same Dutch company - Kusters BV and replicated this in several provinces. Moreover, the small family-owned Dutch company Kusters expanded its business rapidly in Indonesia, Philippines and Pakistan, and more projects were replicated in these counties. Sometimes replication may combine with demonstration: Beijing Shunyi WWTP also received over 30 potential Chinese end users for consultancy. As a result, the Dutch DHV company have replicated about 20 WWTPs in other provinces, using same Dutch Carrousel -2000 technology.

36. Environmental issues in Dutch ORET/MILIEV evaluation 3- Some findings related to 7 environment orientated projects: ■ - Seed Money - After the implementation of the Environmental Master Plan of Yunnan Plateau Lakes, Japanese government used the result of the plan and invested $60 million in this province to enhance the protection of lakes. Also, World Bank has been involved in other environmental projects thereafter.

37. Making the decision to undertake a Country-led Joint Evaluation ■Evaluation Culture. Understand the differences in the evaluation cultures and systems in both sides ■Audiences. Make it clear that who will be the main audiences of the evaluation in both sides ■Jointly draft the TOR. TOR is even more important for the success of the joint evaluation. ■Timeframe. If one is in a great hurry to get an evaluation completed, the first time of joint work may be disadvantageous.

38. Listen toyour Opinions and Advice ■What do you think of this joint evaluation? What are the advantages and weaknesses? ■How to promote the country-led joint evaluation as a successful evaluation model? ■What are the major barriers to the country-led joint evaluation ? Lack of demand or weak evaluation capacity ? ■ What is Donor’s role in CLEs, particularly in the “first generation” of CLEs?

39. Thank you Any Questions, Opinions & Advices……