Managing complexity: methodological challenges for determining attribution and contribution

1. Managing complexity: methodological challenges for determining attribution and contribution Per Mickwitz Research Director, Professor Finnish Environment Institute (SYKE)

2. Helsingin Sanomat (26 January 2011) The car taxation reform has increased the popularity of low-emission cars EU goal 2015 To which degree is the change due to the tax reform?

3. The structure of the presentationManaging complexity: methodological challenges for determining attribution and contribution • The nature of the complexities involved • Complex multi-level governance settings, where policies are formed and implemented • Complex socio-technical systems that policies tries to alter • Complex interdependencies between humans and nature • Some ways to approach the challenge of attribution • Triangulation • Experimentation and RCT • Program theory • The conclusions when evaluations find no or little impacts of policies

4. Part 1. The nature of the complexities involved

5. The Finnish car tax reforme Many policies at many levels • EU: • voluntary agreements with the car industry • a requirement to provide information to consumers about CO2 emissions (1999/94/EC) • Finland • Increasing fuel taxes • Helsinki & other municipalities • Public transport • Land use planning EU goal 2015 To which degree is the change due to the tax reform?

6. The Finnish car tax reforme Many other factors than policies • a financial crisis • large SUVs – at least temporarily – went out of fashion EU goal 2015 To which degree is the change due to the tax reform?

7. Our understanding of Government in its context - “from steering to influence”… • The context where • Government take place • Globalization … • Information overload • What Government do • From hierarchy • to networks • Partnership… Complex multi-level governance“From Government to Governance”

8. Global level Policies & Regulation The Economy Knowledge The Environment Complex multi-level connections Local level

9. Global level Policies & Regulation Complex multi-level connections • International (UN, IMO, …) • Federal (EU, USA) • Country – State (Finland, The UK,…) • Region (Kymenlaakso,…) • Municipal (Helsinki, Stockholm,…) Local level

10. Some Features of Climate Change • The physical processes behind climate change are complex and uncertain • economic, social and political aspects add to the complexity • Long time frames • over a decade before an addition of methane is removed from atmosphere, • CO2 may persist up to 200 years • lifetime in the atmosphere of perfluoromethane more than 50,000 years • the time taken for temperatures to rise due to CO2 > a century • Because of the large heat capacity of the oceans, sea-level rise is expected to continue for centuries even if the climate were to stabilize • Any action against climate change will be largely local but the effects of climate change will be global, different regions will be affected in very different ways. • In 2000, the per capita CO2 emissions in the US were 20.6 metric tons, • the Finnish per capita emissions were 12.0 metric tons, • India were only 1.1 • Ethiopia a mere 0.1 metric tons per person

11. Sustainable transitions of socio-technical systems • The world is not more sustainable now than in 1992 & there is an urgent need for action • Fundamental transformation of the socio-technical systems through which basic services are produced and used • Food, Housing, Energy, Mobility, Clothing, … • At the same time the perception of sustainable development has changed: • from a planning challenge • to stressing complexity, wicked problems and resilience.

12. The interdependence between the socio-technical systems and nature • Interdependence at all scales: e.g. nano, genes and climate

13. Part 2. Some ways to approach the challenge of determining attribution

14. The impact problem for environmental policy instruments, programmes and projects • Many bad ex post evaluations (at least in Finland), because they do not address other explanatory factors, i.e. they attribute the total change to the evaluand. • Also in ex ante evaluation too little effort is often put on exogenous change, even when trends are well known.

15. Addressing the impact problem • No single clear and universal solution • Often several methods instead of just a single one. • Bartlett (1994, p. 183): "Clearly desirable are multiple evaluation, done with a keen appreciation of the strengths and limitations of each approach and a frank recognition of the advantages of others."

16. Often the term “triangulation” is used • Four types of triangulations can be distinguished: • multiple methods; • multiple sources within one method; • multiple analysts; and • multiple theories.

17. Can be combined with information on technological investment, e.g. Helsinki Energy, Hanasaari B 3+4 installed a flue gas desulphurisation (FGD) unit in Jan. 1992 Finding the effects of a policy, example ELV of 230 mg/MJ Triangulation by Mickwitz, Kivimaa & Attila

18. Finding the effects of a policy, Triangulation & the counterfactual case • Triangulation is not just about combining data & methods, but also perspectives. • Several perspectives can help to “question the frame” • Being open to different perspectives is even more challenging than utilising several methods.

19. Triangulation in practice “Norms” “Binding permit limit” Managing evaluations with multiple analysts and theories is not easy!

20. Finding the effects of a policy, Triangulation & the counterfactual case – Example air pollution Both assessments are made based on aggregated data and based on the model of the state as a unitary rational actor (Levy, 1995; Skærseth, 2003): • states are unitary and rational actors; • decisions are based on maximising costs and benefits to the nation; • states have full control of their societies • The UNECE Convention on Long-range Transboundary Air Pollution is a success. • SO2 emissions have decreased • Reduction target of – 30 % was achieved • Scott Barrett (2003) 'Helsinki's failure' • YES many countries reduced SO2 emissions by 30 % • BUT, countries that did not sign the protocol, also reduced their SO2 emissions by more than 30 % • Over-compliance due to national policies or other factors, e.g. restructuring of energy sector.

21. 20 18 16 IMO rule 14 12 SPECIFIC NO EMISSIONS (g/kWh) 10 x 8 6 4 2 0 200 400 600 800 1000 1200 1400 1600 1800 2000 0 RPM Finding the effects of a policy, The counterfactual case – Example Marine NOx emissions Low NOx Combustion • On the agenda 1988 • Proposal 1990 • Adoption 1997 • 15th ratification 2004 • Into force on May 19th, 2005 Over compliance = No policy effect? Slow implementation = Limited effectiveness or Precondition for effectiveness? Direct Water Injection Selective Catalytic Reduction

22. International Policy EU Policy National Implementation 2 Outcomes Example: Marine NOx emissions National R&D support • Low NOx Combustion(25 - 35% reduced NOx) • Today standard in all Wärtsilä engines • Direct Water Injection (50 - 60% reduced NOx) • Development started 1990 • First engine in use in January 1999 • In February 2003: 53 delivered & ordered • Selective Catalytic Reduction(85-95% reduced NOx) • In February 2003: 60 delivered & ordered

23. International Policy EU Policy National Implementation 3 Outcomes Example: Marine NOx emissions • Swedish ports have differentiated harbour & fairway fees based on NOx emissions National Implementation

24. International Policy EU Policy 4 Outcomes Example: Marine NOx emissions National Implementation

25. International Policy EU Policy National Implementation 5 Outcomes Example: Marine NOx emissions • Then EU started to prepare action: • Commission of the European Communities. A European Union strategy to reduce atmospheric emissions from seagoing ships. Communication from the Commission to the European Parliament and the Council. COM(2002) 595 final, Brussels, 20 November 2002. National Implementation

26. International Policy EU Policy National Implementation Outcomes The top-down implementation modelMarine NOx emissions 1. International Policy 4. 2. 7. EU Policy National Implementation National, Regional, Local Policies 5. 3. & 6. Outcomes

27. 20 18 16 IMO rule 14 12 SPECIFIC NO EMISSIONS (g/kWh) 10 x 8 6 4 2 0 200 400 600 800 1000 1200 1400 1600 1800 2000 0 RPM Finding the effects of a policy, The counterfactual case – Example Marine NOx emissions Knowing the outcomes is not enough, An understanding of the process is required Low NOx Combustion • On the agenda 1988 • Proposal 1990 • Adoption 1997 • 15th ratification 2004 • Into force on May 19th, 2005 Over compliance = No policy effect? Slow implementation = Limited effectiveness or Precondition for effectiveness? Direct Water Injection Selective Catalytic Reduction

28. Many approaches to assess impacts: Peer review • In peer review, the essential feature is expert judgement – often based on visits and interviews – by colleagues. • Often called 'quick and dirty' because it is inexpensive and relatively fast (Chambers, 1981). • E.g. the OECD's Environmental Performance Reviews are based on peer review. • While the method may be very useful, it involves several serious risks related to lack of contextual and historical understanding and the reinforcement of general truths rather than creation of new knowledge.

29. Experimental, quasi-experimental evaluations & Randomized Controlled Trials • One approach that is subject to a great deal of discussion in evaluation is experimental or quasi-experimental evaluations (Donaldson et al., 2009). • These approaches have also been discussed and promoted in the environmental context • Can be very useful • But are often not possible to use in practice because of • to complex bundles of policies to design credible experiments • Legal limitations, it would be against the constitution to make experimental designs with car taxation in Finland

30. Program/Intervention theories • An intervention theory can be seen as a model “of the micro steps or linkages in the causal path from program [or, more generally, intervention] to ultimate outcome” (Rogers et al., 2000: 10) on the basis of the detailed assumptions of how the intervention is intended to work. • Chen (1990): “a specification of what must be done to achieve the desired goals, what other important impacts may also be anticipated and how these goals and impacts would be generated”. • Vedung (1997): “All empirical and normative suppositions that public interventions rest upon” The word ‘theory’ does not refer to scientific theory, it is used as the opposite of practice, referring to how an intervention should work, not how it actually works when implemented.

31. Uses of intervention (program) theories • Used to establish the anticipated effects of the instruments and the target areas of each instrument • Used to identify data to be collected • A set of detailed research questions • Used for communication, to make different views explicit • Intervention theories NOT theory

32. Target groups Outputs Inputs Outcomes The actors, i.e. decision making entities, such as companies, other organizations and individuals whose actions the policy is intended to influence. The assumptions about what the administration produces and the target groups are faced with. It is often easier to distinguish outputs from the internal administrative results if one approaches outputs from the side of the target groups. What the administration is supposed to use to produce outputs. Inputs include such resources as personnel and funding, but also matters coming from the target groups that the agencies take into account or respond to, e.g. a permit application or an investment plan. The actions expected and the consequences believed to follow the expected actions taken by the target groups in response to the outputs. Hardly any outcome is a result of policy outputs alone, but instead is affected by a variety of other factors as well. Key elements of the program theories

33. Output Target group Outcomes

34. SD-PAMs – Creating incentives to promote sustainable development with climate benefits as side-effects

35. Program/intervention theory as developed by Huey-Tsyh Chen (2005) Context Change model Intervention(s) and their specific outputs Action model Leverage mechanism(s) Outcome 1 … n Goals Implementing organization Institutional framework Target population

36. Interventions often fail!Why? • Policies may be based on erroneous assumptions (Hoogerwerf 1990) = Change model • Problems of implementation; what the local administration delivers is not exactly what was expected (Pressman and Wildavsy 1984) = Action model • Dynamic contexts, i.e. policies are implemented in circumstances different from those for which they were planned

37. Program/intervention theory as developed by Huey-Tsyh Chen (2005) Context Change model Intervention(s) and their specific outputs Action model Leverage mechanism(s) Outcome 1 … n Goals Implementing organization Institutional framework Target population

39. The level 2 change model of the South African June 2006 proposal

40. The intervention theory (program theory) or the target group may change when policies at different levels interacts • The EU Directive concerning Integrated Pollution Prevention and Control (96/61). • Based on the assumption that the main target group is large scale industrial factories.

41. The intervention theory (program theory) or the target group may change when policies at different levels interacts • The EU IPPC Directive was implemented in Finland through the new Environmental Protection Act (entered into force March 1st 2000) • Still based on the assumption that the main target group is large scale industrial factories. • But incorporated the target groups of all previous Finnish permit legislation, such as waste permits, water permits, air permits, etc.

42. Our evaluation, however, showed that during the two first years:

43. “EU has never managed to make a policy so complicated that the Finnish Ministry of the Environment has not been able to make it more complicated through domestic decisions”LeenaSaviranta June 16, 2006

44. This all sounded very complicated, but, evaluation is not about finding the final truth, it iscareful normative assessment,which is intended to play a role in future, practical action situations.

45. Part 3. The conclusions when evaluations find no or little impacts of policies

46. This is how I used to think • "...if environmental program evaluators fail to find evidence of environmental impacts, procedural change, or potential gains in efficiency, then something is wrong with the environmental programs, environmental program evaluations, or both." (Knaap and Kim 1998, p. 359)

47. Mobility systems • Socio-technical systems as constellations of technologies, material networks (the grid or roads), social networks, actors, organisations, and rules

48. The CO2 emissions from transportation and how they could be reduced: [g CO2] = [passenger-km] x [vehicle-km/passenger-km] x [MJ/vehicle-km] x [g CO2/MJ] Land use & Urban form Travel “needs” Transportation mode Engine types & driving habits Fuel type Car taxes Fuel Taxes

49. Path dependencies in Finnish transport -16 % By 2020

50. How I am thinking – part 1 • "...if environmental program evaluators fail to find evidence of environmental impacts, procedural change, or potential gains in efficiency, then something is wrong with the environmental programs, environmental program evaluations, or both." (Knaap and Kim 1998, p. 359) • Programs or policies should be assessed as bundles and the lack of impacts, might be due to the other policies