Market Transformation Programs: What they are. How to evaluate them.

1. Market Transformation Programs: What they are. How to evaluate them. Kansas Corporation Commission Mitchell Rosenberg, Vice President Topeka, Kansas March 26, 2008

2. Overview • Market Transformation: Definitions and Reality • Importance of MT Concepts in Energy Efficiency Program Design and Evaluation • Key Challenges in MT Evaluation • Steps in Meeting MT Evaluation Challenges: examples from actual programs & studies • Lessons Learned

3. Definition of Market Transformation • Market Effect: “a change in the structure or functioning of a market or the behavior of participants in a market that results from one or more energy efficiency program efforts.” • Typically, these efforts are designed to increase the adoption of energy-efficient products, services, or practices, and are causally related to market interventions. • Market Transformation: Market effects that persist once supporting programs are terminated

4. MT References the Product Life Cycle

5. Policy & Programs to Accelerate MT

6. EE Programs do help transform markets: The case of electronic ballasts Programs Underway

7. The case of resource-efficient clothes washers

8. Why MT Concepts are Important in Program Design & Evaluation • Program designs must be matched to current stage of development to be cost effective • Harnessing supply side motivations enhances cost effectiveness • High GHG reduction goals • Need continuous pipeline of energy-efficient products, services, design practices • Need to extend adoption of established products to all relevant market segments • In some cases, market research provides most fruitful approach to NTG evaluation

9. Key Challenges in MT Evaluation • Effects occur over a long time frame • Events of interest span decades • Program planning & evaluation cycles typically 3 years or less • Multiple program sponsors and policy makers are involved • Often coordinate activities among jurisdictions • How tease out effects of one state’s programs? • Data are difficult to obtain • Manufacturer shipment data often proprietary • Distributors, contractors, retailers, designers tend to be small, scattered, disorganized

10. Steps in Addressing the Challenges of MT Evaluation • Market Assessment/Program Logic Model • Program History Development: Market Presence Indicators • Selection of analysis objectives and strategy (ies) • Appraisal of results/Adjustments for next rounds • Close parallels to steps in program design and revision

11. Program Logic Model: Home Performance

12. Market Assessment/Program Theory • Key Objectives • Location on the product life cycle • Identification of key market actors & segments, motivations, barriers • Assess data availability for program management and evaluation • Get program planners, implementers, regulators on same page re: short & long term goals and expectations • Timing: Should begin in program design phase • Resources: Avoid over-investment • Secondary data & reports useful • Selected local primary research often very useful

13. Program History/Market Presence • Important to build reference for market effects and attribution analysis • Should include national as well as local events & trends over time • Data to capture • Program activities: marketing, training, incentives • Participation: customer and vendor, numbers and characteristics of participants and measures, timing, region • Efficiency levels supported v. baseline practice

14. Example: EVT clothes washer programs

15. Analysis Strategies • Market Effects Indicators • Market share: sales, prevalence of practices, fleet efficiency indicators (e.g. mean EERs, MPG) • Price trends • Changes in codes and standards • Attribution Analysis • Cross sectional • Time Series • Self-reporting • Historical • Best to attempt multiple indicators and attribution analysis approaches

16. Net Effects of Vermont CW programs:Cross Sectional & Historical Approach • Results of cross-sectional modeling • Dependent variable: State level Energy Star CW market share • Independent variables: • Program indicators • Customer Demographics & Energy Prices • Change indicator

17. VT clothes washer programs: What’s happening to local program influence • Methods • ‘No-program Area’ market shares rising faster than market areas • Market conditions • Impending federal minimum standard changes • Profitable product for manufacturers • Fair representation of local program effects?

18. CA commercial lighting programs: presence in the market

19. General Challenges from C&I Programs • Little available market share data • Complex ‘measures’ • Greater customer market segmentation • Multiple levels of supply chain with direct influence on project-level equipment selection and design • Multiple decision makers and criteria in one customer  Generally greater reliance on ‘weight of evidence’ and self-reported approaches than in residential.

20. Other Potential Approaches to Attribution or Baseline Development • Diffusion modeling • Delphi or other expert judging • Conjoint analysis of preferences for efficient substitutes for current products

21. Lessons Learned • Generate as detailed a story as possible • Know the history of the program and its relatives • Know the history of the market and technology • Know what other programs are doing • Develop the program logic with local stakeholders • Know the available data resources • Sampling, contents, collection methods • Know what others have done • Design data collection to the attribution strategy or strategies

22. Lessons Learned: Maximizing data opportunities • Quality of Indicators • Accuracy, face validity, bias (lack of same) • Replicability (ability to support historical or time series analysis) • Sample frame: captures full population, updated regularly, documented compilation • Sampling: keep it kosher and document it • Comparability (supports cross-sectional analysis) • Same definitions as data collected elsewhere • Capitalize on channels to reach national markets

23. The Bigger Picture • Play well with others • Markets addressed by local programs are national and international • Many program operators are heavily involved already – get to know them • Initiatives under way to procure sales & shipment data • Independent program influence? • Many local programs already coordinate operations, or are developed in explicit reference to each other (e.g.) codes & standards • Why try to tease apart effects?