1 / 19

What a Realization: An Ex Post Impact Evaluation of a Performance-Based Program (2002/2003 SPC Evaluation)

What a Realization: An Ex Post Impact Evaluation of a Performance-Based Program (2002/2003 SPC Evaluation). MAESTRO Evaluation Showcase July 26-27, 2006 Project Manager: Pierre Landry, SCE Lead Consultants: Mike Rufo, Itron; Keith Rothenburg, Southern Exposure Engineering; KEMA Inc.

gayora
Download Presentation

What a Realization: An Ex Post Impact Evaluation of a Performance-Based Program (2002/2003 SPC Evaluation)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. What a Realization: An Ex Post Impact Evaluation of a Performance-Based Program(2002/2003 SPC Evaluation) MAESTRO Evaluation Showcase July 26-27, 2006 Project Manager: Pierre Landry, SCELead Consultants: Mike Rufo, Itron; Keith Rothenburg, Southern Exposure Engineering; KEMA Inc.

  2. Overview of Presentation • Evaluation Context and Background • Approach • Results and Key Findings • Recommendations

  3. Context: Custom, Bidding, and SPC Type Programs • History and types • custom rebates, bidding, SPC • Goals • resource acquisition and market transformation • Common features • focus on custom efficiency measures in large C&I • encouragement of comprehensive projects • inclusion of technical engineering review • requirements for proof of project installation

  4. Common Issues for Large C&I Programs • Uncertainty in savings estimates • Risk of gaming and fraud • Costs of measurement and verification • Keeping application requirements manageable yet effective • Distributing funds equitably • Minimizing free riders/maximizing net impacts • Supporting the efficiency services market

  5. Background on 2002-2003 CA SPC • 3 IOUs (PG&E, SCE, SDG&E) • Targeted at Large C&I • Incentives by end use • Lighting ($0.05), HVACR ($0.14), Process/Other ($0.08) • Calculated or Measured Savings Path • Determined by utility • 90+% calculated • All projects require Site Installation Report

  6. Why An Ex Post Evaluation of “Performance Contract” Program? • PY2002 & PY2003 first years SPC had independent ex post impact evaluation • Originally (’98/’99), all SPC projects required in-program M&V • Initially, 2 years of M&V, then 1 year • MT policy environment in CA (1998-2000) • Virtually no attention to ex post impact evaluation • M&V dropped as requirement in 2000 • Utilities retained right to require • Most projects done under calculated savings path • Shift back to resource acquisition focus with energy crisis and 2006-2008 program admin decision

  7. Objectives Re-establish process for site-specific ex post impact evaluation Implement process for representative sample Estimate program realization rate Estimate net-to-gross Recommendations for improvement Constraints Total budget vs. program size, project complexity, # of utilities Per site budgets Limited sub-metering Took 2 years for most of sample to install projects Limited pre-metering available Self report NTGR method PY2002 SPC Evaluation Objectives and Constraints

  8. Approach • Develop and implement sample design (ratio estimation) • Obtain files & savings documentation • Review applications and prepare ex post analysis plans • Conduct on-site data collection (limited metering) • Develop ex post impact estimates • Prepare detailed, site-specific documentation • Carry out quality control review • Extrapolate final ex post results to the population • Estimate free ridership • self-report method • CADMAC protocol-level battery

  9. Population • Combined 2002-2003 SPC Population (apps): • Combined 2002-2003 SPC Population (kWh):

  10. Sample Design • PY2002-PY2003 SPC Sample (apps)PY2002-PY2003 SPC Sample (kWh)

  11. Gross Realization Rate Results

  12. Gross Realization Rate Results

  13. Freeridership and Net-to-Gross DEER SPC NTGR = 0.7

  14. Key Findings • PY2002-2003 SPC more like a custom rebate than performance contract program • Overall realization rate reasonably high but below one • Wide range in individual realization rates • Importance of influence of largest projects • Gross savings may not be as conservative as SPC program managers intended • NTGR slightly lower than 1998-2001 period

  15. Other Savings-Related Key Findings • Wide range in the quality of applications and supporting documentation • Need for increased verification and documentation of assumptions in project files • Experience and expertise levels of the reviewers vary widely • Difficulties in assessing complex industrial process projects • Limited estimation of kW peak demand savings

  16. Impact-Related Recommendations • Consider targeted increases in the level of technical documentation • Consider a stronger application affidavit statement regarding savings assumptions • Further standardize the review approach and documentation requirements for recurring complex projects • Consider providing or requiring more technical support for applicants for complex projects

  17. Savings-Related Recommendations (Cont.) • Improve reviewer documentation • Consider increasing: • conservatism for calculated path savings estimates • measurement for large complex projects • incentive premium for measured projects • Increase pre-installation measurement for very large projects with highly uncertain baseline conditions • Consider independent review of the SPC calculator

  18. Free Ridership-Related Considerations • Increase efforts to reduce free ridership • Consider: • Higher incentive levels for higher payback measures or emerging technologies • Incorporating a payback floor • Bonus payments for first-time participants • Custom baselines for process improvements • Excluding projects that are obvious free riders

  19. Evaluation-Related Recommendations • Consider shifting to ex post impact evaluations from program-year to paid-year basis • or a combination • Expand scope of ex post measurement • more measurement per site • more sites in samples • incorporate uncertainty in ex post estimates into program realization rate confidence interval • Integrate evaluator early into program process to enable pre-measurement where necessary

More Related