Preliminary Strategies for BPA Evaluations

1. Preliminary Strategies for BPA Evaluations March 24, 2008

2. Objectives and Agenda • Goals: • Inform RTF of preliminary evaluation strategies • Receive feedback on approaches • Provide opportunity for regional collaboration • Evaluation Strategies: • Commercial New Construction: ESD Office Package • Energy Smart Grocer Evaluation Strategy • DHP (brief, time-permitting)

3. Commercial New Construction: ESD Office Package • In 2007 RTF approved CNC Small Office package with savings estimated at 2 kWh/sqfoot • Requested evaluation of cost effectiveness of implemented program • BPA has developed a pilot program: Energy Smart Design Office Package • Intention of precursor to packages for other small commercial construction buildings • Evaluation strategy is an integrated process and impact evaluation

4. CNC – ESD Office Package:Process Evaluation • Process Evaluation • Process evaluation will be in the field early, to see how program is working; test the program logic model • Stakeholders to be interviewed include • Builders and designers (including non-participants) • Retail utilities • NEEA and Betterbricks program • Staff of BPA, RTF, Council and ETO • Timing: 6 months after program kickoff. Likely to provide an interim memo after the first one to three buildings are complete. Improvement feedback will be continuous.

5. CNC – ESD Office Package:Impact Evaluation • Primary goals of the impact evaluation: • Estimate savings and cost-effectiveness of program • Provide feedback to the RTF’s prototype models • Data collection: • Rebate requires: • Building architectural drawings, certification by the building design professional, cooling system cut sheets, window NFRC labels, and checklists for economizer commissioning, integrated design and lighting controls • For baseline cost data, evaluator will collect information during early process evaluation with stakeholders and sub-contractors • Some additional sub-metering may be required

6. CNC – ESD Office Package:Impact Evaluation, cont’d • Estimation of Savings • Limited options due to difficulty of baseline (no pre-bills, matching populations tricky) • To estimate savings per square foot for buildings: • Create an engineering-based energy use model of each building as it is built and operated • Calibrate model with actual metered data • Likely monthly bills, but sub-metering will be considered • Normalize for weather • Estimate consumption for standard efficiency baseline when program measures are “removed” • Difference between baseline and as-built represents the estimate of gross savings. • Extrapolating to the program level is problematic and therefore, the evaluation will strive for a near-census • Due to small sample size and heterogeneity of population, results are unlikely to prove/disprove RTF prototypes. Will provide indication of reasonableness.

7. Energy Smart Grocer: Background • In 2007, RTF reviewed measures for grocery stores; provisionally deemed some measures, reviewed GrocerSmart software of PECI • Requested an impact evaluation to estimate cost-effectiveness of program • BPA’s EnergySmart Grocer program targets businesses with refrigeration equipment (convenience stores, restaurants, specialty retailers, institutional kitchens and liquor stores). Implemented by PECI. • Currently, 66 customer utilities are enrolled in the program, including Snohomish PUD and Seattle City Light. • Offered measures can be categorized into four different types based on the type of ex-ante savings estimates: • No Savings: auto-closers and gaskets do not have proven energy savings • Deemed Measures: CFLs, other lighting measures, solid door freezers and refrigerators • Provisionally deemed. LED case lighting, anti-sweat heat controls, evaporator fan controls, vending machine controllers, evaporative motors in display cases and walk-ins, and night covers. • Deemed Calculated: condensers, cases, motors, multiplex compressors and controls, doors and suction line insulation.

8. Energy Smart Grocer: Process Evaluation • Process evaluation to assess strengths and weaknesses of program, including direct acquisition structure • Interviews will include: • BPA program and field staff • PECI program and field staff • Participating utility DSM and customer account staff • Small sample of end-use consumers (stratified by utility, customer type and energy savings) • Examples of topics to be addressed include: • Is the incentive structure effective and consistent with industry approaches? • Are utility customers satisfied with the program offerings and support services, including outreach, initial customer contact, audit results, audit follow-up, delivery services, PECI communication, quality control, measure installations, program comprehensiveness? • What aspects of program design/contracting led to administrative efficiencies or burdens? • Are end-use consumers satisfied with the program offerings and support services? • Are end-use customers signing up for an increasing amount of energy savings measures?

9. Energy Smart Grocer: Impact Evaluation • Two-stage impact evaluation to assess cost-effectiveness of program and estimate savings deemed calculated and provisionally deemed measures

10. Grocer Smart Evaluation:Stage 1 Impact Evaluation • Stage 1 is bottom-up, engineering based assessment of total program savings • No savings measures: Research is currently being conducted. Stage 1 savings will be consistent with the current RTF ruling on these measures. • Deemed measures: Installation will be verified, deemed savings values used. • Provisionally Deemed measures: Based on resources available, engineers will estimate as-installed savings of provisionally deemed resources and provide feedback to RTF. • Deemed Calculated measures comprised over 60% of program savings as of late 2007, including floating head pressure and floating suction controls. • Completed Project Review: An engineer will conduct thorough review of the inputs and calculations used in the GrocerSmart software for completed installations of deemed-calculated measures. On-site inspections will be conducted with approximately 10 sites, prioritized by estimated energy savings. • Targeted M&V: Sites will be chosen and monitored prior to measure installation. Pre-data collection for 2 weeks. Results provide estimates of measure-specific and overall site savings. Results will be compared with EnergySmart software calculations. • Costs will be collected from PECI. Cost-effectiveness analysis will be conducted and recommendations for improvement will be provided.

11. Grocer Smart Evaluation:Stage 2 Impact Evaluation • Impact evaluation must estimate total energy impacts and cost-effectiveness of the EnergySmart Program. • Performed using a statistical analysis of pre- and post-installation energy consumption characteristics. • Multiple methods may prove to be successful in estimating energy savings, including a fixed-effects regression or a statistically-adjusted engineering (SAE) approach. • Robust regression analyses require minimal variation in the sample characteristics. No sites will be removed, methods will be used to reduce bias in regression results. • E.g., account for customer types, square footage and number of meters; separate analysis into subsets of buildings with known size variances • Due to the heterogeneity of the population and measures installed, it may be difficult to estimate savings measure categories. • To extent possible, analysis will provide granular savings realization rates, e.g., end-use category, building type, size of facility. • Impact results from the analyzed sample will be extrapolated to the population and cost-effectiveness will be recalculated.

12. Ductless Heat Pumps • BPA is close to offering a pilot rebate for ductless heat pumps, with participation requirements • BPA is working closely with NEEA to coordinate implementation • Evaluation is one area where regional coordination is likely necessary • Suggestions welcome of best regional way to proceed with the effort • BPA has reviewed the RTF scope of work for DHP and is committed to: • 50 units of sub-metering (quad) of retrofit sites for 1 year, ASAP • 14 already in the field, will sub-contract analysis and installation of additional 35 • Billing analysis starting in summer of 2009 • Will require basic participant statistics and billing data (at time of participation and one-year later)