Presentation Overview

1. Empirical Data on Settlement of Weather Sensitive LoadsJosh Bode, M.P.P.ERCOT Demand Side Working GroupAustin, TXSeptember 20, 2012

2. Presentation Overview Why is settlement of weather sensitive loads an issue? Testing accuracy of settlement methods Empirical results Using smart meter data and control groups for evaluation

3. Baselines are a tool to estimate demand reductions • Measuring demand reductions is an entirely different task than measuring power production • Power production is metered and thus is measured directly. • Demand reductions cannot be metered. They must be estimated by indirect approaches. • In principle, the reduction is simply the difference between electricity use with and without the load curtailment • However, it is not possible to directly observe or meter what electricity use would have been in the absence of the curtailment – the counterfactual. • Instead, the counterfactual must be estimated.

4. The accuracy of baseline estimates for large C&I customers has been studied multiple times Highly volatile load customer study (CAEC) Ontario Power Authority Study (FSC) WG2 Report Baseline Accuracy Analysis (Quantum) WG2 Baseline accuracy analysis (Quantum) California Aggregator Programs (FSC) California Aggregator and DBP Evaluations (CAEC) KEMA Baseline Analysis for CEC LBNL Study (proxy events) PJM Baseline Study (KEMA) ISO-NE Baseline Study (KEMA)

5. Settlement of reductions from weather sensitive loads like AC has been studied less • Weather sensitive loads have demonstrated the ability to support multiple grid functions • 4.8 million residential AC units and more than 500,000 water heaters in the U.S. have load control devices • Recent technological innovations enable aggregation and real time visibility of small scale loads • Many load control devices now include over and under frequency relays, providing an automated fail-safe mechanism that is synchronized with the grid

6. Visibility of loads has been tested • Because of the sheer number of AC units, it is not practical to monitor all data points • Real time monitoring of AC units is expensive, even for a sample Feeder Load Estimated loads for population AC end use sample

7. Load control programs have shown the ability to provide contingency reserves • Highly granular dispatch is possible – it is possible to dispatch all or some of the resources in a specific area • No one has tested the built-in under frequency relays which provide a failsafe mechanism, do not rely on central dispatch and should respond even faster • Customers that were curtailed 68-71 over the summer report same comfort and frequency of events as customers that were curtailed once Fast ramp up to full resource capability Fast start up time

8. Water heaters have demonstrated the ability to follow regulation signals • Graph is from PJM pilot (Joe Callis) • The initial test was for a unit and has since expanded to wider scale testing

9. However, these loads are highly variable The fact that some loads are very weather sensitive does not mean they are unpredictable Average Hourly Residential AC Loads by Temperature

10. Testing the accuracy of settlement methods

11. We tested 11 different settlement alternatives for baselines for short curtailments

12. To test accuracy, one needs to know the correct values Because the demand reductions values are artificially introduced and known, we can determine the accuracy of each baseline alternative

13. For the tables and approaches that relied on control groups, we used a split sample approach Randomly split data into two groups Simulate the reduction in one group Use the second group to produce the counterfactual or baseline Calculate impacts and store Repeat 100 times

14. There are two key issues in assessing accuracy – bias and goodness of fit

15. Limitations of analysis • The analysis of settlement approaches focuses on ancillary services • Short term reductions to stabilize the grid • Usually triggered by generation or transmission outages and sometimes unexpected changes in wind or loads • Not always in the hottest hours • The errors in the demand reduction estimates depend on the magnitude of the reduction signal • The estimates were based on 50% standard cycling • Air conditioner use is lower in California than in Texas • The direction of the findings likely hold up but the magnitude of the errors will be different

16. How does the magnitude of the demand reduction signal affects accuracy? Example The customers are nearly identical, but the estimation error differ because of they reduce ad different amount of demand

17. Empirical Results

18. What is the value of more complex approaches? In each case, we compare results to the most simple approach – the pre-calculated load reduction tables We present the results for within-subject and control group approaches separately We present the bias and goodness of fit metrics separately All graphs used the same scale

19. Matching and regression approaches with individual AC data

20. Matching and regression approaches with aggregated AC data

21. Matching and regression approaches with whole house data

22. Matching and regression approaches with feeder data

23. why are feeder results so inaccurate? Example • Feeder characteristics • 2,672 accounts on feeder • 266 AC load control accounts (10% of feeder) • 292 AC controllable AC units • Likely includes commercial • Penetration higher than 90% of feeders • Event day characteristics • August 24, 2010, max temp 103 F • Simulated event period 12:00-14:00 • AC load per unit 0.63 kW • Load Impact 35% • Controllable AC load 183.3 kW (0.63 kW per unit x 292 AC units) • Feeder Impact 64 kW • Actual load without DR 7772 kW • Simulated load with DR 7708 kW • Percent impact on feeder 0.83%!!!

24. Control group methods with AC end use data(500 control group, 500 treatment)

25. Control group methods with whole house data (2,000 control group, 2,000 treatment)

26. Implications of study Don’t rely on feeder data for settlement Day matching baselines are the least accurate approach with weather sensitive loads Day-matching baselines are not well suited for measuring demand reductions from highly weather sensitive loads More granular meters do not necessarily increase the accuracy of demand reduction measurement because measuring demand reduction is fundamentally different Complex methods provide limited improvement Pre-calculated load reduction tables can produce results that on average are correct, but may err for individual days, especially if they are cooler Methods with control groups and large sample sizes perform best

27. Using Smart Meter Data and Control Groups for Evaluation

28. Impact estimate tables are not developed in a vacuum They should be based on a history of results Ideally this includes systematic testing of load control devices under different condition and with different operation and control strategies The estimates from the operations underlying the tables need to be unbiased and, ideally, precise The more data points, the better the results One may need to account for changes in the customer mix, if relevant

29. With large samples and random assignment, estimation error is virtually eliminated • Randomly assign population into 10 groups • For each test event, one group was activated and the other 9 were held as a control groups • For a few events, we tested different operation strategies side-by-side Actual example for 2011 PG&E SmartAC evaluation Wide availability of smart meter data and individually addressable devices are a pre-requisite

30. It enables side by side testing of different operation strategies

31. It also enables side by side testing of different control strategies

32. By using control groups and short events, once can get a substantial history of results In the PG&E study, each customer only experienced one event, but we obtained results from 7 days, including 3 with side by side testing It is reasonable to call up to 10 events per customers, especially if the curtailments are short (e.g. 1-2 hrs) This can yield results under 100 different curtailments to inform the impact estimate tables

33. For any questions, feel free to contact Josh Bode, M.P.P. Freeman, Sullivan & Co. 101 Montgomery Street 15th Floor, San Francisco, CA 94104 joshbode@fscgroup.com 415.777.0707