330 likes | 456 Views
ERCOT. LRS Precision Analysis PWG Presentation February 27, 2007. Options for Determining Round Two Sample Size Increases. Option 1:
E N D
ERCOT LRS Precision Analysis PWG Presentation February 27, 2007
Options for Determining Round Two Sample Size Increases • Option 1: • Determine minimum sample size needed to obtain ±10% Accuracy at 90% Confidence for a selected percent of intervals for the year independently for each Profile Type / Weather Zone Combination • For Options 2 – 5 ERCOT recomputed daily energy totals and dollars • Used SAS data aggregation tool developed for transition analysis • Applied load profiles from new models to spread monthly LSEG totals from Lodestar to intervals • Multiplied by weather zone weighted MCPE to associate a dollar value with each interval • Option 2: • Determine minimum sample size needed to obtain ±10% Accuracy at 90% Confidence for enough intervals to account for a selected percent of the MWh for each Profile Type / Weather Zone Combination • Option 3: • Determine minimum sample size needed to obtain ±10% Accuracy at 90% Confidence for enough intervals to account for a selected percent of the dollars (ΣMWh * MCPE) for each Profile Type / Weather Zone Combination • Option 4: • Iteratively allocate increments of 20 sample points to the Profile Type / Weather Zone Combination which produces the most gain in terms of reducing MWh estimation error • Option 5: • Iteratively allocate increments of 20 sample points to the Profile Type / Weather Zone Combination which produces the most gain in terms of reducing Dollar estimation error
Option 1 • To obtain ±10% Accuracy at 90% Confidence By Profile Type and Weather Zone - Independent of Interval For example: to obtain ±10% Accuracy at 90% Confidence for 50% of the intervals for BUSHILF_COAST would require a sample size of 11 points
Option 1 • To obtain ±10% Accuracy at 90% Confidence By Profile Type and Weather Zone - Independent of Interval
Option 1 • To obtain ±10% Accuracy at 90% Confidence By Profile Type and Weather Zone - Independent of Interval
Option 1 • To obtain ±10% Accuracy at 90% Confidence By Profile Type - Independent of Interval For example: to obtain ±10% Accuracy at 90% Confidence for 50% of the intervals for all Profile Type/Weather Zone combinations would require a sample size of 8,447 points
Option 2 • To obtain ±10% Accuracy at 90% Confidence for Intervals Accounting for Selected Percents of the MWH within Each Profile Type and Weather Zone For example: to obtain ±10% Accuracy at 90% Confidence for intervals accounting for 50% of the BUSMEDLF MWh would require asample size of 1677 points
Option 2 • To obtain ±10% Accuracy at 90% Confidence for Intervals Accounting for Selected Percents of the MWH within Each Profile Type and Weather Zone
Option 2 • To obtain ±10% Accuracy at 90% Confidence for Intervals Accounting for Selected Percents of the MWH within Each Profile Type and Weather Zone
Option 2 • To obtain ±10% Accuracy at 90% Confidence for Intervals Accounting for Selected Percents of the MWH within Each Profile Type For example: to obtain ±10% Accuracy at 90% Confidence for intervals accounting for 50% of the MWh for each of the Profile Type / Weather Zone combinations would require a sample size of 7,438 points
Option 3 • To obtain ±10% Accuracy at 90% Confidence for Intervals Accounting for Selected Percents of the dollars within each Profile Type & Weather Zone Note: Dollars = Σ (MWh * MCPE)
Option 3 • To obtain ±10% Accuracy at 90% Confidence for Intervals Accounting for Selected Percents of the dollars within each Profile Type & Weather Zone
Option 3 • To obtain ±10% Accuracy at 90% Confidence for Intervals Accounting for Selected Percents of the dollars within each Profile Type & Weather Zone
Option 3 • To obtain ±10% Accuracy at 90% Confidence for Intervals Accounting for Selected Percents of the dollars within each Profile Type For example: to obtain ±10% Accuracy at 90% Confidence for intervals accounting for 50% of the Dollars for each of the Profile Type / Weather Zone combinations would require a sample size of 6,969 points Continues on next slide
Option 3 • To obtain ±10% Accuracy at 90% Confidence for Intervals Accounting for Selected Percents of the dollars within each Profile Type
Class Level MWH & Dollars Not all Profile Type / Weather Zone Combinations are created equal in either MWh or Total Dollars (ΣMWh * MCPE) !
Class Level MWH & Dollars Not all Profile Type / Weather Zone Combinations are created equal in either MWh or Total Dollars (ΣMWh * MCPE) !
Class Level MWH & Dollars Not all Profile Type / Weather Zone Combinations are created equal in either MWh or Total Dollars (ΣMWh * MCPE) !
Class Level MWH & Dollars • Totals from previous three slides. Is accuracy more important for RESLOWR (33.4% of Dollars) than for BUSNODEM (1.6% of Dollars)?
Class Level MWH & Dollars - Descending Order by Dollars Top 5 classes account for 53% of the MWh and 54% of the dollars * Note: Dollars = Σ (MWh * MCPE) Continues on next slide
Class Level MWH & Dollars - Descending Order by Dollars Bottom 28 classes account for only 10% of the MWh and dollars * Dollars = Annual MWh * MCPE Continues on next slide
Class Level MWH & Dollars - Descending Order by Dollars * Dollars = Annual MWh * MCPE Continues on next slide
Precision vs Sample Size • Increasing sample size has a diminishing return on precision improvement • Error Ratio (thus Precision improvement) varies across Profile Types / Weather Zones and across intervals • Thus the impact of adding sample points varies by Profile Type and Weather Zone
Options 4 & 5 • Options 4 & 5 iteratively allocate increments of 20 sample points to the next Profile Type / Weather Zone Combination in order to produce the most gain in • Reducing MWh (Option 4) estimation error (Precision × MWh) summed across all intervals • Reducing Dollar (Option 5) estimation error (Precision × Dollars) summed across all intervals • The allocations are based on • The MWh (or Dollars) associated with each of the Profile Type / Weather Zone combinations in each interval • The Error ratio in each interval for each Profile Type / Weather Zone combination • The cumulative number of sample points allocated by preceding iterations (including the original sample size) • The precision improvement that would be realized by adding 20 sample points, and the diminishing return on that improvement • Minimum Sample Size • Profile to profile migration resulted in numerous instances of small sample sizes within strata • Small sample sizes resulted in both accuracy degradation and the need to drop strata from load research analysis • A minimum of 3 strata will be specified for each Profile Type / Weather Zone combination • A minimum of 40 sample points will be allocated to each stratum • Minimum sample size per Profile Type / Weather Zone combination will be 120 • Maximum Sample Size • Maximum for Business Profile / Weather Zone combinations set to 400 • Maximum for Residential Profile / Weather Zone combinations set to 600
Option 4 – MWh Error Reduction Optimization Cumulative sample sizes are shown in increments of 1,000; they were determined iteratively in increments of 20 sample points
Dollar Error Reduction Based on Sample Size Increases Non - Optimized Allocation of points Optimized Allocation of Additional Points
Impact of UFE Allocation • The precision of estimates produced by a sample design will affect the accuracy of the resulting profile models • The profile model outputs will be adjusted for UFE • Sample design should take the UFE adjustment into consideration • Intuitively the iterative sample design process should minimize the impact of UFE adjustment • Used iterative design based on dollar error minimization • ERCOT ran Monte Carlo simulations to evaluate the impact of UFE • Simulated sample outcomes for 48 Profile Type / Weather Zone combinations for all intervals of the 19 month analysis period • Adjusted sample outcomes for UFE • Compared MAPE before and after UFE adjustment • Replicated this process 500 times for 3 sample designs (8000, 9000, 10000 sample points)
Impact of UFE Allocation UFE Impact with the 3 selected sample size designs and levels is negligible
Dealing with Oil and Gas Migrations • Round two sample design and selection will be complete prior to BUSOGFLT implementation • ERCOT has obtained list of Oil and Gas ESIIDs from TXU-ED and AEP • ERCOT will check with other TDSPs for additional ESIIDs • The ESIIDs will be eliminated from sample design, selection and the analysis population • Sample estimates will be adjusted to reflect ESIIDs which have not migrated to BUSOGFLT • Usage for these ESIIDs will be aggregated and profiled as flat load • No sample points will be needed for Oil and Gas ESIIDs
Dealing with Primary Voltage Migrations • In Round one sample design for most Profile Type / Weather Zone combinations 3 – 4 separate strata were set up for Primary Voltage ESIIDs • These strata usually had small sample sizes, many were 100% sampled • Profile migrations were particularly problematic in performing analysis as a result of empty or sparse strata • ERCOT has done preliminary statistical analysis of the Primary Voltage population (excluding Oil and Gas ESIIDs) • Profile Type / Weather Zone combination • Appear to have significant differences from Secondary Voltage ESIIDs in the same weather zone and profile type • Separate Primary Voltage strata will be beneficial for Round Two sample accuracy • A single Primary Voltage stratum will be established in each Profile Type / Weather Zone combination were applicable • Each stratum will be allocated a minimum sample size (40) • Profile migration issues should be less significant to future analysis • ERCOT plans to evaluate the introduction of Primary Voltage as a potential new profile type • Adoption would be contingent on significant sample (and load profile model) accuracy improvements for both Secondary and Primary Voltage ESIIDs • If adopted, migrations to the new profile type would not create future analysis issues • Augmented samples would probably be necessary to build adequate models for Primary
Conclusions and Follow-up Actions • The iterative sample point allocation process has intuitive appeal • Seems to allocate sample points where they do the most good • Would be expected to maximize UFE reduction • UFE allocation has negligible impact on the final accuracy • ERCOT will be selecting a total sample size of 9,000 points for secondary voltage ESIIDs and also will select a sample of primary voltage ESIIDs … 40 per profile type / weather zone combination were applicable • ERCOT will run MBSS to determine stratum boundaries based on annualized kWH and to allocate sample points to the strata • ERCOT will then randomly select primary and replacement sample points based on the design and forward the sample lists to TDSPs in March • ERCOT will update the sample tracking database with the new samples, sample points, and will add the sample points to the samples