MAPE Publication. Neil McAndrews For Bob Ryan of Deutsche Bank. What makes a forecast a forecast?.
PowerPoint Slideshow about ' MAPE Publication' - tacita
An Image/Link below is provided (as is) to download presentation
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Forecasts are designed to estimate, in an unbiased manner, the future behavior of a directly observable event or variable. In power markets, ERCOT included, forecasts transfer market information from public to private marketparticipants.
Forecasts must be unbiased estimates,otherwise they are not valid forecasts. Unbiased means that they are wrong as often as they are right.
The Expected Value of a Forecast’s Error is, by definition, zero. These error termsshould be normally distributed.
A forecast’s accuracy – hence its reliability – is measuredby the use of statistical tools such as Mean Absolute Percentage Error (MAPE), Absolute Percent Error (APE), and Percent Error (PE).
When assessing load forecasts, MAPE is the preferred measurement of performance. It tells the user how the forecast is performing over time. It is calculated by the equation: Average of all intervals where ”interval” is defined as
Absolute value of (Actual minus Forecast) divided by Actual times 100.
A large or steadily increasing MAPE typically is the result of persistent misspecification in a model – i.e., overestimation and underestimation of actual loads.
Overestimation of actual load in electric markets leads to inefficient allocation of scarce reserve resources because the market is uncertain of actual load levels .
Underestimation leads to shortages and increases in use of reserves because the market is uncertain of actual load levels.
Currently ERCOT publishes monthly MAPE from time to time. This PRR makes it monthly.
Forecasts of load are really the combination of two forecasts: the load model forecast, and the weather forecast (predominately temperature forecast).
Publishing MAPE data for both model error and weather error will lead to corrections of particular problems.
Market participants and the ISO can continually iterate on their forecasts to improve forecast accuracy (i.e., minimize forecast error). This is common practice in other regions, e.g., PJM.
For instance, weather models are periodically updated and not all of the updates immediately improve performance in all geographic regions. Through the use of timely weather MAPE publication, problems with weather forecasts can be more quickly identified by both ERCOT and private participants reviewing the weather data.
The forecast is overestimating peak load for the months of September and October by an average of 1,677 MWs (3.6%). Actual error should be much closer to zero. (Overestimation does appear to be declining later in the period)
The error is not normally distributed about actual peak load. 77% of time the forecast overestimates load. This should be approximately 50%.
We in ERCOT should be asking the question: “How does one percentage point improvement in MAPE effect the cost of reserves?”
We suggest an estimate can be made by running the Ancillary Service selection process with actual load data and comparing it to the current forecast process. With enough samples an estimate of the reduction of reserve cost in AS market can be made. Then a calculation can be performed that estimates the cost of improving the forecast by one percent. A cost benefit will result from the two estimates.