1 / 38

Introduction to Forecast Verification

Introduction to Forecast Verification. Laurence J. Wilson MSC Dorval Quebec. Outline. Why verify: Principles, goodness and goals of verification General Framework for verification (Murphy-Winkler) Joint and conditional distributions Scores and measures in context of the framework

Download Presentation

Introduction to Forecast Verification

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Introduction to Forecast Verification Laurence J. Wilson MSC Dorval Quebec

  2. Outline • Why verify: Principles, goodness and goals of verification • General Framework for verification (Murphy-Winkler) • Joint and conditional distributions • Scores and measures in context of the framework • Murphy’s attributes of forecasts • Value of forecasts • Model Verification • Data matching issues • Verification issues • Verification of probability distributions • Tricks of the trade • Conclusion

  3. References • Wilks, D.S., 1995: Statistical methods in the atmospheric sciences. Academic Press. Chapter 7. • Stanski, H.R., L.J. Wilson, and W.R. Burrows, 1990: Survey of common verification methods in meteorology. WMO WWW Technical Report No. 8. Also available on the web – see below. • Jolliffe, I.T. and D.B. Stephenson, 2003: Forecast verification: A practitioner’s guide in atmospheric science. Wiley. • Murphy, A.H. and R.L. Winkler, 1987: A general framework for forecast verification. Mon. Wea. Rev. 115, 1330-1338. • Murphy, A.H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Wea. Forecasting 8, 281-293. Web Page of the Joint WGNE/WWRP Verification Working Group: Has lots of links, and lots of information. (and is changing all the time) ANNOUNCEMENT: This group is organizing a workshop on Verification methods in or near Montreal, Quebec, September 13 to 17, 2004. http://www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html

  4. Evaluation of forecasts • Murphy’s “goodness” • CONSISTENCY: forecasts agree with forecaster’s true belief about the future weather [strictly proper] • QUALITY: correspondence between observations and forecasts [ verification] • VALUE: increase or decrease in economic or other kind of value to someone as a result of using the forecast [decision theory]

  5. Evaluation of forecast system • Evaluation of forecast “goodness” • Evaluation of delivery system • timeliness (are forecasts issued in time to be useful?) • relevence (are forecasts delivered to intended users in a form they can understand and use?) • robustness (level of errors or failures in the delivery of forecasts)

  6. Principles of (Objective) Verification • Verification activity has value only if the information generated leads to a decision about the forecast or system being verified • User of the information must be identified • Purpose of the verification must be known in advance • No single verification measure provides complete information about the quality of a forecast product. • Forecast must be stated in such a way that it can be verified • “chance” of showers • What does that gridpoint value really mean? • Except for specific validation studies, verification should be carried out independently of the issuer of the product.

  7. Goals of Verification • Administrative • Justify cost of provision of weather services • Justify additional or new equipment • Monitor the quality of forecasts and track changes • Scientific • To identify the strengths and weaknesses of a forecast product in sufficient detail that actions can be specified that will lead to improvements in the product, ie to provide information to direct R&D.

  8. Verification Model (cont’d) • Predictand Types • Continuous: Forecast is a specific value of the variable • wind • temperature • upper air variables • Categorical/probabilistic: Forecast is the probability of occurrence of ranges of values of the variable (categories) • POP and other weather elements (fog etc) • Precipitation type • cloud amount • precipitation amount • Probability distributions (ensembles)

  9. Framework for Verification (Murphy-Winker) • All verification information contained in the joint distribution of forecasts and observations • Factorizations: • calibration-refinementp(f,x) = p(x|f) p(f)p(x|f) = conditional distribution of observation given forecast (calibration/reliability)p(f) = marginal distribution of forecasts (refinement) • likelihood-base ratep(f,x) = p(f|x)p(x)p(f|x) = conditional distribution of forecasts given observations (likelihood/discrimination)p(x) = marginal distribution of observations (base rate/climatology)

  10. Verification Samples • Joint distn of forecasts and observations may be: • A time series at points • One or more spatial fields • A combination of these • In meteorological applications: • The events of the sample are not usually even close to independent in the statistical sense • Importance of carefully assessed confidence limits of verification results

  11. Spot Temperature Scatter Plot Exercise

  12. Spot Temperature Scatter Plot Exercise

  13. Contingency Tables - Basic 2 X 2 Forecasts Yes No Yes Observations No

  14. Verification -A general model

  15. Skill Scores • Format: • Where Sc=score (MAE, Brier etc) • PSc= score for a perfect forecast • SSc= score for a standard (unskilled) forecast

  16. ROC - ECMWF Ensemble Forecasts24 h POP (>1 mm)

  17. Current ROC – Canadian EPSSpring 2003

  18. Summary • The choice of verification measure depends on: • The purpose of the verification (admin – science) • The nature of the predictand • The attributes of the forecast to be measured

  19. The meaning of ‘Value’ • “Weather forecasts possess no intrinsic value in an economic sense. They acquire value by influencing the behaviour of individuals or organizations (“users”) whose activities are sensitive to weather.” • Allan Murphy, Conference on economic benefits of Meteorological and Hydrological services (Geneva, 1994)

  20. Types of “Value” • Social value - Minimization of Hazards to human life and health • Value to individual users • Economic value of forecasts • Value to a specific business • Value to a weather-sensitive industry • Value to a weather-sensitive sector • Value to the economy of a country • Market value (e.g. futures) • Environmental value • minimizing risk to the environment • optimal use of resources

  21. Value vs. Quality • Quality refers only to forecast verification; Value implicates a user • A perfect forecast may have no value if no one cares about it • An imperfect forecast will have less value than a perfect forecast • See Murphy and Ehrendorfer 1987

  22. Measuring value • The cost-loss decision model • focus on maximizing gain or loss-avoidance • requires objective cost information from user • user specific, difficult to generalize • economic value to weather-sensitive operation only. • easy to evaluate relative value • Contingent-valuation method • focuses on demand for service and “willingness to pay” • requires surveys of users to determine variations in demand as function of variations in price and/or quality of service • less user-specific; a larger crossection of users/industries can be evaluated in one study • measures in terms of perception rather than actual accuracy. • e.g. evaluation of ATADs, Rollins and Shaykewich, Met Apps Mar. 03

  23. Model Verification • Data matching issues • Verification issues

  24. Model Verification – Data Matching Issues • Typical situation: Model gridded forecasts, observations at points • Point in situ observations undersample the field, contain information on all spatial and most temporal scales. (“representativeness error”? Not really) • Alternatives: • Model to data: What does the model predict at the obs point? • Interpolation – if the model gives point predictions • Gives answers at all verification points • Nearest gridpoint value – if the model prediction is a grid box average • Verify only those grid boxes where there is at least one obs • UPSCALING: - estimate grid box average using all obs in grid box. • Data to model: Analysing point data: • NOT RECOMMENDED because treats networks of point observations as if they contain information only on the scales represented by the grid on which the analysis is done.

  25. Model Verification- Issues • Resolution – scale separation • Spatial verification – object-oriented

  26. Model Verification – Scale separation • Mesoscale verification: Separating model errors due to resolution from other errors. • IF high resolution spatial data is available: • Scale separation, wavelets or other method (Mike Baldwin) • Repeat verification on same dataset at different scales to get performance curve • Data combination: Use high resolution data to “inform” statistical estimates such as grid box averages.

  27. Spatial Verification • Object-oriented methods • The calculation of displacement, size errors for specific objects (e.g. rain areas, fronts) • Hoffman, 1995; Ebert and McBride 2000 CRA method • Decomposition of errors into location, shape, size components • Others (Mike Baldwin) • Problem always is the matching of the forecast and observed object.

  28. Verification of probability distributions • Problem: • Comparison of distribution with a single outcome • Solutions: • Verify density in vicinity of observation (Wilson, Burrows and Lanzinger, 1999) • Verify cdf against observation represented as cdf (CRPS, Hersbach 2000) • Extract probabilities from distribution and verify as probability forecasts (sample several thresholds) • Compare parameters of pooled distribution with sample climatology (Talagrand diagram)

  29. Ensemble verification - distribution

  30. CRPS example

  31. Rank Histogram example

  32. Tricks of the trade • “How can I get a better (higher) number?” • Remove the bias before calculating scores (works really well for quadratic scoring rules) and don’t tell anyone. • Claim that the model predicts grid box averages, even if it doesn’t . Make the boxes as large as possible. • Never use observation data. It only contains a lot of “noise”. As an alternative,: • Verify against an analysis that uses the model being verified as a trial field. Works best in data-sparse areas • Use a shorter range forecast from the model being verified and call it observation data. • Design a new or modified score. Don’t be bothered by restrictions such as strictly properness. Then the values can be as high as desired. • Stratify the verification data using posteriori rules. One can always get rid of pathological cases that bring down the average. • When comparing the performance of your model against others, make sure it is your analysis that is used as the verifying one. • Always insist on doing the verification of your own products…. • Remember, you already know you have a good product. The goal of the verification is to show it “objectively”

  33. Conclusions • Quick survey of verification and value assessment • A data-oriented perspective

More Related