The Impact of Probabilistic Information on Deterministic
& Threshold Forecasts
Susan Joslyn, Earl Hunt & Karla Schweitzer
University of Washington
- OUR QUESTION: Can weather forecasters incorporate uncertainty information expressed in probabilities into a non-probabilistic forecasting decision in a way that improves the forecast? Without bias?
- How do people understand and use probabilistic information?
- People, in general, do not treat probability linearly
- (Gonzalez & Wu, 1999)
- Even experts have trouble incorporating prior probabilities
- (Eddy 1982 )
- However weather forecasters are good at estimating probabilities of e.g precipitation
- (Baars & Mass, 2004)
- Probabilistic information is increasingly important product of ensemble forecasts.
- • Yet few operational weather forecasters make use of available probability products
- (Joslyn, Jones & Tewson, 2005)
10 Participants: Atmospheric Science Students
Task: Forecast wind speed and direction for a 48 hour period. .Decide whether to issue high wind advisory (20 + knots). Forecasts were made for 4 days for four different locations on each day.
- Information Provided
- Historical data for all products used in previous cognitive task analysis (Joslyn et al, submitted)
- Radar Imagery
- Satellite Imagery
- TAFs and current METARs
- Model output (AVN, MM5 & NGM)
- Design: Within subjects
- • Each subject made 2 forecasts with the probability product,
- 2 forecasts without probability product
- • All subjects saw weather data from all of the same dates
- • Probability product rotated through the dates, making sure that no subject saw the same date twice
- On 1/2 of the forecasts, participants had the
- MM5 ACME Ensemble Probability of winds
- greater than 20 knots
- Free choice of products
- Except: forecasts with the probability product
- Required to read & record the range provided by the probability product
Wind speed forecast: Deterministic forecast
R2 Predicting observed from forecasted wind speeds
- We conducted regression analyses predicting observed wind speeds from forecasted wind speeds. See R2 for each condition in the table to the right.
- For 3 dates people did better with the probability product
- Note especially 3/21 which was the most difficult and for which the probability product made a significant difference.
Posting wind advisories: Thresholdforecast
- d with and without the probability product were similar
- However, WITH the probability product:
- the criterion was higher (less willing to post advisory)
- Fewer advisories overall.
- Same forecasters/same weather
- Difference must be due to probability product
Probability product had a slight impact on the deterministic wind speed forecast.
Wind Advisory Forecasts:
- Forecasters, in general, had a liberal bias in the low to mid probability ranges: Biased to post wind advisories
- Makes sense in this task: being cautious, keeping people off the water if there is any chance of danger
- However, boaters may come to disregard the advisory if it often proves to be a false alarm (Roulston & Smith, 2004)
- Probability product attenuated this tendency without causing them to lose sensitivity--they weren’t more likely to miss high wind situations
- Forecasters, in general, had a conservative bias in the highest probability ranges
- They did not post advisory as often as they should have
- Probability product attenuated this tendency
- more advisories in high range with probability product
Gonzalez, R., & Wu, G. (1999). On the form of the probability weighting function. Cognitive Psychology, 38, 129-166.
Eddy, D.M. (1982). Probabilistic reasoning in clinical medicine: Problems and opportunities. In D. Kahneman, P. Slovic, &
A. Tversky (Eds), Judgment under uncertainty: Heuristics and biases pp. 249-267)
Baars, J. A. & Mass, C. F. (2004) Performance of National Weather Service Forecasts Compared to Operational, Consensus
and Weighted Model Output Statistics.
Joslyn, S. Jones, D.W. & Tewson, P.(2005) Designing Tools for Uncertainty Estimation in Naval Weather Forecasting. 7th
International Conference on Naturalistic Decision Making
Roulston, M.S. & Smith, L. A. (2004) The boy who cried wolf revisited: The impact of false alarm intolerance on cost-loss
scenarios. Weather & Forecasting (19) 391-397.
- Good in very low probability situation
- Overforecast in the midranges
- In general, improved subjects performance
Subjects tended to
- Over forecast in low-mid probability situations
- Underforecast in high probability ranges
- Attenuated with probability product
This research was supported by the DOD Multidisciplinary University Research Initiative (MURI) program administered by the Office of Naval Research under Grant N00014-01-10745