slide1 n.
Skip this Video
Download Presentation
Julie Demargne, James Brown, Yuqiong Liu and D-J Seo

Loading in 2 Seconds...

play fullscreen
1 / 37

Julie Demargne, James Brown, Yuqiong Liu and D-J Seo - PowerPoint PPT Presentation

  • Uploaded on

Application of Forecast Verification Science to Operational River Forecasting in the National Weather Service. Julie Demargne, James Brown, Yuqiong Liu and D-J Seo. UCAR. NROW, November 4-5, 2009. Approach to river forecasting. Observations. Forecasters. Models. Input forecasts. Users.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Julie Demargne, James Brown, Yuqiong Liu and D-J Seo' - mateja

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Application of Forecast Verification Science to Operational River Forecasting in the National Weather Service

Julie Demargne, James Brown,

Yuqiong Liu and D-J Seo


NROW, November 4-5, 2009

approach to river forecasting
Approach to river forecasting




Input forecasts





where is the

In the past


  • Limited verification of hydrologic forecasts
  • How good are the forecasts for application X?
Where is the …?


where is the1
Where is the …?





Verification Experts

Verification Products

Verification Systems

hydrologic forecasting a multi scale problem
Hydrologic forecasting: a multi-scale problem

Major river system


River basin with river forecast points

Forecast group

High resolution flash

flood basins

Headwater basin with radar rainfall grid

Hydrologic forecasts must be verified consistently across all spatial scales and resolutions.

hydrologic forecasting a multi scale problem1








Hydrologic forecasting: a multi-scale problem

Forecast Uncertainty

Forecast Lead Time


Protection of Life & Property


State/Local Planning




Flood Mitigation & Navigation

Reservoir Control




Seamless probabilistic water forecasts are required for all lead times and all users; so is verification information.


Need for hydrologic forecast verification

  • In 2006, NRC recommended NWS expand verification of its uncertainty products and make it easily available to all users in near real time
  • Users decide whether to take action with risk-based decision
  • Must educate users on how to interpret forecast and verification info

River forecast verification service rfcdev/docs/ Final_Verification_Report.pdf

river forecast verification service
River forecast verification service
  • To help us answer
    • How good are the forecasts for application X?
    • What are the strengths and weaknesses of the forecasts?
    • What are the sources of error and uncertainty in the forecasts?
    • How are new science and technology improving the forecasts and the verifying observations?
    • What should be done to improve the forecasts?
    • Do forecasts help users in their decision making?
river forecast verification service1
River forecast verification service

River forecasting system




Verification systems


Input forecasts



Verification products


river forecast verification service2
River forecast verification service
  • Verification Service within Community Hydrologic Prediction System (CHPS) to:
    • Compute metrics
    • Display data & metrics
    • Disseminate data & metrics
    • Provide real-time access to metrics
    • Analyze uncertainty and error in forecasts
    • Track performance
verification challenges
Verification challenges
  • Verification is useful if the information generated leads to decisions about the forecast/system being verified
    • Verification needs to be user oriented
  • No single verification measure provides complete information about the quality of a forecast product
    • Several verification metrics and products are needed
  • To facilitate communication of forecast quality, common verification practices and products are needed from weather and climate forecasts to water forecasts
    • Collaborations between meteorology and hydrology communities are needed (e.g., Thorpex-Hydro, HEPEX)
verification challenges two classes of verification
Verification challenges: two classes of verification
  • Diagnostic verification:
    • to diagnose and improve model performance
    • done off-line with archived forecasts or hindcasts to analyze forecast quality relative to different conditions/processes
  • Real-time verification:
    • to help forecasters and users make decisions in real-time
    • done in real-time (before the verifying observation occurs) using information from historical analogs and/or past forecasts and verifying observations under similar conditions
diagnostic verification products
Diagnostic verification products
  • Key verification metrics for 4 levels of information for single-valued and probabilistic forecasts
    • Observations-forecasts comparisons (scatter plots, box plots, time series plots)
    • Summary verification (e.g. MAE/Mean CRPS, skill score)
    • More detailed verification (e.g. measures of reliability, resolution, discrimination, correlation, results for specific conditions)
    • Sophisticated verification (e.g. for specific events with ROC)

To be evaluated by forecasters and forecast users

diagnostic verification products1

Forecast value

User-defined threshold

Observed value

Diagnostic verification products
  • Examples for level 1:scatter plot, box-and-whiskers plot
diagnostic verification products2

‘Errors’ for

one forecast








Diagnostic verification products
  • Examples for level 1:box-and-whiskers plot

American River in California – 24-hr precipitation ensembles (lead day 1)

Zero error line

“Blown” forecasts

Forecast error (forecast - observed) [mm]

High bias

Low bias

Observed daily total precipitation [mm]

diagnostic verification products3




Diagnostic verification products
  • Examples for level 2:skill score maps by months

Smaller score, better

diagnostic verification products4
Diagnostic verification products
  • Examples for level 3:more detailed plots



Performance for different months

Performance under different conditions

diagnostic verification products5

Probability of Detection POD

Probability of False Detection POFD

Diagnostic verification products
  • Examples for level 4:event specific plots

Event: > 85th percentile from observed distribution




Observed frequency


Predicted Probability

diagnostic verification products6
Diagnostic verification products
  • Examples for level 4:user-friendly spread-bias plot

“Hit rate” = 90%

60% of time, observation should fall in window covering middle 60% (i.e. median ±30%)




diagnostic verification analyses
Diagnostic verification analyses
  • Analyze any new forecast process with verification
  • Use different temporal aggregations
    • Analyze verification statistic as a function of lead time
    • If similar performance across lead times, data can be pooled
  • Perform spatial aggregation carefully
    • Analyze results for each basin and results plotted on spatial maps
    • Use normalized metrics (e.g. skill scores)
    • Aggregate verification results across basins with similar hydrologic processes(e.g. by response time)
  • Report verification scores with sample size
    • In the future, confidence intervals
diagnostic verification analyses1
Diagnostic verification analyses
  • Evaluate forecast performance under different conditions
      • w/ time conditioning: by month, by season
      • w/ atmospheric/hydrologic conditioning:
        • low/high probability threshold
        • absolute thresholds (e.g., PoP, Flood Stage)
      • Check that sample size is not too small
  • Analyze sources of uncertainty and error
      • Verify forcing input forecasts and output forecasts
      • For extreme events, verify both stage and flow
      • Sensitivity analysis to be set up at all RFCs:
        • what is the optimized QPF horizon for hydrologic forecasts?
        • do run-time modifications made on the fly improve forecasts?
diagnostic verification software
Diagnostic verification software
  • Interactive Verification Program (IVP) developed at OHD:verifies single-valued forecasts at given locations/areas
diagnostic verification software1
Diagnostic verification software
  • Ensemble Verification System (EVS) developed at OHD:verifies ensemble forecasts at given locations/areas
dissemination of diagnostic verification

Data Visualization

  • Error
  • Conditional on lead time, year
  • Skill
  • Skill relative to
  • Climatology
  • Conditional
  • Categorical
  • FAR, POD, contingency table (based on climatology or user definable)
Dissemination of diagnostic verification
  • Example: WR water supply website

dissemination of diagnostic verification1
Dissemination of diagnostic verification
  • Example: OHRFC bubble plot online

real time verification
Real-time verification
  • How good could the ‘live’ forecast be?

Live forecast


real time verification1

Analog 3

Analog 2


Live forecast

Analog Observed

Analog Forecast

Real-time verification
  • Select analogs from a pre-defined set of historical events and compare with ‘live’ forecast

Analog 1

“Live forecast for Flood is likely to be too high”

real time verification2
Real-time verification
  • Adjust ‘live’ forecast based on info from the historical analogs

Live forecast

What happened

Live forecast was too high

real time verification3
Real-time verification
  • Example for ensemble forecasts

Live forecast (L)

Analog forecasts (H):μH = μL ± 1.0˚C

Analog observations

Temperature (oF)

“Day 1 forecast is probably too high”

Forecast lead day

real time verification4
Real-time verification
  • Build analog query prototype using multiple criteria
    • Seeking analogs for precipitation: “Give me past forecasts for the 10 largest events relative to hurricanes for this basin.”
    • Seeking analogs for temperature: “Give me all past forecasts with lead time 12 hours whose ensemble mean was within 5% of the live ensemble mean.”
    • Seeking analogs for flow: “Give me all past forecasts with lead times of 12-48 hours whose probability of flooding was >=0.95, where the basin-averaged soil-moisture was > x and the immediately prior observed flow exceeded y at the forecast issue time”.

Requires forecasters’ input!

outstanding science issues
Outstanding science issues
  • Define meaningful reference forecasts for skill scores
  • Separate timing error and amplitude error in forecasts
  • Verify rare events and specify sampling uncertainty in metrics
  • Analyze sources of uncertainty and error in forecasts
  • Consistently verify forecasts on multiple space and time scales
  • Verify multivariate forecasts (issued at multiple locations and for multiple time steps) by accounting for statistical dependencies
  • Account for observational error (measurement and representativeness errors) and rating curve error
  • Account for non-stationarity (e.g., climate change)
verification service development
Verification service development


Thorpex-Hydro project




Forecasters Users


Forecast agencies



collaboration on training

OHD-Deltares collaboration for CHPS enhancements

HEPEX Verification Test Bed

(CMC, Hydro-Quebec, ECMWF)

looking ahead
Looking ahead
  • 2012:
    • Info on quality of forecast service available online
    • real-time and diagnostic verification implemented in CHPS
    • RFC verification standard products available online along with forecasts
  • 2015:
    • Leveraging grid-based verification tools


thank you questions
Thank youQuestions?



diagnostic verification products7
Diagnostic verification products
  • Key verification metrics from NWS Verification Team report