Application of Forecast Verification Science to Operational River Forecasting
Download
1 / 37

Julie Demargne, James Brown, Yuqiong Liu and D-J Seo - PowerPoint PPT Presentation


  • 137 Views
  • Uploaded on

Application of Forecast Verification Science to Operational River Forecasting in the National Weather Service. Julie Demargne, James Brown, Yuqiong Liu and D-J Seo. UCAR. NROW, November 4-5, 2009. Approach to river forecasting. Observations. Forecasters. Models. Input forecasts. Users.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Julie Demargne, James Brown, Yuqiong Liu and D-J Seo' - mateja


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Application of Forecast Verification Science to Operational River Forecasting in the National Weather Service

Julie Demargne, James Brown,

Yuqiong Liu and D-J Seo

UCAR

NROW, November 4-5, 2009


Approach to river forecasting
Approach to river forecasting River Forecasting

Observations

Forecasters

Models

Input forecasts

Users

Forecast

products

Forecasters


Where is the

In the past River Forecasting

???

  • Limited verification of hydrologic forecasts

  • How good are the forecasts for application X?

Where is the …?

Verification


Where is the1
Where is the …? River Forecasting

Now

Verification

!!!

Papers

Verification Experts

Verification Products

Verification Systems


Hydrologic forecasting a multi scale problem
Hydrologic forecasting: a multi-scale problem River Forecasting

Major river system

National

River basin with river forecast points

Forecast group

High resolution flash

flood basins

Headwater basin with radar rainfall grid

Hydrologic forecasts must be verified consistently across all spatial scales and resolutions.


Hydrologic forecasting a multi scale problem1

Years River Forecasting

Seasons

Months

Weeks

Days

Hours

Minutes

Hydrologic forecasting: a multi-scale problem

Forecast Uncertainty

Forecast Lead Time

Benefits

Protection of Life & Property

Recreation

State/Local Planning

Hydropower

Environment

Ecosystem

Flood Mitigation & Navigation

Reservoir Control

Agriculture

Health

Commerce

Seamless probabilistic water forecasts are required for all lead times and all users; so is verification information.


Need for hydrologic forecast verification River Forecasting

  • In 2006, NRC recommended NWS expand verification of its uncertainty products and make it easily available to all users in near real time

  • Users decide whether to take action with risk-based decision

  • Must educate users on how to interpret forecast and verification info


River forecast verification service River Forecasting

http://www.nws.noaa.gov/oh/rfcdev/docs/NWS-Hydrologic-Forecast-Verification-Team_Final-report_Sep09.pdf.pdf

http://www.nws.noaa.gov/oh/ rfcdev/docs/ Final_Verification_Report.pdf


River forecast verification service
River forecast verification service River Forecasting

  • To help us answer

    • How good are the forecasts for application X?

    • What are the strengths and weaknesses of the forecasts?

    • What are the sources of error and uncertainty in the forecasts?

    • How are new science and technology improving the forecasts and the verifying observations?

    • What should be done to improve the forecasts?

    • Do forecasts help users in their decision making?


River forecast verification service1
River forecast verification service River Forecasting

River forecasting system

Forecasters

Users

Observations

Verification systems

Models

Input forecasts

Forecast

products

Verification products

Users


River forecast verification service2
River forecast verification service River Forecasting

  • Verification Service within Community Hydrologic Prediction System (CHPS) to:

    • Compute metrics

    • Display data & metrics

    • Disseminate data & metrics

    • Provide real-time access to metrics

    • Analyze uncertainty and error in forecasts

    • Track performance


Verification challenges
Verification challenges River Forecasting

  • Verification is useful if the information generated leads to decisions about the forecast/system being verified

    • Verification needs to be user oriented

  • No single verification measure provides complete information about the quality of a forecast product

    • Several verification metrics and products are needed

  • To facilitate communication of forecast quality, common verification practices and products are needed from weather and climate forecasts to water forecasts

    • Collaborations between meteorology and hydrology communities are needed (e.g., Thorpex-Hydro, HEPEX)


Verification challenges two classes of verification
Verification challenges: two classes of verification River Forecasting

  • Diagnostic verification:

    • to diagnose and improve model performance

    • done off-line with archived forecasts or hindcasts to analyze forecast quality relative to different conditions/processes

  • Real-time verification:

    • to help forecasters and users make decisions in real-time

    • done in real-time (before the verifying observation occurs) using information from historical analogs and/or past forecasts and verifying observations under similar conditions


Diagnostic verification products
Diagnostic verification products River Forecasting

  • Key verification metrics for 4 levels of information for single-valued and probabilistic forecasts

    • Observations-forecasts comparisons (scatter plots, box plots, time series plots)

    • Summary verification (e.g. MAE/Mean CRPS, skill score)

    • More detailed verification (e.g. measures of reliability, resolution, discrimination, correlation, results for specific conditions)

    • Sophisticated verification (e.g. for specific events with ROC)

To be evaluated by forecasters and forecast users


Diagnostic verification products1

Forecast value River Forecasting

User-defined threshold

Observed value

Diagnostic verification products

  • Examples for level 1:scatter plot, box-and-whiskers plot


Diagnostic verification products2

‘Errors’ for River Forecasting

one forecast

Max.

90%

80%

Median

20%

10%

Min.

Diagnostic verification products

  • Examples for level 1:box-and-whiskers plot

American River in California – 24-hr precipitation ensembles (lead day 1)

Zero error line

“Blown” forecasts

Forecast error (forecast - observed) [mm]

High bias

Low bias

Observed daily total precipitation [mm]


Diagnostic verification products3

January River Forecasting

October

April

Diagnostic verification products

  • Examples for level 2:skill score maps by months

Smaller score, better


Diagnostic verification products4
Diagnostic verification products River Forecasting

  • Examples for level 3:more detailed plots

Score

Score

Performance for different months

Performance under different conditions


Diagnostic verification products5

Probability of Detection POD River Forecasting

Probability of False Detection POFD

Diagnostic verification products

  • Examples for level 4:event specific plots

Event: > 85th percentile from observed distribution

Reliability

Discrimination

Perfect

Observed frequency

Perfect

Predicted Probability


Diagnostic verification products6
Diagnostic verification products River Forecasting

  • Examples for level 4:user-friendly spread-bias plot

“Hit rate” = 90%

60% of time, observation should fall in window covering middle 60% (i.e. median ±30%)

60%

Perfect

“Underspread”


Diagnostic verification analyses
Diagnostic verification analyses River Forecasting

  • Analyze any new forecast process with verification

  • Use different temporal aggregations

    • Analyze verification statistic as a function of lead time

    • If similar performance across lead times, data can be pooled

  • Perform spatial aggregation carefully

    • Analyze results for each basin and results plotted on spatial maps

    • Use normalized metrics (e.g. skill scores)

    • Aggregate verification results across basins with similar hydrologic processes(e.g. by response time)

  • Report verification scores with sample size

    • In the future, confidence intervals


Diagnostic verification analyses1
Diagnostic verification analyses River Forecasting

  • Evaluate forecast performance under different conditions

    • w/ time conditioning: by month, by season

    • w/ atmospheric/hydrologic conditioning:

      • low/high probability threshold

      • absolute thresholds (e.g., PoP, Flood Stage)

    • Check that sample size is not too small

  • Analyze sources of uncertainty and error

    • Verify forcing input forecasts and output forecasts

    • For extreme events, verify both stage and flow

    • Sensitivity analysis to be set up at all RFCs:

      • what is the optimized QPF horizon for hydrologic forecasts?

      • do run-time modifications made on the fly improve forecasts?


  • Diagnostic verification software
    Diagnostic verification software River Forecasting

    • Interactive Verification Program (IVP) developed at OHD:verifies single-valued forecasts at given locations/areas


    Diagnostic verification software1
    Diagnostic verification software River Forecasting

    • Ensemble Verification System (EVS) developed at OHD:verifies ensemble forecasts at given locations/areas


    Dissemination of diagnostic verification

    Data Visualization River Forecasting

    • Error

    • MAE, RMSE

    • Conditional on lead time, year

    • Skill

    • Skill relative to

    • Climatology

    • Conditional

    • Categorical

    • FAR, POD, contingency table (based on climatology or user definable)

    Dissemination of diagnostic verification

    • Example: WR water supply website

    http://www.nwrfc.noaa.gov/westernwater/


    Dissemination of diagnostic verification1
    Dissemination of diagnostic verification River Forecasting

    • Example: OHRFC bubble plot online

    http://www.erh.noaa.gov/ohrfc/bubbles.php


    Real time verification
    Real-time verification River Forecasting

    • How good could the ‘live’ forecast be?

    Live forecast

    Observations


    Real time verification1

    Analog 3 River Forecasting

    Analog 2

    Observed

    Live forecast

    Analog Observed

    Analog Forecast

    Real-time verification

    • Select analogs from a pre-defined set of historical events and compare with ‘live’ forecast

    Analog 1

    “Live forecast for Flood is likely to be too high”


    Real time verification2
    Real-time verification River Forecasting

    • Adjust ‘live’ forecast based on info from the historical analogs

    Live forecast

    What happened

    Live forecast was too high


    Real time verification3
    Real-time verification River Forecasting

    • Example for ensemble forecasts

    Live forecast (L)

    Analog forecasts (H):μH = μL ± 1.0˚C

    Analog observations

    Temperature (oF)

    “Day 1 forecast is probably too high”

    Forecast lead day


    Real time verification4
    Real-time verification River Forecasting

    • Build analog query prototype using multiple criteria

      • Seeking analogs for precipitation: “Give me past forecasts for the 10 largest events relative to hurricanes for this basin.”

      • Seeking analogs for temperature: “Give me all past forecasts with lead time 12 hours whose ensemble mean was within 5% of the live ensemble mean.”

      • Seeking analogs for flow: “Give me all past forecasts with lead times of 12-48 hours whose probability of flooding was >=0.95, where the basin-averaged soil-moisture was > x and the immediately prior observed flow exceeded y at the forecast issue time”.

    Requires forecasters’ input!


    Outstanding science issues
    Outstanding science issues River Forecasting

    • Define meaningful reference forecasts for skill scores

    • Separate timing error and amplitude error in forecasts

    • Verify rare events and specify sampling uncertainty in metrics

    • Analyze sources of uncertainty and error in forecasts

    • Consistently verify forecasts on multiple space and time scales

    • Verify multivariate forecasts (issued at multiple locations and for multiple time steps) by accounting for statistical dependencies

    • Account for observational error (measurement and representativeness errors) and rating curve error

    • Account for non-stationarity (e.g., climate change)


    Verification service development
    Verification service development River Forecasting

    OHD-NCEP

    Thorpex-Hydro project

    OHD

    OCWWS

    NCEP

    Forecasters Users

    Academia

    Forecast agencies

    Private

    COMET-OHD-OCWWS

    collaboration on training

    OHD-Deltares collaboration for CHPS enhancements

    HEPEX Verification Test Bed

    (CMC, Hydro-Quebec, ECMWF)


    Looking ahead
    Looking ahead River Forecasting

    • 2012:

      • Info on quality of forecast service available online

      • real-time and diagnostic verification implemented in CHPS

      • RFC verification standard products available online along with forecasts

    • 2015:

      • Leveraging grid-based verification tools

    FUTURE


    Thank you questions
    Thank you River Forecasting Questions?

    FORECASTER

    FORECASTER

    [email protected]


    Extra slide
    Extra slide River Forecasting


    Diagnostic verification products7
    Diagnostic verification products River Forecasting

    • Key verification metrics from NWS Verification Team report


    ad