Ensemble forecasting
1 / 75

Ensemble Forecasting - PowerPoint PPT Presentation

  • Updated On :

Ensemble Forecasting. Yuejian Zhu Ensemble & Probabilistic Guidance Team Environmental Modeling Center December 7 th 2010 Acknowledgment for: Jun Du, Dingchen Hou, Geoff DiMego, John Ward, Hendrik Tolman Bill Lapenta and Steve Lord. Outline. Service for users

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Ensemble Forecasting' - Gideon

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Ensemble forecasting l.jpg

Ensemble Forecasting

Yuejian Zhu

Ensemble & Probabilistic Guidance Team

Environmental Modeling Center

December 7th 2010

Acknowledgment for:

Jun Du, Dingchen Hou, Geoff DiMego, John Ward, Hendrik Tolman

Bill Lapenta and Steve Lord

Outline l.jpg

  • Service for users

    • Requirements / decision support

    • Ensemble’s responsibilities

  • Upgrades and Plans for

    • SREF

    • Ocean Wave Ensemble

    • GEFS



  • Ensemble processing plans

    • Bias correction

    • Downscaling

    • Products

User requirements simplified l.jpg
User Requirements - Simplified

  • Applications affected by (extreme/high impact) weather

    • Must consider information on weather to

      • Minimize losses due to adverse weather

  • Optimal user decision threshold equals

    • Probability of adverse weather exceeding

      • Cost / loss ratio of decision situation (simplified decision theory)

  • Probability of weather events must be provided

    • Only option in past, based on error statistics of single value forecasts

      • e.g., MOS POP

      • Now can be based on ensemble statistical information (e.g., RMOP)

    • Users act when forecast probability exceeds their cost/loss ratio (example)

  • Advantages

    • A set of products (e.g., 10 / 50 / 90 percentile forecast , metagram, mean and mode)

  • Advanced - Problems???

    • Proliferation of number of products

      • For different variables, probability / weather element thresholds, joint probabilities

    • Limited usage

      • Downstream applications severely limited (e. g., wave, streamflow, etc, ensembles not possible)

User requirements advanced l.jpg
User Requirements - Advanced

  • Advanced information

    • Statistical reliable ensemble forecast products

    • Ensemble statistical data – historical information

    • 6D-cube – space (3D) + time + variables + ensemble

      • Expanded NDFD – future official NWS weather / climate / water forecast database

  • Joint probabilities

    • Many variables, different probabilities / critical value decision thresholds

    • Some (or many) of forecast events are related joint probabilities.

      • Probability of significant convection

      • Fire weather

  • User application model (UAM)

    • Must be easy operating with quick information access

    • Simulating optimal weather related operations

    • Simulating different user procedures for multiple plausible weather scenarios

    • Able to tell – what are actions / costs / benefits? - assuming weather is known

  • Application

    • Run UAM n x n times with multiple weather scenarios from each ensemble member (n) and user procedures (n)

    • Weather scenario from each ensemble - generated from optimized user procedures

    • Take ensemble mean of economic outcome (costs + losses) for each set of user procedures

    • Choose operating procedures to minimize costs and losses in expected sense.

    • Make optimizing weather related decisions

  • Challenge

    • Requires - storage / telecom bandwidth

    • Requires - smart sub-setting & interrogation tools – can derive any weather related information include joint probabilities

Responsibilities of ensemble team l.jpg
Responsibilities of Ensemble Team

- Assess, model, communicate uncertainty in numerical forecasts

  • Present uncertainty in numerical forecasting

    • Tasks

      • Design, implement, maintain, and continuously improve ensemble systems

    • Topics

      • Initial value related uncertainty

      • Model related forecast uncertainty

    • Ensemble systems

      • Global – GEFS / NAEFS

      • Regional – SREF / HREF / VSREF / HWAF ensemble

      • Climate – Contributions to future CFS configuration

      • NAEFS/GEFS downscaled

      • Ocean wave ensemble (MMA/EMC)

  • Statistical correction of ensemble forecasts

    • Tasks

      • Correct for systematic errors on model grid

      • Downscale information to fine resolution grid (NDFD)

      • Combine all forecast info into single ensemble/probabilistic guidance

  • Probabilistic product generation / user applications

    • Contribute to design of probabilistic products

    • Support use of ensembles by

      • Internal users (NCEP Service Center, WFOs, OHD/RFC forecasters and et al.)

      • External users (research, development, and applications

Review sref implementation oct 27 th 2009 l.jpg
Review SREF Implementation (Oct. 27th 2009)

  • Upgrade model versions

    • WRF-NMM from v2.0+ to v2.2+

    • WRF-ARW from v2.0+ to v2.2+

    • RSM from v2007 to v2009

  • Increase horizontal resolution

    • WRF-NMM from 40km to 32km

    • WRF-ARW from 45km to 35km

    • RSM from 45km to 32km

  • Adjust membership – total membership = 21

    • Replace 2 Eta (BMJ-sat) members with 2 WRF-NMM members

    • Replace 2 Eta (KF-det) members with 2 WRF-ARW members

  • Enhancement physics diversity of RSM: replace Zhao cloud scheme with Ferrier cloud scheme for 3 SAS members

  • Enhance initial perturbation diversity: Replace regional bred perturbations with global ET perturbations for 10 WRF members

  • Improvements (T2m, precipitation)

  • Add/fix/unify variables in SREF output

    • Increase output frequency from every 3hr to hourly for 1st 39hr (for SPC, AWC)

    • Wind variation products (for DTRA)

    • Radar (composite reflectivity + echo top) (for FAA)

    • Etc …

Next sref implementation plan q4fy2011 geoff dimego and jun du l.jpg
Next SREF Implementation Plan (Q4FY2011)- Geoff DiMego and Jun Du

Models and configurations

  • Add NMMB members (7), WRF-ARW (2) and WRF-NMM (2) to replace RSM (5) and Eta members (6)

  • Future configuration of SREF will be

    • 7 NEMS_NMMB, 7 WRF_NMM and 7 WRF-ARW – totally 21 members

  • Update WRF-NMM and WRF-ARW model versions

  • NMMB version is under NEMS infrastructure

  • Resolutions (versions):

    • NEMS_NMMB – 22km

    • WRF_NMM – 22km

    • WRF_ARW – 22km

  • Downscaled ETR perturbations from global ensemble

  • Output same fields as the current operational SREF

  • Performance evaluation

    Post process (and regional ensemble related)

  • Precipitation calibration (starting NCO evaluation now)

  • HREF – dynamical downscaling from NMM and ARW, 44m, 4km, out to 48 hours

  • VSREF – lag ensemble from RUC and NMM, 11m, 12km, hourly out to 12hours

Sref implementation plan for fy12 15 geoff dimego and jun du l.jpg
SREF Implementation Plan for FY12-15- Geoff DiMego and Jun Du

North America Ensemble Forecast System’s extension to regional ensemble (NAEFS_LAM)

  • Need - Both U.S. and Canada need to run high-resolution regional ensembles for high-impact weather.

  • Benefit - More resource can be spent on increasing model resolution but not on increasing ensemble membership as well as increasing forecast diversity.

  • Canadian REPS: 20 members with GEM, downscaled GEFS IC perturbations, 30km, 48hr, NA domain, later 2010 implementation (B08RDP, VC2010, Haiti earth quake relief effort) (21+20=41 member SREF)

  • Problem - Both countries are big in domain and don’t have enough computing resources to run such a high-resolution ensemble with large enough ensemble size.

    NRRE - NAM Rapid Refresh Ensemble, NA domain, 12km, hourly, 24hr for aviation forecast.

    HRRE – Hi-res Rapid Refresh Ensemble, smaller domains, 3km, hourly, 24hr, for storm scale events

    HWRF ensemble – in testing – supported by HFIP program


  • Stochastic physics

    • Convective parameterization of Teixeira et al - NRL

  • Ensemble transform with rescaling (ETR) initial perturbations

    • Consistent with boundary perturbations from GEFS

  • Resolution

    • Looking for higher resolution

Ocean wave ensemble system hendrik tolman l.jpg
Ocean Wave Ensemble System- Hendrik Tolman

  • Configuration of ocean wave ensemble system

    Wave ensemble has been running since 2008

    • Running on 1°×1° wave model grid as the control.

    • 20 wave members generated through GEFS using ETR method

    • Cycling initial conditions for individual members to introduce uncertainty in swell results.

    • 10 day forecast using the GEFS bias corrected 10m wind (future operation)

  • Improving forecast uncertainties through

    • Introducing ensemble initial perturbations from previous model cycle

    • Introducing bias corrected ensembles as external forcing.

    • Example of comparison (wave heights)

  • Plans

    • Work towards a combined NCEP-FNMOC ensemble

    • Analyze the role of swell played in the wave ensemble

Review gefs implementation feb 23 rd 2010 l.jpg
Review GEFS Implementation (Feb. 23rd 2010)

  • Using current operational GFS version 8.0

  • Upgrade horizontal resolution from T126 to T190

    • 4 cycles per day, 20+1 members per cycle

    • Up to 384 hours (16 days)

  • Use 8th order horizontal diffusion for all resolutions

    • Improved forecast skills and ensemble spread

  • Introduce ESMF (Earth System Modeling Framework) for GEFS

    • Version 3.1.0rp2

    • Allows concurrent generation of all ensemble members

    • Needed for efficiency of stochastic perturbation scheme

  • Add stochastic perturbation scheme to account for random model errors

    • Increased ensemble spread and forecast skill (reliability)

  • Add new variables (28 more) to pgrba files

    • Based on user request

    • From current 52 (variables) to future 80 (variables)

    • For NAEFS ensemble data exchange

  • What do we expect from coming GEFS implementation?

    • For large scale (see 500hPa height AC for NH)

    • For ensemble distributions (850hPa temperature CRPS and RMS/SPREAD)

    • For tropical storms (track errors)

Next gefs implementation plan q4 fy11 l.jpg
Next GEFS Implementation Plan (Q4 FY11)

According to total resource distribution for each model (Jigsaw puzzle)

GEFS has 40% of total CPUs (52 of 130) during +4:35 and +6:00 for main integration and main post-process

Current: GEFS and GEFS/NAEFS post processing

T190L28 for all 384 hours lead-time

20+1 members per cycle, 4 cycles per day

Computation usage: average 20 nodes (22 high mark) for 50 minutes

Next GEFS and GEFS/NAEFS post processing (Q4FY2011):

T254L42 (0-192hr) – increasing both horizontal and vertical resolutions

Factor of 3.6 by comparing T190L28

T190L42 (192-384hr) – increasing vertical resolution

Factor of 1.5 by comparing T190L28

20+1 members per cycle, 4 cycles per day

Total cost for integration and post processing

Factor of 3.6 for 0-192hrs, factor of 1.5 for 192-384

Average factor for processing (0-384hrs) is 2.55

51 nodes for 50 minutes (start: +4:35 end: +5:25)

Products will be delayed by approximately 20 minutes because CCS can’t offer 51 nodes

40 nodes for 70 minutes (start +4:35 end: +5:45)

Why do we make this configurations?

Considering the limited resources

Resolution makes difference (T126 .vs T190)

What do we expect from this implementation?

Preliminary results (NH 500hPa and SH 500hPa height)

Gefs implementation plan for fy12 15 l.jpg
GEFS Implementation Plan for FY12-15

Hybrid data assimilation based GEFS initializations

Using 6-hr EnKF forecast combined improved ETR (without cycling) (Schematic diagram)

Improved ETR

Adaptive modification of initial and stochastic model perturbation variances

Based on recursive average monitoring of forecast errors and ensemble spread

Avoid having to tune perturbation size after each analysis/model/ensemble changes

Improving performance and easy maintenance

Real time generation of hind-casts (plan)

Make control forecast once every ~5th day (6 runs for each cycle)

T254L42 (0-192) and T190L42 (192-384), and use new reanalysis (~30y)

Increasing sample of analysis – forecast pairs for statistic corrections

Improving bias correction beyond 5-d

Potential for regime/situation dependent bias correction

Coupled ocean-land-atmosphere ensemble

Couple MOM4/HYCOM with land-atmosphere component using ESMF

Depending on skill, extend integration to 35 days

Merge forecasts with CFS ensemble for seamless weather climate interface

Land perturbation and surface perturbation (later)

Explore predictability in intra-seasonal time scale (week 3-4)

Potential skill beyond 15 days

Hydro-meteorological (river flow) ensemble forecasting

Pending on operational LDAS/GLDAS

Ensemble post processing steps l.jpg
Ensemble Post Processing Steps

  • Bias correction / combination of all information

    • Remove lead time dependent model bias on model grid

    • Calibrate higher moments of ensemble

    • Combine information from all sources into single set of ensemble

  • Forecaster modification

    • Subjective changes to ensemble data (over US only? Pending on?)

  • Proxy for truth

    • Create observationally based fine resolution analysis for use in downscaling

  • Downscaling

    • Interpret bias corrected ensemble on user relevant grid – NDFD

  • Additional variables

    • Derive further variables from bias corrected / downscaled NWP output

  • Visualization / Derived products

    • Interrogate bias corrected / downscaled ensemble dataset

Bias correction combination of all information l.jpg
Bias Correction / Combination of All Information

  • Method (current and plan)

    • Kaman filter method (decaying average) – current in NAEFS operation

      • Multi-model multi-center ensembles

    • Based on Bayesian principle (future plan – THORPEX proposal)

      • Combines information from all sources

      • Fuses forecast data with climatological distribution (“prior”)

      • Adjusts “spread” according to skill observed in forecast sample

      • Outputs statistically corrected distribution (“posterior”)

        • Ensemble members adjusted to represent posterior distribution

  • Data sources

    • Reanalysis as prior (use new reanalysis when available)

    • Sample of past forecasts - most recent 60-90 days

      • Include control hind-casts when available

    • Sample of reforecast – Tom Hamill’s approach

    • Latest analyzed or observed data

  • Current status

    • 35 (will be 49 for NAEFS - table ) NAEFS & SREF variables bias corrected

      • 1st moment corrected only

      • NAEFS & SREF processed separately

    • CMC + NCEP ensembles + GFS hires control combined

  • Plan

    • Bias correct all model output variables on model native grid (2-3 yrs)

      • Include precipitation (use observationally based analysis as truth – 1-2 yrs)

    • Add hind-casts for NCEP ensemble calibration (1-2 yrs)

    • Combine all forecasts into single guidance (2-3 yrs)

      • Ensemble & hires from NCEP (GEFS and SREF), CMC, FNMOC, ECMWF

Downscaling statistic l.jpg
Downscaling (statistic)

  • Method

    • Perfect prognostic

      • Establish relationship (“donwscaling vector”) between

        • Proxy for truth (high resolution observationally based analysis) &

        • NWP analysis (used as reference in bias correction step)

      • Level of sophistication

        • Climatological (statistical)

        • Regime dependent (statistical)

        • Case dependent (dynamical, using LAM) – most expensive

      • Sub-NWP-grid resolution variance

        • Need to be stochastically added in statistical methods

      • Outputs ensemble members statistically consistent with

        • Bias corrected forecasts on NWP grid

        • Proxy for truth on fine resolution grid

  • Data sources

    • Sample of

      • Fine resolution observationally based analysis fields

      • Corresponding NWP analysis fields

  • Current status (table)

    • RTMA used proxy for truth

    • 4 NDFD variables downscaled using regime dependent downscaling vector for COUNS

      • Surface pressure, T2m, U10m and V10m

    • 8 NDFD variables downscaled for Alaska (Q1FY11)

      • Surface pressure, T2m, Tmax, Tmin, U10m, V10m, Wdir and Wspd

  • Plan

    • Add more NDFD variables by

      • Expanding RTMA analysis & using derived variables (e.g., dew point temperature and etc…)

      • Using SMARTINIT + downscaled NDFD variables

High resolution ensemble forecast href l.jpg
High-Resolution Ensemble Forecast (HREF)

HREF is a dynamically downscaled ensemble using the dual-resolution “Hybrid Ensembling” method (Du 2004) to superimpose forecast variances from a low-res ensemble onto a hi-res single run. In this case, the 32km 21-member SREF was combined with 4km Hi-res window’s NMM and ARW two single runs to produce a 44-member 4km ensemble over US East/US West/Alaska three domains. Below shows an ensemble mean precipitation forecast before and after downscaled. To be implemented together with Hi-Res window run package (Q2FY11, Matt Pyle).

Very short range ensemble forecast vsref l.jpg
Very-Short Range Ensemble Forecast (VSREF)

  • VSREF is a time-lagged ensemble based on 12km NAM and RUC) with 11 members for the purpose of aviation forecasts. It runs up to 12 hour with hourly update cycle over the continental U.S. domain. The products are listed in Table 1 and display at the following web for real time use: http://www.emc.ncep.noaa.gov/mmb/SREF_avia/FCST/VSREF/web_site/html/icing.html.

Naefs upgrade and nuopc ioc l.jpg
NAEFS upgrade and NUOPC-IOC

  • Multi-model, multi-center ensemble

    • NCEP GFS high resolution deterministic

    • NCEP GFS based ensembles (GEFS)

    • CMC global ensembles

  • Current status

    • NCEP 20+1+1 members per cycle, 4 cycle per day

    • CMC 20+1 members per cycle, twice per day

    • Every 6 hours, out to 16 days

  • Products and Benefits

    • Bias corrected at 1*1 degree (35 variables, will be 49)

    • Hybrid with NCEP bias corrected GFS forecast

    • Combined (with adjusted) bias corrected CMC’s ensemble

      • Probabilistic products (10%, 50%, 90%, mean, mode and spread)

    • Downscaled products for CONUS (T2m MAE and CRPS) and Alaska (Q1FY11 – verification , HPC’s evaluation, Tmax and Windspeed)

    • Improving track skills from NAEFS (tracks, compare to GFS)

  • Experimental multi-model ensemble

    • Track skills from super-ensemble (AL+EP, WP)

    • Guidance of track forecast: strike probability, fuzzy map

  • NAEFS upgrade / NUOPC-IOC - include FNMOC ensemble

    • What do we get from this inclusion directly? (see T2m skills)

    • What do we get from downscaling? ( T2m skills)

Precipitation calibration plan l.jpg
Precipitation Calibration (Plan)

Background (QPF bias correction in NCEP)

Implemented May 2004 (HPC, CPC endorsed)

Bias corrected GFS/GEFS forecasts

At 2.5 degree resolution, every 24 hours, using Gauge (12UTC-12UTC)

Using decay average (or Kalman Filter) method for sampling

Using frequency match algorithm for CDF of OBS/FCST

Climatological Calibrated Precipitation Analysis (CCPA) – Q3FY10

Use 30year CPC unified analysis at 1/8 degree, daily, global land - reliability

Use 8year RFC/QPE (stage IV) 5km resolution, 6-h(CONUS) – resolution

Use regression method to generate a and b from above two datasets

Produce CCPA analysis ( CCPA = a*QPErfc + b)

Resolution is 5km (NDFD) grid (and subsets) for CONUS (verify 1 and 2)

Update QPF bias correction from #1 – Q4FY11

Bias corrected GFS/GEFS forecasts at 1.0 degree and 6 hours (example)

Bias corrected NAM/SREF forecasts at 30km and 1 hours (optional) (example)

Statistical downscaling to 5km – Q4 FY11

Proxy of truth - CCPA at NDFD grid (5km) or RTMA (if it creates)

Decaying average (or Kalman filter) methods to generate downscaling (DS) vectors

Downscaled forecasts

Based on bias corrected forecasts (#3), interpolated to 5km, applying DS vectors

Jointed development with ESRL/GSD through THORPEX

Final calibrated precipitation forecast with 2nd moment adjustment (FY12-13)

Multi-model ensemble after bias correction

Bayesian Process of Ensemble (BPE) (joined with GSD/ESRL through THORPEX)

Slide20 l.jpg

End-to-end Forecast Process








NCEP, other centers

Ensemble generation





MDL/NCEP, partners

Ensemble processing






QC, enhancements

Added value






Slide22 l.jpg

Optimal Threshold = 15%

Decision Theory Example



Critical Event: sfc winds > 50kt

Cost (of protecting): $150K

Loss (if damage ): $1M















Ranked probabilistic skill score l.jpg
Ranked Probabilistic Skill Score

CONUS 2 meter temperature

02 February – 10 August 2009



New SREF is more skillful than the old SREF


Slide26 l.jpg

Warm season 24h Accumulated Precip from EMC parallel (Mar. 12 – Aug. 30, 2009)

Red = new SREF

Black = old SREF




Comparison of the ensemble systems l.jpg
Comparison of the ensemble systems



Old ensemble setup, ensemble with cycling of initial conditions and wind bias correction (BC).

Mean wave height (contours) and spread (shading)

2008/03/28 t06z 120h forecast

cycle, BC


Nh anomaly correlation for 500hpa height period august 1 st september 30 th 2007 l.jpg
NH Anomaly Correlation for 500hPa HeightPeriod: August 1st – September 30th 2007

GEFSg is better than GFS at 48 hours

GEFSg could extend skillful forecast (60%) for 9+ days

24 hours better than current GEFS

48 hours better than current GFS


What do we expect from next gefs impl l.jpg
What do we expect from next GEFS impl. ?

RMSE / SPREAD 500 hPa Height

CRPS NH 850 hPa Temp

Extend current 5-day skill to 6.5-day


Slide30 l.jpg

STATS for all basins 00UTC only

2 months (August and September 2008)


Slide31 l.jpg

Resolution makes difference for Typhoon Morakot

Ini: 2009080600

T126 ensemble

T190 ensemble

Most models do not

make right forecasts

Ini: 2009080700

T126 ensemble

T190 ensemble


Slide32 l.jpg


+4:35 --- +5:15




+4:35 --- +5:35


+7:20 --- +7:30

20m late finish


+4:37 --- +5:17

NAEFS products start


+4:37 --- +5:37









+7:20 --- +7:22


+4:40 --- +5:24


+5:00 --- +5:44


+7:22 --- +7:26


+5:24 --- +6:45


+7:33 -- +8:08


+5:44 --- +6:40

20m late start

NAEFS products (2)

+8:08 --- +9:08

5m early finish

GEFS/NAEFS 6-hr window flow chart


Slide33 l.jpg

11 days

Skill line

10.5 days


Courtesy of Jessie Ma

Slide34 l.jpg




Courtesy of Jessie Ma

Slide35 l.jpg

Flow Chart for Hybrid Variation and Ensemble Data Assimilation System (HVEDAS) - concept







Ensemble fcst (1)

t=j-1  j

Ensemble fcst

t=j,  j+1

Lower resolution

Ensemble fcst (2)

t=j  16d




Ensemble Mean

Estimated Background

Error Covariance from

Ensemble Forecast

(6 hours)

Estimated Background

Error Covariance from

Ensemble Forecast

(6 hours)








Higher resolution

Slide36 l.jpg

Real Time Generation of Hind-cast dataset Assimilation System (HVEDAS) - concept

Today’s Julian Date


TJD + 30

TJD - 30

Actual ensemble generated today









Hind-casts for TJD+30 generated today

Hind-casts (or its statistics) for TJD+/- 30 saved on disc

Slide37 l.jpg

Future seamless forecast system Assimilation System (HVEDAS) - concept

NCEP/GEFS will plan for T254L42 (2010 GFS

version) resolution with tuned ETR initial

perturbations and adjusted STTP scheme

for 21 ensemble members, forecast out to

16 days and 4 cycles per day. Extended to 45

days at T126L28/42 resolution, 00UTC only

(coupling is still a issue?)

NAEFS will include FNMOC ensemble in 2011,

with improving post process which include

bias correction, dual resolution and down scaling

Main products:

ENSO predictions???

Seasonal forecast???

Main event








one month

Weather/Climate linkage


  • Main products:

  • Probabilistic forecasts for every 6-hr out to 16 days, 4 times per day: 10%, 50%, 90%, ensemble mean, mode and spread.

  • D6-10, week-2 temperature and precipitation probabilistic mean forecasts for above, below normal and normal forecast

  • MJO forecast (week 3 & 4 … )

Next Operational CFS will plan to be implemented

by Q2FY2011 with T126L64 atmospheric model

resolution (CFSv2, 2010version) which is fully

coupled with land, ocean and atmosphere

(GFS+MOM4+NOAH), 4 members per day (using

CFS reanalysis as initial conditions, one day older?),

integrate out to 9 months.

Future: initial perturbed CFS


Development of statistical post processing for naefs l.jpg
Development of Statistical Post-Processing for NAEFS Assimilation System (HVEDAS) - concept

  • Opportunities for improving the post-processor

    • Utilization of additional input information

      • More ensemble, high resolution control forecasts (hybrid?)

      • Using reforecast information to improve week-2 forecast and precipitation

      • Analysis field (such as RTMA and etc..)

    • Improving calibration technique

      • Calibration of higher moments (especially spread)

      • Use of objective weighting in input fields combination

      • Processing of additional variables with non-Gaussian distribution

    • Improve downscaling methods

Future Configuration of EMC Ensemble Post-Processor


Slide39 l.jpg

NEXT NAEFS pgrba_bc files Assimilation System (HVEDAS) - concept

(bias correction)


Slide40 l.jpg

NAEFS downscaling parameters and products Assimilation System (HVEDAS) - concept

Last update: May 1st 2010

(NDGD resolutions)

All products at 1*1 (lat/lon) degree globally

Ensemble mean, spread, 10%, 50%, 90% and mode


Slide41 l.jpg

2-m temp 10/90 probability forecast verification Assimilation System (HVEDAS) - conceptNorthern Hem, period of Dec. 2007 – Feb. 2008


3-month verifications


Top: 2-m temperature probabilistic

forecast (10% and 90%) verification

red: perfect, blue: raw, green: NAEFS

Left: example of probabilistic forecasts

(meteogram) for Washington DC, every

6-hr out to 16 days from 2008042300


Slide42 l.jpg

NCEP/GEFS raw forecast Assimilation System (HVEDAS) - concept

4+ days gain from NAEFS

NAEFS final products

From Bias correction (NCEP, CMC)

Dual-resolution (NCEP only)

Down-scaling (NCEP, CMC)

Combination of NCEP and CMC


Slide43 l.jpg

NCEP/GEFS raw forecast Assimilation System (HVEDAS) - concept

8+ days gain

NAEFS final products

From Bias correction (NCEP, CMC)

Dual-resolution (NCEP only)

Down-scaling (NCEP, CMC)

Combination of NCEP and CMC


Slide44 l.jpg

Application for Alaska region and HPC Alaska desk Assimilation System (HVEDAS) - concept

10-m U

Max Temp

Solid – RMS error

Dash - spread

Bias (absolute value)

10-m wind speed

Max Temp

Bias (absolute value)

CRPS – small is better


Slide45 l.jpg

Objective Evaluation Assimilation System (HVEDAS) - concept

Mean, 50th, and HPC best



Courtesy of Dave Novak

Slide46 l.jpg

50 Assimilation System (HVEDAS) - conceptth (median) and mean are best


Courtesy of Dave Novak

Slide47 l.jpg

Track forecast error for 2009 season (AL+EP+WP) Assimilation System (HVEDAS) - concept

Cases 240 223 196 169 144 110 75 42

NAEFS is combined NCEP (NCEPbc) and CMC’s (CMCbc) bias corrected ensemble and bias corrected GFS


Contributed by Dr. Jiayi Peng (EMC/NCEP)

Slide48 l.jpg

Track forecast error for 2009 season (AL+EP+WP) Assimilation System (HVEDAS) - concept

Cases 240 223 196 169 144 110 75 42

NAEFS is combined NCEP (NCEPbc) and CMC’s (CMCbc) bias corrected ensemble and bias corrected GFS


Contributed by Dr. Jiayi Peng (EMC/NCEP)

Slide49 l.jpg

NEMN – NCEP raw ensemble Assimilation System (HVEDAS) - concept

3EMN – NCEP + CMC + EC raw ensemble


S6MN – 3EMN + NCEPgfs + CMCgfs + ECgfs

OFCL – NHC official forecast


Courtesy of Jiayi Peng

Slide50 l.jpg

NEMN – NCEP raw ensemble Assimilation System (HVEDAS) - concept

3EMN – NCEP + CMC + EC raw ensemble


S6MN – 3EMN + NCEPgfs + CMCgfs + ECgfs

OFCL – JTWC official forecast


Courtesy of Jiayi Peng

Slide51 l.jpg

back Assimilation System (HVEDAS) - concept

Courtesy of Jiayi Peng

Slide52 l.jpg

TC Detection – Fuzzy Based Probabilistic Forecast Assimilation System (HVEDAS) - concept


Courtesy of Tsai and Lu (Taiwan)

Slide53 l.jpg

CRPSS for October Assimilation System (HVEDAS) - concept

E20s – NCEP raw ensemble

E20sb – NCEP bias corrected ensemble

E40nb – NAEFS

E60gb – NUOPC (planned)


E60gb (NUOPC) has similar skill as E40nb (NAEFS)



Slide54 l.jpg

2-meter temperature downscaling Assimilation System (HVEDAS) - concept

NCEP/GEFS raw forecast

NAEFS upgrade products

From: Bias correction (NCEP, CMC, FNMOC)

Dual-resolution (NCEP only)

Combination of NCEP and CMC

Down-scaling (NCEP, CMC, FNMOC)



Slide55 l.jpg

Verification against CPC at 1/8 degree Assimilation System (HVEDAS) - concept






Courtesy of Yan Luo

Slide56 l.jpg

Verification against CPC at 1/8 degree Assimilation System (HVEDAS) - concept

Results – RMSE and ABSE


Courtesy of Yan Luo

How the precipitation calibration system works upgrade from 2004 implementation l.jpg
How the Precipitation Calibration System Works Assimilation System (HVEDAS) - concept- Upgrade from 2004 implementation


6hrly FCSTs (6hr-384hr), valid at day i-1

cycle j


Valid at day i-1,cycle j

9 Thresholds

0.2, 1, 2, 3.2, 5, 7, 10, 15, 25




For day i, cycle j

CDFs (fcst)


For day i, cycle j, 6-384hr

Decaying Average Method

CDF i,,j= (1-W) * CDFi-1,j + W * CDFi,j



For day i, cycle j

CDFs (fcst)


For day i, cycle j, 6-384hr


CDF0: initialized from any a 30-day average of CDF

How the precipitation calibration system works upgrade from 2004 implementation58 l.jpg
How the Precipitation Calibration System Works Assimilation System (HVEDAS) - concept- Upgrade from 2004 implementation



For day i, cycle j

Frequency match algorithm


CDFs (fcst)


For day i, cycle j, 6-384hr



Linear inter/extrapolation

No bias correction on zero forecast value



INI at day i, cycle j, for 6-384hr

At each model grid



INI at day i, cycle j, for 6-384hr

At each model grid



Slide59 l.jpg

Precipitation calibration for 2009-2010 winter season (CONUS only)

Comparison for GFS and ensemble control (raw and bias corrected)

ETS for all lead-time

ETS for 0-6hr fcst

BIAS for all lead-time

BIAS for 0-6hr fcst

Perfect bias = 1.0


Courtesy of Yan Luo

The probabilistic scores (CRPS -not show here) is much improved as well.

We are still working on the different weights, different RFC regions, downscaled

to 5km as well. More results will come in soon. Plan for implementation: Q4FY11

Slide60 l.jpg
SREF product only): bias correcting precipitation forecasts(EMC parallel running, for Q2 or Q4FY11 implementation)

Individual member: absolute error of 3h-apcp

Ensemble mean: RMSE of 24h-apcp

Ensemble spread: 24h-apcp at 87hr

Probabilistic forecast: RPSS of 24h-apcp


Data on nomads l.jpg
Data on NOMADS only)

  • SREF grid221(North America, 30km) for individual members

  • SREFbias corrected grid212 (CONUS, 40km) for individual members

  • GEFS raw forecast (1*1 degree globally) for individual members

  • GEFS bias corrected (1*1 degree globally) for individual members and products

  • NAEFS probabilistic forecast (1*1 degree globally)

  • NAEFS downscaled probabilistic forecast (5*5 km CONUS only)

  • NAEFS downscaled probabilistic forecast (6*6 km Alaska)

Background62 l.jpg

Post process derived variables plan l.jpg
Post Process - Derived Variables (Plan) only)

  • Objective

    • Generate variables not carried in NWP models

    • Or variables can not be easy calibrated

      • E. g. relative humidity

  • Input data

    • Bias corrected and downscaled ensemble data (NWP model output)

  • Methods

    • Model “post-processing” algorithms

      • Apply after downscaling for variables affected by surface processes

    • SMARTINIT for global forecast – Geoff Manikin et al.

      • NDFD weather element generator

    • Other tools?

      • Text generation, etc?

Sref probability of stp ingredients time trends l.jpg
SREF Probability of STP Ingredients: Time Trends only)

24 hr SREF Forecast Valid 21 UTC 7 April 2006

Prob (MLCAPE > 1000 Jkg-1)


Prob (6 km Shear > 40 kt)


Prob (0-1 km SRH > 100 m2s-2)


Prob (MLLCL < 1000 m)


Prob (3h conv. Pcpn > 0.01 in)

Shaded Area Prob > 5%

Max 50%

Slide65 l.jpg

Background: QPF bias correction at NCEP only)

Implemented May 2004 (HPC and CPC endorsed)


  • Construct cumulative frequency distributions for forecast QPF & corresponding observed values

  • For each forecast value, find the observed value that has the same frequency as forecast value

  • Re-label forecast value with corresponding observed value


  • Observations used:

  • US RFC rain-gage network

  • ~10,000 obs. analyzed over model grid

  • Adaptive method, training data accumulated over:

  • Most recent ~30-day period – Decaying averaging

  • More weight on most recent data

    • Continental US

    • Linear inter/extrapolation

  • Corrections applied globally on model grid

    • Correction is function of forecast value

    • 2.5*2.5 degree spatial resolution

    • Every 24 hour forecast interval


Slide66 l.jpg

Track forecast error for 2010 season (AL+EP) only)

Cases 120 110 96 82 71 53 37 26



Contributed by Dr. Jiayi Peng (EMC/NCEP)

Slide67 l.jpg

Track forecast error for 2010 season (AL+EP) only)

Cases 121 110 96 82 71 53 37 26



Contributed by Dr. Jiayi Peng (EMC/NCEP)

Slide68 l.jpg


  • Bias assessment - separately for:

    • GFS and ensemble control fcsts

    • Each lead time

  • Bias correction - applied on:

    • Hi- & low resol. control forecasts

  • All ensemble member fcsts

  • At 0000 UTC initial time only

  • 24-hr amounts only (00-00, 12-12Z)

  • 2.5x2.5 lat/lon resolution

  • Bias-corrected QPF data – provided in:

  • enspost / ensstat files


1) QPF bias much reduced

2) PQPF bias reduced

3) Much enhanced probabilistic skill

Observed frequency

Slide69 l.jpg

Value-added by including FNMOC ensemble into NAEFS only)T2m: Against analysis (NCEP’s evaluation)


Stat. corr.

0.5 CRPS skill



Raw NCEP ensemble has modest skill (3.4d)

Statistically corrected NCEP ensemble has improved skill (4.8d)

Combined NCEP – CMC (NAEFS) show further increase in skill (6.2d)

Addition of FNMOC to NAEFS leads to modest improvement (6.7d)


Slide70 l.jpg

MDL GMOS & NAEFS Downscaled Forecast only)

Mean Absolute Error w.r.t. RTMA Average For Sept. 2007

12-h GMOS


12-h NAEFS Forecast

For CONUS: (unit: C)

NAEFS(1.01) : GMOS(1.59)

36% impr. over GMOS

Slide71 l.jpg

40 day average absolute errors of 2-meter temperature (NDFD has 12hr advantage)

COUNS only – verified against RTMA

Slide72 l.jpg

Overall temperature forecasts: Average over past 30 days: (20080929-20081028)

MAE Bias >10 err <3 err off. rank Best G. 2nd G. Worst G.

1 12-hr 2.44 0.7 0.1% 67.3% 1 out of 7 NAM40 65.4% NAM12 60.1% NGM80 44.4%

2 24-hr 2.84 1.0 0.3% 59.1% 2 out of 7 NAM40 60.3% NAM12 56.9% SREF 47.0%

3 36-hr 2.94 0.8 0.3% 57.8% 1 out of 7 NAM40 55.9% NAM12 52.6% NGM80 44.0%

4 48-hr 3.36 1.6 2.1% 52.8% 1 out of 7 MOSGd 48.9% NAM40 48.3% NGM80 12.9%

5 60-hr 3.26 1.0 1.7% 54.8% 1 out of 6 MOSGd 50.1% NAM12 48.8% NAM40 6.2%

6 72-hr 3.35 1.3 2.1% 53.1% 1 out of 5 MOSGd 49.9% NAM12 49.5% SREF 44.0%

7 84-hr 3.80 0.6 4.7% 49.0% 1 out of 5 NAEFS 48.6% SREF 44.5% NAM12 2.6%

8 96-hr 3.96 0.7 4.0% 44.4% 2 out of 4 NAEFS 46.2% HPCGd 42.6% MOSGd 40.6%

9 108-hr 4.43 0.9 5.5% 38.5% 2 out of 3 NAEFS 41.7% MOSGd 37.7% MOSGd 37.7%

10 120-hr 4.57 1.0 5.9% 36.6% 2 out of 4 NAEFS 40.9% HPCGd 36.5% MOSGd 36.3%

11 132-hr 4.83 0.7 7.8% 34.7% 1 out of 3 NAEFS 34.5% MOSGd 34.4% MOSGd 34.4%

12 144-hr 4.83 0.5 7.4% 34.7% 3 out of 4 HPCGd 36.4% NAEFS 35.5% MOSGd 33.3%

13 156-hr 5.43 0.1 11.9% 30.3% 3 out of 3 NAEFS 32.1% MOSGd 30.8% MOSGd 30.8%

14 168-hr 5.74 0.3 14.4% 27.7% 2 out of 4 HPCGd 27.7% MOSGd 26.9% NAEFS 26.1

Minimum temperature forecast: Average over past 30 days: (20080929-20081028)

1 12-hr 3.17 -1.2 1.0% 53.4% 3 out of 7 NAEFS 59.7% SREF 57.1% NGM80 21.8%

2 24-hr 3.03 -0.9 0.6% 55.5% 2 out of 7 SREF 57.2% NAEFS 54.2% NGM80 24.9%

3 36-hr 3.25 -0.8 0.9% 51.6% 3 out of 7 NAEFS 54.2% SREF 53.9% NGM80 23.2%

4 48-hr 3.94 -1.1 2.9% 43.2% 3 out of 7 NAEFS 51.9% SREF 45.8% NGM80 6.2%

5 60-hr 4.30 -0.4 4.4% 39.1% 4 out of 6 NAEFS 49.2% SREF 43.0% NAM40 8.9%

6 72-hr 4.76 0.1 6.4% 33.7% 5 out of 5 NAEFS 42.9% SREF 40.1% NAM12 35.2%

7 84-hr 4.85 0.3 7.5% 34.7% 2 out of 6 NAEFS 40.0% MOSGd 33.4% NAM12 8.9%

8 96-hr 5.24 0.4 13.0% 33.1% 1 out of 3 NAEFS 32.7% MOSGd 29.9% MOSGd 29.9%

9 108-hr 5.11 0.8 12.8% 35.4% 1 out of 4 HPCGd 34.5% NAEFS 32.1% MOSGd 30.5%

10 120-hr 5.31 0.7 12.0% 31.9% 1 out of 3 MOSGd 31.6% NAEFS 24.8% NAEFS 24.8%

11 132-hr 4.97 0.7 9.9% 35.1% 2 out of 4 HPCGd 38.0% MOSGd 30.9% NAEFS 27.2%

12 144-hr 5.42 0.6 15.0% 35.0% 1 out of 3 MOSGd 31.3% NAEFS 29.0% NAEFS 29.0%

13 156-hr 5.40 0.5 14.9% 35.7% 1 out of 4 HPCGd 32.9% MOSGd 32.7% NAEFS 23.4%

14 168-hr 5.46 1.1 17.7% 38.1% 1 out of 3 MOSGd 35.6% NAEFS 28.4% NAEFS 28.4%

Official Guidance: NGM80, NAM40, SREF, NAM12, MOSGd, HPCGd, NAEFS

Contributed by Richard Grumm (WFO)

Slide73 l.jpg

NAEFS Products Distribution (20080929-20081028)

Slide74 l.jpg

MDL GMOS & NAEFS Downscaled Forecast (20080929-20081028)

Mean Absolute Error w.r.t. RTMA Average For Sept. 2007

Valery Dagostaro, Kathy Gilbert,

Bo Cui, Yuejian Zhu

24-h GMOS


24-h NAEFS Forecast


NAEFS(1.45) : GMOS(1.72)

19% impr. over GMOS

Emc mdl collaboration l.jpg
EMC-MDL COLLABORATION (20080929-20081028)

  • Compare quality of current operational / experimental products

    • Gridded MOS vs. Downscaled NAEFS

      • Ongoing

        • Kathy Gilbert, Val Dragostano – Zoltan Toth, Bo Cui, Yuejian Zhu

      • Proxy for truth issue unresolved

        • Need observations independent of MOS

    • MDL experimental ensemble guidance vs. Downscaled NAEFS

      • 10/50/90 percentiles to be evaluated

        • Matt Peroutka & Zoltan Toth

      • Proxy for truth issue

  • Proxy for truth?

    • Agree on best proxy for truth

      • Collaborate on

        • Improving RTMA, including bias correction for FG

        • Creating best CONUS precipitation analysis & archive

  • Joint research into best downscaling methods?

    • Climate, regime, case dependent methods

    • Addition of fine temporal/spatial variability into ensemble