Application of the cra method
Download
1 / 45

Application of the CRA Method - PowerPoint PPT Presentation


  • 71 Views
  • Updated On :

Application of the CRA Method . William A. Gallus, Jr. Iowa State University Beth Ebert Center for Australian Weather and Climate Research Bureau of Meteorology. (4) 125 pts to the right and rotated. (5) 125 pts to the right and huge. (3) 125 pts to the right and big. Observed.

Related searches for Application of the CRA Method

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Application of the CRA Method' - jacqui


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Application of the cra method

Application of the CRA Method

William A. Gallus, Jr.

Iowa State University

Beth Ebert

Center for Australian Weather and Climate Research

Bureau of Meteorology


Idealized cases geometric

(4) 125 pts to the right and rotated

(5) 125 pts to the right and huge

(3) 125 pts to the right and big

Observed

(1) 50 pts to the right

(2) 200 pts to the right

Idealized cases - geometric

Which forecast is best?



5 th case traditional verification
5 and 2th case – traditional verification

forecast

forecast

THE WINNER


1 st case cra verification
1 and 2st case – CRA verification


2 nd case cra verification
2 and 2nd case – CRA verification

CRA Technique yields similar results with cases 3 and 4


5 th case cra verification
5 and 2th case – CRA verification





1 st case vs 5 th case
1 additional changest case vs. 5th case

THE WINNER


Perturbed cases

1000 km additional change

Perturbed cases

(1) Shift 24 pts right, 40 pts down

"Observed"

(2) Shift 12 pts right, 20 pts down, intensity*1.5

Which forecast is better?


1 st case traditional verification
1 additional changest case – traditional verification

(1) Shift 24 pts right, 40 pts down


2 nd case traditional verification
2 additional changend case – traditional verification

(2) Shift 12 pts right, 20 pts down, intensity*1.5

THE WINNER


Cra verification
CRA verification additional change

Threshold=5 mm/h

Case 1

Case 2


Cra verification1
CRA verification additional change

Threshold=5 mm/h

Case 1

Case 2


Cra verification2
CRA verification additional change

Threshold=5 mm/h

Case 1

Case 2


Problem? additional change

System far from boundary in Case 1 – small shift – behaves as expected






Summary
Summary all error is displacement

  • CRA requirement for forecast and observed systems to be contiguous may limit some applications

  • Problems occur for systems near the domain boundaries – not yet clear what causes the problems


Results from separate study using object oriented techniques to verify ensembles
Results from separate study using object-oriented techniques to verify ensembles

  • Both CRA and MODE have been applied to 6-hr forecasts from two 15km 8 member WRF ensembles integrated for60 h for 72 cases

  • This results in 10 x 16 x 72 = 11,520 evaluations (plots, tables….) from each approach

  • Results were compared to Clark et al (2008) study


Clark et al study
Clark et al. study to verify ensembles

  • Clark et al. (2008) looked at two 8 member WRF ensembles, one using mixed IC/LBC, the other mixed physics/dynamic cores

  • Spread & skill initially may have been better in mixed physics ensemble vs. IC/LBC one, but spread grew much faster in the IC/LBC one, and it performed better than the mixed physics ensemble at later times (after 30-36 h) in these 120 h integrations.


0.5 mm to verify ensembles

2.5 mm

Areas under ROC curves for both ensembles (Clark et al. 2007)

Skill initially better in mixed ensemble but IC/LBC becomes better after hour 30-36


Diurnal Cycle to verify ensembles

Variance continues to grow in IC/LBC ensemble but levels off after hour 30 in mixed ensemble. MSE always worse for mean of mixed ensemble – and performance worsens with time relative to IC/LBC ensemble.


0.5 mm to verify ensembles

2.5 mm

Spread Ratio also shows dramatically different behavior with increasing spread in IC/LBC ensemble but little or no growth in mixed ensemble after first 24 hours


Questions
Questions: to verify ensembles

  • Do the object parameters from the CRA and MODE techniques show the different behaviors between the Mix and IC/LBC ensembles?

  • Do the object parameters from the CRA and MODE techniques show an influence from the diurnal trends in observed precipitation?


Diurnal signal not pronounced, only weak hint of IC/LBC tendency to have increasing spread with time – and only in MODE results

Rain Rate Standard Deviation (in.) – mean usually around .5 inch

Mix-MODE

IC/LBC-MODE

Mix-CRA

Wet times in blue

IC/LBC-CRA

06 12 18 24 30 36 42 48 54 60

Forecast Hour


Standard Deviation of Rain Volume (km tendency to have increasing spread with time – and only in MODE results3) – MODE values multiplied by 10 (mean ~ 1)

No diurnal signal, hard to see different trends between 2 ensembles

Mix-MODE

IC/LBC-CRA

Mix-CRA

IC/LBC-MODE

06 12 18 24 30 36 42 48 54 60


CRA results show both ensembles with growing spread, and IC/LBC having faster growth

Areal Coverage Standard Deviation (number of points above .25 inch) – Mean ~ 800 pts

IC/LBC-CRA

Mix-CRA

Mix-MODE

IC/LBC-MODE

06 12 18 24 30 36 42 48 54 60


No clear diurnal signal, both CRA & MODE show max in 24-48 h IC/LBC having faster growth

Mix-MODE

IC/LBC-MODE

Mix-CRA

IC/LBC-CRA

06 12 18 24 30 36 42 48 54 60


Mix-CRA IC/LBC having faster growth

IC/LBC-MODE

Mix-MODE

IC/LBC-CRA

No diurnal signal, no obvious differences in behavior of Mix and IC/LBC

06 12 18 24 30 36 42 48 54 60


Other questions
Other questions: IC/LBC having faster growth

  • Is the mean of the ensemble’s distribution of object-based parameters a good forecast (better than ensemble mean put into CRA/MODE)?

  • Does an increase in spread imply less predictability?

  • How should a forecaster handle a case where only a subset of members show an object?

These questions have been examined using CRA results


Mix Ensemble – in general, slight positive bias in rain rate, with Probability Matching forecast slightly less intense than mean of rates from members (PM usually better but not by much). Only during 06-18 period does observed rate not fall within forecasted range.

wet

mean

PM

dry

06 12 18 24 30 36 42 48 54 60

wet

IC/LBC – usually too dry with rain rate (at all hours except 06-18), Probability Matching forecast exhibits much more variable behavior, again its performance is comparable to mean of rates of members

mean

PM

dry

06 12 18 24 30 36 42 48 54 60


Notice that at all times, the observed rain rate falls within the range of values from the full 16 member ensemble – indicating potential value for forecasting


Mix Ensemble – clear diurnal signal, usually too much rain volume except at times of observed peak, when it is too small. Probability Matching equal in skill to mean of member volumes

wet

mean

PM

dry

06 12 18 24 30 36 42 48 54 60

wet

mean

IC/LBC Ensemble – also clear diurnal signal, less volume than Mix ensemble, Probability Matching usually a little wetter but generally comparable to mean of members

PM

dry

06 12 18 24 30 36 42 48 54 60


NOTE: Even with all 16 members, there are still times when observed volume does NOT fall within range of predictions --- not enough spread (indicated with red bar)

Mix

Mix-PM

Mix

IC/LBC

IC/LBC-PM

IC/LBC

Mix

IC/LBC

06 12 18 24 30 36 42 48 54 60


Percentage of times the observed value fell within the min/max of the ensemble

Rate - Mix

Volume-Mix

Volume-IC/LBC

Rate-IC/LBC

IC/LBC

Areal Coverage

Mix


Skill (MAE) as a function of spread (> 1.5*SD cases vs < .5*SD cases)

CRA applied to Mix Ensemble (IC/LBC similar)

Area/1000 big SD

Vol big SD

Rate*10 low SD

Rate*10 big SD

Vol low SD

Area/1000 low SD

06 12 18 24 30 36 42 48 54 60


  • It thus appears that total system rain volume and total system areal coverage of rainfall show a clear signal for better skill when spread is smaller

  • Rain rate does not show such a clear signal – (especially when 4 bins of SDs are examined).

  • Perhaps average rain rate for systems is not as big a problem in the forecasts as areal coverage (and thus volume)? Seems to be ~5-10% error for rate, 10-20% for volume, 10-20% for area


Summary1
Summary system areal coverage of rainfall show a clear signal for better skill when spread is smaller

  • Ensemble spread behavior for object-oriented parameters may not behave like traditional ensemble measures

  • Some similarities but some differences also in output from CRA vs MODE

  • Some suggestion that ensembles may give useful information on probability of systems having a particular size, intensity, volume


Acknowledgments
Acknowledgments system areal coverage of rainfall show a clear signal for better skill when spread is smaller

  • Thanks to Eric (and others?) for organizing the workshop

  • Thanks to John Halley-Gotway and Randy Bullock for help with MODE runs for ensemble work, and Adam Clark for precip forecast output

  • Partial support for the work was provided by NSF Grant ATM-0537043


Gulf near-boundary system with increased rainfall system areal coverage of rainfall show a clear signal for better skill when spread is smaller


ad