a forecast evaluation tool fet for cpc operational forecasts n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
A Forecast Evaluation Tool (FET) for CPC Operational Forecasts PowerPoint Presentation
Download Presentation
A Forecast Evaluation Tool (FET) for CPC Operational Forecasts

Loading in 2 Seconds...

play fullscreen
1 / 47

A Forecast Evaluation Tool (FET) for CPC Operational Forecasts - PowerPoint PPT Presentation


  • 86 Views
  • Uploaded on

A Forecast Evaluation Tool (FET) for CPC Operational Forecasts. Edward O’Lenic Chief, Operations Branch, NOAA-NWS-CPC. Outline. CPC operational products We have discovered users Users are diverse We need to know how users use products We need a new service paradigm

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'A Forecast Evaluation Tool (FET) for CPC Operational Forecasts' - verda


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
a forecast evaluation tool fet for cpc operational forecasts

A Forecast Evaluation Tool (FET) for CPC Operational Forecasts

Edward O’Lenic

Chief, Operations Branch, NOAA-NWS-CPC

outline
Outline
  • CPC operational products
  • We have discovered users
  • Users are diverse
  • We need to know how users use products
  • We need a new service paradigm
  • FET addresses these needs
  • Tour of FET
  • Current status
  • Implications
background
Background
  • People and organizations need planning tools.
  • For decades, climate forecasts have been sought for use in planning.
  • The obvious signs of climate change have made this desire more urgent.
  • The uncertain nature of climate forecasts can lead the uninformed to make poor choices.
  • Users need an “honest broker” they can trust.
current operational capabilities
Current Operational Capabilities
  • CPC products, include extreme events for days 3-14, extended-range forecasts of T, P for week 2, 1-month and 3-month outlooks, and ENSO forecasts.
  • These products have a scientific basis, have skill, and therefore ought to be useful, but -
  • We have a less-than adequate understanding of user requirements, or actual practical use.
  • CPC has neither the staff, nor the expertise to effectively pursue understanding these things.
challenges from users
Challenges from Users

Western Governor’s Association (Jones, 2007):

- More accurate, finer-resolution long range forecasts

- Continued and expanded funding for data collection, monitoring and prediction

- Partnerships with federal and state climatologists, RCCs, agricultural extension services, resource management agencies, federal, state and local governments.

USDA ARS Grazinglands Research Laboratory (Schneider, 2002)

- Fewer “EC” forecasts

- Better correspondence between F probability and O frequency

- Forecast more useful than climatology

- Forecasts of impacts, not meteorological variables

bridging research operations users
Bridging Research, Operations, Users

Know WHO stakeholders are and HOW they USE climate products, (relationships),

LEARN from stakeholders WHAT we need to provide (iterative refinement of requirements, relationships),

Implement USER-, and SCIENCE-VETTED products, (iterative refinement of requirements, relationships),

Operationally support evolving user and producer REQUIREMENTS (extension function, relationships, operations, research)

?

Use it

if you can

Assumed User Needs

Basic data and Forecasts

GOVT. PROVIDERS

Basic Research

bridging research operations users1
Bridging Research, Operations, Users

users

?

Use it

if you can

Iterative Use

refinement

Assumed User Needs

Intermediary Applications Products

CTB-RISA/PRIVATE/RCC/SC-SBIR,

NIDIS

Decision-Support Development

REQUIREMENTS

Iterative

Technical

Refinement

Basic data, Forecasts

Transfer

User-Vetted

products

Transfer

User-Vetted

products

GOV PROVIDERS (CPC/EMC)

NCEP/NCDC/USGS

CTB SUPPOPRT

PROVIDERS

GOV

OPS

PRIVATE

OPS

R2O

R2O

O2R

O2R

R2O: CTB

O2R: Model Test Facility

RESEARCH, MODELING (dyn, stat), OBSERVATIONS

Research, Modeling, Obs

slide11

Лроверяй, и Доверяй,

Verify, and Trust

Trust is at the core of a successful product suite.

Transparent verification is one way to secure trust.

slide12
Лроверяй, и Доверяй,

Verify, and Trust

Trust is at the core of a successful product suite.

Transparent verification is one way to secure trust.

VERIFICATION: A MEASURE OF FORECAST

QUALITY, SKILL AND VALUE

types of verification
Types of Verification
  • Accuracy – e.g., AC, error, #correct, rmse, mae, etc…
  • Skill –

e.g., HSS, RPSS, Brier SS

  • Bias – forecast too high/low?
  • Resolution – how well are different events discriminated?
  • Sharpness – able to predict extreme events?
how to proceed
How to Proceed?

Over the last decade, Dr. Holly C. Hartmann, and programmers Ellen Lay and Damian Hammond, of CLIMAS, have developed an interactive, on-line “Forecast Evaluation Tool” (FET) which allows users to evaluate the meaning and skill of CPC 3-Month Outlooks of temperature and precipitation.

CPC proposes to:

  • Make the FET CPC’s outlet to users for forecast skill information,
  • Become a partner with CLIMAS and others to make the FET a community resource.
  • Expand the capabilities of FET
forecast evaluation tool example of a means to address gaps
Forecast Evaluation Tool: Example of a Means to Address Gaps

What FET provides:

  • User-centric forecast evaluation and data access and display capability.
  • Leveraging of community software development capabilities.
  • Opportunity to DISCOVER and collect user requirements.
cpc temperature and precipitation outlooks
CPC Temperature and Precipitation Outlooks
  • Probability of three categories
  • Bottom, middle, top 10 years, 1971-2000
  • Maps show probability of likeliest category
  • Each category has a value at every point
  • Probabilities sum to 100%
  • If middle favored, borrow from extremes
  • “EC” means pr(b,n,a)=33.33, 33.33, 33.33
cpc temperature and precipitation outlooks1
CPC Temperature and Precipitation Outlooks

Pr(b,n,a)= 16.67, 33.33, 50

  • Probability of three categories
  • Bottom, middle, top 10 years, 1971-2000
  • Maps show probability of likeliest category
  • Each category has a value at every point
  • Probabilities sum to 100%
  • If middle favored, borrow from extremes
  • “EC” means pr(b,n,a)=33.33, 33.33, 33.33

Pr(b,n,a)=40, 33.33, 26.67

recent cpc skill improvements
Recent CPC Skill Improvements

HSS=% improvement the forecast makes over random forecasts.

More non-EC forecasts, and a large HSS are GOOD.

Non-EC

forecasts

O’Lenic et al (2008) compared the HSS and % non-EC of official (OFF) forecasts

made in real-time from 1995 through 2004 with forecasts made using an objective

consolidation (CON) of the identical four main forecast tools which were used to

prepare the real-time forecasts. Period (1995-2004) mean T, P forecast skills for

OFF/CON were 22/26 for T, and 8.8/12.1 for P. CON % non-EC is also higher.

CPC began using CON as a first guess in 2006.

slide21

Top 2 rows: 1995-2004 HSS (lines) of 3-month P Outlooks, Official (OFF) and Consolidation (CON). Colors are the fraction of the time non-EC is predicted (%).Bottom row: Difference, CON-OFF (lines and colors). (See O’Lenic et al, 2008)

SPRING SUMMER FALL WINTER

FMA, MAM, AMJ MJJ, JJA, JAS ASO, SON, OND NDJ, DJF, JFM

HSS

HSS

OFF

HSS

HSS

OFF

OFF

OFF

HSS

CON

HSS

CON

CON

HSS

HSS

CON

DIF

DIF

DIF

DIF

DIF

+20%

+18%

+8%

+16%

slide22

Top 2 rows: 1995-2004 HSS (lines) of 3-month T Outlooks, Official (OFF) and Consolidation (CON). Colors are the fraction of the time non-EC is predicted (%).Bottom row: Difference, CON-OFF (lines and colors).

SPRING SUMMER FALL WINTER

FMA, MAM, AMJ MJJ, JJA, JAS ASO, SON, OND NDJ, DJF, JFM

HSS

HSS

HSS

OFF

OFF

OFF

HSS

OFF

HSS

HSS

HSS

HSS

CON

CON

CON

CON

DIF

DIF

DIF

DIF

DIF

+31%

+55%

+11%

+40%

what is the forecast evaluation tool fet
What is the Forecast Evaluation Tool (FET)?
  • A web tool that allows users to interact with a database of CPC 3-Month Mean Temperature, Total Precipitation Outlooks, and verifying observations from1995-present,
  • A tracker of user preferences.
  • A self-teaching tool to enable a wide range of users to learn what CPC forecasts are, what they mean, and what their implications are for user applications.
  • A collector of user requirements.
  • A community tool for CPC, IRI, IPCC, …???
  • A laboratory for growing services to users
ctb user centric forecast tools progress
CTB User-Centric Forecast Tools Progress
  • Simple Object Access Protocol (SOAP) Tested
  • Secured Go-Ahead to Place FET on NWS Web Operations Center (WOC)
  • Trained CPC Staff in JAVA language
  • Scheduled Ellen Lay Training session in Nov.
future of the fet
FUTURE of the FET

Next 1-4 months:

  • Finalize and implement FET project plan at CPC.
  • Ellen Lay (CLIMAS) to train CPC personnel on FET version control and bug tracking at CPC, November 18-21, 2008.
  • Necessary software (APACHE TOMCAT, JAVA, Desktop View) acquired and installed at CPC.
  • Forecast, observations datasets in-place at CPC.
  • FET code ported to CPC, installed, tested.
  • FET installed to NWS Web Operations Center (WOC) servers
future of the fet1
FUTURE of the FET +

In partnership with CLIMAS and community we will add:

  • Other forecasts and organizations
  • Time and space aggregation options
  • Significance tests/cautions to users
  • Requirements requests option
  • Questions option

The stakes are high…..

slide46

Source: The Washington Post

Outlook Section, July 13, 2008

summary
Summary
  • Users want partnership, accuracy, specificity, flexibility
  • “Relationship” is synonymous with “partnership”
  • TRUST (honest brokerage) is central to these requirements.
  • Producers must learn WHO users are, HOW they use products and WHAT their evolving requirements are
  • Need to involve users and producers in iteratively optimizing products
  • A continuous flow of requirements from users toward research may avert VOD.
  • Means to fund an ever-expanding, perpetual product suite needed
  • Stakes are high: 6 of top 20 news/media June 2008 sites were weather-related. 10s-100s of B$ at stake. Climate will only add to this.