Forecasters view of climate change methodology also counts
Download
1 / 33

Forecasters - PowerPoint PPT Presentation


  • 364 Views
  • Uploaded on

Polar bear population has grown in recent decades due to restrictions on hunting. ... Polar Bear Population Forecasts: A Public-Policy Forecasting Audit. ...

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Forecasters' - Kelvin_Ajay


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Forecasters view of climate change methodology also counts l.jpg

Forecasters’ view ofclimate-change:Methodology also counts

J. Scott Armstrong

The Wharton School

University of Pennsylvania

Kesten C. Green

Business and Economic Forecasting Unit

Monash University

March 9, 2009

File: Heartland 09-JSA v36


Role of methodology l.jpg
Role of methodology

Inputs

Data

Problems with the data

Lack of disclosure

Changes in measurement

Relationships

Which variables

Direction and effects of variables

Forecasting methodology

How to integrate inputs to produce forecasts and prediction intervals for policy analysis

2


Climate change forecasting problem l.jpg
Climate change forecasting problem

  • Structuring the forecasting problem (JSA)

  • Auditing the forecasting process (JSA)

  • Selecting a proper forecasting method (JSA + KCG)

  • Testing forecast validity (KCG)

3


The forecasting problems l.jpg

The forecasting problems

Rational public policy requires scientific forecasts of the direction and magnitude—and confidence intervals—for:

1. Climate change

2. Effects of climate changes (costs & benefits)

3. Effects of alternative policies (ditto)

We have been unable to find any scientific forecasts for #1, climate change: warming or cooling. (Green, Armstrong & Soon 2009)

Our search for #2 & #3 continues, also without success to date.


Handouts for armstrong green talks l.jpg

Handouts for Armstrong & Green talks

References for papers cited in our talks /

WWF solicitation based on false advertising?

Boxer vs.Armstrong Wager

Gore vs. Armstrong Prediction Market.

Suggestions on what to do?

• Analogies study

• WWF action steps

• CO2 policy forecasting

“Climate Change Forecasts are Useless for Policy Making”


Slide6 l.jpg

Forecasting methodology

  • Structuring the forecasting problem

  • Auditing the forecasting process

  • Selecting a forecasting method

  • Validation tests

6


Auditing ipcc forecasts g l.jpg
Auditing IPCC forecasts g

IPCC “projections” of global temperature change used improper procedures. Green & Armstrong audit* showed:

  • IPCC authors violated 72 forecasting principles.

    2. Forecasts by scientists, not scientific forecasts.

    3. No proper evidence on predictive validity

    * Green, K. C. & J. S. Armstrong (2007)

7


Audit of polar bear pb population forecasting l.jpg

Audit of Polar bear (PB)population forecasting

Polar bear population has grown in recent decades due to restrictions on hunting.

Government forecast global warming assumed to reduce sea ice.

Analysis of 5 years of data (with “outlier” removed) related PB population to ice.

PB experts used “a” and “b” to forecast rapid decline in PB population over 50 years.

Government assumed that an “endangered” classification would boost the PB population. No forecasts for this, or for alternative policies.


Polar bear audit results l.jpg

Only 12% of the principles were properly applied*.

Government researchers declined to provide peer review or to answer questions we had about their paper.

Offered to act as reviewers for the journal.

Government researchers declined invitation to review or to provide commentary.

* Armstrong, Green, & Soon (2008)

Polar bear audit results


Example of a principle full disclosure l.jpg

Example of a principle: Full disclosure

Government researchers used complex mathematics to forecast PB populations.

We requested the data they used on the PB population.


Slide11 l.jpg

Here are the data provided to us in our requests for our audit of the U.S. Fish and Wildlife Services forecasts:


Current polar bear situation l.jpg

Current polar bear situation audit of the U.S. Fish and Wildlife Services forecasts:

Lacking scientific forecasts*, the U.S. Fish and Wildlife Services classified the PB as “threatened.”

* Armstrong, Green & Soon (2008)


Announcing the boxer armstrong charity wager l.jpg

Announcing the Boxer-Armstrong charity wager audit of the U.S. Fish and Wildlife Services forecasts:

Senator Barbara Boxer supports government forecasts that the polar bear population will decrease by about 23% over the next ten years.

With an upward trend over recent decades and high uncertainty, Armstrong, Green & Soon forecast no decline in the PB population.

Professor Armstrong challenges Senator Boxer to a $5,000 charity wager on the polar bear population for the next ten years.

An independent two-person panel would determine which forecast was closest to the actual population at the end of 2018.


Senator boxer s response as of march 8 l.jpg

Senator Boxer’s response audit of the U.S. Fish and Wildlife Services forecasts:as of March 8


Policy based on gw forecasts world wildlife fund l.jpg

Policy based on GW forecasts: audit of the U.S. Fish and Wildlife Services forecasts:World Wildlife Fund

WWF TV commercials: In an appeal for members and donations, the ads claim that due to global warming, the number of polar bears is decreasing rapidly, thereby putting the species at risk of extinction.

WWF website says: “The general status of polar bears is currently stable, though there are differences between the populations. Some are stable, some seem to be increasing, and some are decreasing due to various pressures. The status of several populations is not well documented.”


False advertising by the wwf l.jpg

False advertising by the WWF? audit of the U.S. Fish and Wildlife Services forecasts:

Since Dec. 12, 2008, I have made attempts to contact the CEO and various Trustees of the WWF to raise this issue of what appears to be false advertising

I also offered to report on their position at this conference.


Wwf response as of march 8 l.jpg
WWF response as of March 8 audit of the U.S. Fish and Wildlife Services forecasts:

17


Prevalence of false advertising l.jpg
Prevalence of false advertising audit of the U.S. Fish and Wildlife Services forecasts:

Are many organizations engaged in false advertising with respect to global warming?

If so, what might be done to eliminate such false advertising?


Wwf action steps l.jpg

WWF action steps? audit of the U.S. Fish and Wildlife Services forecasts:

Please put your suggestions on the sheets that we have passed out.

I will summarize these on theclimatebet.com


Slide20 l.jpg

Forecasting methodology audit of the U.S. Fish and Wildlife Services forecasts:

  • Structuring the forecasting problem

  • Auditing the forecasting process

  • Selecting a forecasting method

  • Validation tests

20


Selecting a proper forecasting method l.jpg
Selecting a proper audit of the U.S. Fish and Wildlife Services forecasts:forecasting method

Evidence-based forecasting methods*

  • Expert judgment

  • Extrapolation

  • Causal models

    * Forecasting methods that have been experimentally tested for efficacy in given situations

21


Expert judgment l.jpg
Expert judgment audit of the U.S. Fish and Wildlife Services forecasts:

Unaided expert judgment not useful given high complexity and high uncertainty. (Tetlock 2005)

Applies with respect to

  • Accuracy

  • Experts’ statements about confidence

  • Agreement among experts

22


Evidence based expert judgment methods use structure l.jpg

Evidence-based expert judgment methods (use structure) audit of the U.S. Fish and Wildlife Services forecasts:

Delphi

Prediction markets

Structured analogies

Causal models


Delphi l.jpg
Delphi audit of the U.S. Fish and Wildlife Services forecasts:

Delphi: A multi-round survey of

unbiased experts who make anonymous forecast with

feedback on predictions and

reasons given between rounds.

Assumes that experts can make useful forecasts.

It also helps to assess expert opinions in an objective manner and provides decision makers with reasons.

24


Prediction markets l.jpg
Prediction markets audit of the U.S. Fish and Wildlife Services forecasts:

  • We are currently examining the use of prediction markets

    1) assess expert opinion

    2) forecast climate change

  • Potential problems:

    a. If individuals can’t forecast, does combining their forecasts help?

    b. Is a market for long-term bets feasible

    c. Would a series of short-term bets be useful?

    d. Is play money adequate?

    e. Might people manipulate the market?

25


Prediction markets can also summarize prevailing opinions in an objective manner l.jpg
Prediction markets can also summarize prevailing opinions in an objective manner

When will “Manmade Global Climate Change” next be disputed by a major media outlet? (HubDub.com)

26


Gore vs armstrong conducted via a prediction market l.jpg

Gore vs. Armstrong conducted an objective mannervia a prediction market

To stimulate work on validation of alternative methods to forecast climate change.

I have informed Mr. Gore of this prediction market and asked for his comments and suggestions.


Gore s response as of march 8 l.jpg
Gore’s response as of March 8 an objective manner

28


Progress on prediction market l.jpg
Progress on prediction market an objective manner

Assume Armstrong and Gore made a ten-year bet starting January 1, 2008.

Armstrong forecast “no change” in global mean temperature.

Gore implied that global mean temperatures would increase at least as rapidly as the IPCC’s 1992 projection of 0.03oC per year

Criterion: mean absolute errors of annual forecasts for the ten-year period using UAH global mean temperature record.

29


Current betting on a play money market hub dub l.jpg

Current betting on a an objective mannerplay-money market (hub-dub)

Armstrong by 62% as of March 4, 2009 (100 bets)

http://tinyurl.com/gore-armstrong-bet


Plans on prediction markets l.jpg
Plans on prediction markets an objective manner

  • Alternative criteria (e.g., Hadley temperatures)

  • Real money sites

  • Add information via comments

  • Vary time horizon (e.g., one-year ahead?)

  • Compare with Delphi expert panel

  • Graefe, A. J., J. S. Armstrong, K.C. Green (2009)

31


Handouts for armstrong green talks32 l.jpg

Handouts for Armstrong & Green talks an objective manner

References for papers cited in our talks /

WWF solicitation based on false advertising?

Boxer vs.Armstrong Wager

Gore vs. Armstrong Prediction Market.

Suggestions on what to do?

• Analogies study

• WWF action steps

• CO2 policy forecasting

“Climate Change Forecasts are Useless for Policy Making”


Slide33 l.jpg

References for the Armstrong and Green talks an objective manner

[Papers available at http://publicpolicyforecasting.com unless otherwise indicated]

Armstrong, J. S., Green, K. C., & Soon, W. (2008). Polar Bear Population Forecasts: A Public-Policy Forecasting Audit. Interfaces, 38, 5, 382–405. (includes commentary & response).

Graefe, A., Armstrong, J. S., & Green, K. C. (2009). Using Prediction Markets to Solve Complex Problems: An Application to the ‘Climate Bet’. Working paper available at http://kestencgreen.com/cbpm.pdf

Green, K.C., & Armstrong, J. S. (2007). Global Warming: Forecasts by Scientists versus Scientific Forecasts. Energy and Environment, 18, No. 7+8, 995-1019.

Green, K.C., & Armstrong, J. S. (2007). Structured Analogies in Forecasting. International Journal of Forecasting, 23, 365-376.

Green, K.C., Armstrong, J. S., & Soon, W. (2009). Validity of Climate Change Forecasting for Public Policy Decision Making. International Journal of Forecasting, Forthcoming.

Tetlock, P. E. (2005). Expert Political Judgment: How Good Is It? How Can We Know? Princeton University Press, Princeton, NJ.

Gore-Armstrong prediction market at http://tinyurl.com/gore-armstrong-bet

Details on Gore-Armstrong bet at http://theclimatebet.com