prediction markets and the wisdom of crowds l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Prediction Markets and the Wisdom of Crowds PowerPoint Presentation
Download Presentation
Prediction Markets and the Wisdom of Crowds

Loading in 2 Seconds...

play fullscreen
1 / 46

Prediction Markets and the Wisdom of Crowds - PowerPoint PPT Presentation


  • 404 Views
  • Uploaded on

Prediction Markets and the Wisdom of Crowds. David Pennock Yahoo! Research NYC. Outline. Prediction Markets Survey What is a prediction market? Examples Some research findings The Wisdom of Crowds: A Story. Bet = Credible Opinion. Hillary Clinton will win the election.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Prediction Markets and the Wisdom of Crowds' - Roberta


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
prediction markets and the wisdom of crowds

Prediction Markets and the Wisdom of Crowds

David Pennock

Yahoo! Research NYC

outline
Outline
  • Prediction Markets Survey
    • What is a prediction market?
    • Examples
    • Some research findings
  • The Wisdom of Crowds:A Story
bet credible opinion
Bet = Credible Opinion

Hillary Clinton will win the election

  • Which is more believable?More Informative?
  • Betting intermediaries
    • Las Vegas, Wall Street, Betfair, Intrade,...
    • Prices: stable consensus of a large number of quantitative, credible opinions
    • Excellent empirical track record

“I bet $100 Hillary will win at 1 to 2 odds”

more socially redeemable example
More Socially Redeemable Example

http://intrade.com

Screen capture 2007/05/18

a prediction market
A Prediction Market
  • Take a random variable, e.g.
  • Turn it into a financial instrument payoff = realized value of variable

Bird Flu Outbreak US 2007?(Y/N)

I am entitled to:

Bird FluUS ’07

Bird FluUS ’07

$1 if

$0 if

slide7
Why?

Get information

  • price  probability of uncertain event(in theory, in the lab, in the field, ...more later)
  • Is there some future event you’d like to forecast?A prediction market can probably help
a prediction market8

 6

= 6 ?

= 6

I am entitled to:

$1 if

$0 if

A Prediction Market
  • Take a random variable, e.g.
  • Turn it into a financial instrument payoff = realized value of variable

2008 CAEarthquake?

US’08Pres =Dem?

aside terminology
Aside: Terminology
  • Key aspect: payout is uncertain
  • Called variously: asset, security, contingent claim, derivative (future, option), stock, prediction market, information market, gamble, bet, wager, lottery
  • Historically mixed reputation
    • Esp. gambling aspect
    • A time when options were frowned upon
  • But when regulated serve important social roles...
getting information

 6

= 6

I am entitled to:

$1 if

$0 if

Getting Information
  • Non-market approach: ask an expert
    • How much would you pay for this?
  • A: $5/36  $0.1389
    • caveat: expert is knowledgeable
    • caveat: expert is truthful
    • caveat: expert is risk neutral, or ~ RN for $1
    • caveat: expert has no significant outside stakes
getting information11

 6

= 6

= 6

Getting Information
  • Non-market approach:pay an expert
    • Ask the expert for his report r of the probabilityP( )
    • Offer to pay the expert
      • $100 + log r if
      • $100 + log (1-r) if
  • It so happens that the expert maximizes expected profit by reporting r truthfully
    • caveat: expert is knowledgeable
    • caveat: expert is truthful
    • caveat: expert is risk neutral, or ~ RN
    • caveat: expert has no significant outside stakes

“logarithmic scoring rule”,

a “proper” scoring rule

getting information12

I am entitled to:

 6

= 6

= 6

$1 if

$0 if

Getting Information
  • Market approach: “ask” the public—experts & non-experts alike—by opening a market:
  • Let any person i submit a bid order: an offer to buy qi units at price pi
  • Let any person j submit an ask order: an offer to sell qj units at price pj(if you sell 1 unit, you agree to pay $1 if )
  • Match up agreeable trades (many poss. mechs...)
non market alternatives vs markets
Opinion poll

Sampling

No incentive to be truthful

Equally weighted information

Hard to be real-time

Ask Experts

Identifying experts can be hard

Incentives

Combining opinions can be difficult

Prediction Markets

Self-selection

Monetary incentive and more

Money-weighted information

Real-time

Self-organizing

Non-Market Alternatives vs. Markets
non market alternatives vs markets14
Machine learning/Statistics

Historical data

Past and future are related

Hard to incorporate recent new information

Prediction Markets

No need for data

No assumption on past and future

Immediately incorporate new information

Non-Market Alternatives vs. Markets
function of markets 2 risk management

$1 if $0 otherwise

Function of Markets 2: Risk Management
  • If is bad for me,

I buy a bunch of

  • If my house is struck by lightening, I am compensated.
risk management examples
Risk Management Examples
  • Insurance
    • I buy car insurance to hedge the risk of accident
  • Futures
    • Farmers sell soybean futures to hedge the risk of price drop
  • Options
    • Investors buy options to hedge the risk of stock price changes
giving getting information
What you can say/learn% chance that

Hillary wins

GOP wins Texas

YHOO stock > 30

Duke wins tourney

Oil prices fall

Heat index rises

Hurricane hits Florida

Rains at place/time

Where

IEM, Intrade.com

Intrade.com

Stock options market

Las Vegas, Betfair

Futures market

Weather derivatives

Insurance company

Weatherbill.com

Giving/Getting Information
slide19

http://intrade.com

http://tradesports.com

Screen capture 2007/05/18

slide22

http://www.wsex.com/

http://www.hedgestreet.com/

Screen capture 2007/05/18

Screen capture 2007/05/18

play money real predictions
Play money;Real predictions

http://www.hsx.com/

slide24

http://www.ideosphere.com

http://us.newsfutures.com/

Cancercuredby 2010

Machine Gochampionby 2020

yahoo o reilly tech buzz game
Yahoo!/O’Reilly Tech Buzz Game

http://buzz.research.yahoo.com/

more prediction market games
More Prediction Market Games
  • BizPredict.com
  • CasualObserver.net
  • FTPredict.com
  • InklingMarkets.com
  • ProTrade.com
  • StorageMarkets.com
  • TheSimExchange.com
  • TheWSX.com
  • Alexadex, Celebdaq, Cenimar, BetBubble, Betocracy, CrowdIQ, MediaMammon,Owise, PublicGyan, RIMDEX, Smarkets, Trendio, TwoCrowds
  • http://www.chrisfmasse.com/3/3/markets/#Play-Money_Prediction_Markets
catalysts
Catalysts
  • Markets have long history of predictive accuracy: why catching on now as tool?
  • No press is bad press: Policy Analysis Market (“terror futures”)
  • Surowiecki's “Wisdom of Crowds”
  • Companies:
    • Google, Microsoft, Yahoo!; CrowdIQ, HSX, InklingMarkets, NewsFutures
  • Press: BusinessWeek, CBS News, Economist, NYTimes, Time, WSJ, ...http://us.newsfutures.com/home/articles.html
does it work
Does it work?
  • Yes, evidence from real markets, laboratory experiments, and theory
    • Racetrack odds beat track experts [Figlewski 1979]
    • Orange Juice futures improve weather forecast [Roll 1984]
    • I.E.M. beat political polls 451/596 [Forsythe 1992, 1999][Oliven 1995][Rietz 1998][Berg 2001][Pennock 2002]
    • HP market beat sales forecast 6/8 [Plott 2000]
    • Sports betting markets provide accurate forecasts of game outcomes [Gandar 1998][Thaler 1988][Debnath EC’03][Schmidt 2002]
    • Market games work [Servan-Schreiber 2004][Pennock 2001]
    • Laboratory experiments confirm information aggregation[Plott 1982;1988;1997][Forsythe 1990][Chen, EC’01]
    • Theory: “rational expectations” [Grossman 1981][Lucas 1972]
    • More later …
example iem32
Example: IEM

[Source: Berg, DARPA Workshop, 2002]

MP1-35

example iem33
Example: IEM

[Source: Berg, DARPA Workshop, 2002]

MP1-36

speed tradesports
Speed: TradeSports

[Source: Wolfers 2004]

Contract: Pays $100 if Cubs win game 6 (NLCS)

Price of contract

(=Probability that Cubs win)

Fan reaches over and spoils Alou’s catch. Still 1 out.

Cubs are winning 3-0 top of the 8th1 out.

The Marlins proceed to hit 8 runs in the 8th inning

Time (in Ireland)

MP1-37

does money matter play vs real head to head
Experiment

2003 NFL Season

ProbabilitySports.com Online football forecasting competition

Contestants assess probabilities for each game

Quadratic scoring rule

~2,000 “experts”, plus:

NewsFutures (play $)

Tradesports (real $)

Used “last trade” prices

Results:

Play money and real money performed similarly

6th and 8th respectively

Markets beat most of the ~2,000 contestants

Average of experts came 39th (caveat)

Does money matter? Play vs real, head to head

Electronic Markets, Emile Servan-Schreiber, Justin Wolfers, David Pennock and Brian Galebach

does money matter play vs real head to head37
Does money matter? Play vs real, head to head

Statistically:TS ~ NFNF >> Avg

TS > Avg

real markets vs market games
Real marketsvs. market games

IEM

HSX

averagelog

score

arbitrageclosure

MP1-41

real markets vs market games39
Real marketsvs. market games

probabilisticforecasts

HSX

FX, F1P6

forecast source avg log score

F1P6 linear scoring -1.84

F1P6 F1-style scoring -1.82

betting odds -1.86

F1P6 flat scoring -2.03

F1P6 winner scoring -2.32

expectedvalueforecasts489 movies

MP1-42

a wisdom of crowds story

1/7

Story

Survey

Research

A Wisdom of Crowds Story

Opinion

  • ProbabilitySports.com
  • Thousands of probability judgments for sporting events
    • Alice: Jets 67% chance to beat Patriots
    • Bob: Jets 48% chance to beat Patriots
    • Carol, Don, Ellen, Frank, ...
  • Reward: Quadratic scoring rule:Best probability judgments maximize expected score
individuals
Individuals
  • Most individuals are poor predictors
  • 2005 NFL Season
    • Best: 3747 points
    • Average: -944 Median: -275
    • 1,298 out of 2,231 scored below zero(takes work!)
individuals42
Poorly calibrated (too extreme)

Teams given < 20% chance actually won 30% of the time

Teams given > 80% chance actually won 60% of the time

Individuals
the crowd
The Crowd
  • Create a crowd predictor by simply averaging everyone’s probabilities
    • Crowd = 1/n(Alice + Bob + Carol + ... )
    • 2005: Crowd scored 3371 points(7th out of 2231) !
  • Wisdom of fools: Create a predictor by averaging everyone who scored below zero
    • 2717 points (62nd place) !
    • (the best “fool” finished in 934th place)
the crowd how big
The Crowd: How Big?

More:http://blog.oddhead.com/2007/01/04/the-wisdom-of-the-probabilitysports-crowd/http://www.overcomingbias.com/2007/02/how_and_when_to.html

can we do better ml stats
Can We Do Better?: ML/Stats

[Dani et al. UAI 2006]

  • Maybe Not
    • CS “experts algorithms”
    • Other expert weights
    • Calibrated experts
    • Other averaging fn’s (geo mean, RMS, power means, mean of odds, ...)
    • Machine learning (NB, SVM, LR, DT, ...)
  • Maybe So
    • Bayesian modeling + EM
    • Nearest neighbor (multi-year)