Prediction markets and the wisdom of crowds
Download
1 / 46

Prediction Markets and the Wisdom of Crowds - PowerPoint PPT Presentation


  • 397 Views
  • Updated On :

Prediction Markets and the Wisdom of Crowds. David Pennock Yahoo! Research NYC. Outline. Prediction Markets Survey What is a prediction market? Examples Some research findings The Wisdom of Crowds: A Story. Bet = Credible Opinion. Hillary Clinton will win the election.

Related searches for Prediction Markets and the Wisdom of Crowds

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Prediction Markets and the Wisdom of Crowds' - Roberta


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Prediction markets and the wisdom of crowds l.jpg

Prediction Markets and the Wisdom of Crowds

David Pennock

Yahoo! Research NYC


Outline l.jpg
Outline

  • Prediction Markets Survey

    • What is a prediction market?

    • Examples

    • Some research findings

  • The Wisdom of Crowds:A Story


Bet credible opinion l.jpg
Bet = Credible Opinion

Hillary Clinton will win the election

  • Which is more believable?More Informative?

  • Betting intermediaries

    • Las Vegas, Wall Street, Betfair, Intrade,...

    • Prices: stable consensus of a large number of quantitative, credible opinions

    • Excellent empirical track record

“I bet $100 Hillary will win at 1 to 2 odds”



More socially redeemable example l.jpg
More Socially Redeemable Example

http://intrade.com

Screen capture 2007/05/18


A prediction market l.jpg
A Prediction Market

  • Take a random variable, e.g.

  • Turn it into a financial instrument payoff = realized value of variable

Bird Flu Outbreak US 2007?(Y/N)

I am entitled to:

Bird FluUS ’07

Bird FluUS ’07

$1 if

$0 if


Slide7 l.jpg
Why?

Get information

  • price  probability of uncertain event(in theory, in the lab, in the field, ...more later)

  • Is there some future event you’d like to forecast?A prediction market can probably help


A prediction market8 l.jpg

6

= 6 ?

= 6

I am entitled to:

$1 if

$0 if

A Prediction Market

  • Take a random variable, e.g.

  • Turn it into a financial instrument payoff = realized value of variable

2008 CAEarthquake?

US’08Pres =Dem?


Aside terminology l.jpg
Aside: Terminology

  • Key aspect: payout is uncertain

  • Called variously: asset, security, contingent claim, derivative (future, option), stock, prediction market, information market, gamble, bet, wager, lottery

  • Historically mixed reputation

    • Esp. gambling aspect

    • A time when options were frowned upon

  • But when regulated serve important social roles...


Getting information l.jpg

6

= 6

I am entitled to:

$1 if

$0 if

Getting Information

  • Non-market approach: ask an expert

    • How much would you pay for this?

  • A: $5/36  $0.1389

    • caveat: expert is knowledgeable

    • caveat: expert is truthful

    • caveat: expert is risk neutral, or ~ RN for $1

    • caveat: expert has no significant outside stakes


Getting information11 l.jpg

6

= 6

= 6

Getting Information

  • Non-market approach:pay an expert

    • Ask the expert for his report r of the probabilityP( )

    • Offer to pay the expert

      • $100 + log r if

      • $100 + log (1-r) if

  • It so happens that the expert maximizes expected profit by reporting r truthfully

    • caveat: expert is knowledgeable

    • caveat: expert is truthful

    • caveat: expert is risk neutral, or ~ RN

    • caveat: expert has no significant outside stakes

“logarithmic scoring rule”,

a “proper” scoring rule


Getting information12 l.jpg

I am entitled to:

 6

= 6

= 6

$1 if

$0 if

Getting Information

  • Market approach: “ask” the public—experts & non-experts alike—by opening a market:

  • Let any person i submit a bid order: an offer to buy qi units at price pi

  • Let any person j submit an ask order: an offer to sell qj units at price pj(if you sell 1 unit, you agree to pay $1 if )

  • Match up agreeable trades (many poss. mechs...)


Non market alternatives vs markets l.jpg

Opinion poll

Sampling

No incentive to be truthful

Equally weighted information

Hard to be real-time

Ask Experts

Identifying experts can be hard

Incentives

Combining opinions can be difficult

Prediction Markets

Self-selection

Monetary incentive and more

Money-weighted information

Real-time

Self-organizing

Non-Market Alternatives vs. Markets


Non market alternatives vs markets14 l.jpg

Machine learning/Statistics

Historical data

Past and future are related

Hard to incorporate recent new information

Prediction Markets

No need for data

No assumption on past and future

Immediately incorporate new information

Non-Market Alternatives vs. Markets


Function of markets 2 risk management l.jpg

$1 if $0 otherwise

Function of Markets 2: Risk Management

  • If is bad for me,

    I buy a bunch of

  • If my house is struck by lightening, I am compensated.


Risk management examples l.jpg
Risk Management Examples

  • Insurance

    • I buy car insurance to hedge the risk of accident

  • Futures

    • Farmers sell soybean futures to hedge the risk of price drop

  • Options

    • Investors buy options to hedge the risk of stock price changes



Giving getting information l.jpg

What you can say/learn% chance that

Hillary wins

GOP wins Texas

YHOO stock > 30

Duke wins tourney

Oil prices fall

Heat index rises

Hurricane hits Florida

Rains at place/time

Where

IEM, Intrade.com

Intrade.com

Stock options market

Las Vegas, Betfair

Futures market

Weather derivatives

Insurance company

Weatherbill.com

Giving/Getting Information


Slide19 l.jpg

http://intrade.com

http://tradesports.com

Screen capture 2007/05/18




Slide22 l.jpg

http://www.wsex.com/

http://www.hedgestreet.com/

Screen capture 2007/05/18

Screen capture 2007/05/18


Play money real predictions l.jpg
Play money;Real predictions

http://www.hsx.com/


Slide24 l.jpg

http://www.ideosphere.com

http://us.newsfutures.com/

Cancercuredby 2010

Machine Gochampionby 2020


Yahoo o reilly tech buzz game l.jpg
Yahoo!/O’Reilly Tech Buzz Game

http://buzz.research.yahoo.com/


More prediction market games l.jpg
More Prediction Market Games

  • BizPredict.com

  • CasualObserver.net

  • FTPredict.com

  • InklingMarkets.com

  • ProTrade.com

  • StorageMarkets.com

  • TheSimExchange.com

  • TheWSX.com

  • Alexadex, Celebdaq, Cenimar, BetBubble, Betocracy, CrowdIQ, MediaMammon,Owise, PublicGyan, RIMDEX, Smarkets, Trendio, TwoCrowds

  • http://www.chrisfmasse.com/3/3/markets/#Play-Money_Prediction_Markets


Catalysts l.jpg
Catalysts

  • Markets have long history of predictive accuracy: why catching on now as tool?

  • No press is bad press: Policy Analysis Market (“terror futures”)

  • Surowiecki's “Wisdom of Crowds”

  • Companies:

    • Google, Microsoft, Yahoo!; CrowdIQ, HSX, InklingMarkets, NewsFutures

  • Press: BusinessWeek, CBS News, Economist, NYTimes, Time, WSJ, ...http://us.newsfutures.com/home/articles.html


Does it work l.jpg
Does it work?

  • Yes, evidence from real markets, laboratory experiments, and theory

    • Racetrack odds beat track experts [Figlewski 1979]

    • Orange Juice futures improve weather forecast [Roll 1984]

    • I.E.M. beat political polls 451/596 [Forsythe 1992, 1999][Oliven 1995][Rietz 1998][Berg 2001][Pennock 2002]

    • HP market beat sales forecast 6/8 [Plott 2000]

    • Sports betting markets provide accurate forecasts of game outcomes [Gandar 1998][Thaler 1988][Debnath EC’03][Schmidt 2002]

    • Market games work [Servan-Schreiber 2004][Pennock 2001]

    • Laboratory experiments confirm information aggregation[Plott 1982;1988;1997][Forsythe 1990][Chen, EC’01]

    • Theory: “rational expectations” [Grossman 1981][Lucas 1972]

    • More later …





Example iem32 l.jpg
Example: IEM

[Source: Berg, DARPA Workshop, 2002]

MP1-35


Example iem33 l.jpg
Example: IEM

[Source: Berg, DARPA Workshop, 2002]

MP1-36


Speed tradesports l.jpg
Speed: TradeSports

[Source: Wolfers 2004]

Contract: Pays $100 if Cubs win game 6 (NLCS)

Price of contract

(=Probability that Cubs win)

Fan reaches over and spoils Alou’s catch. Still 1 out.

Cubs are winning 3-0 top of the 8th1 out.

The Marlins proceed to hit 8 runs in the 8th inning

Time (in Ireland)

MP1-37


Does money matter play vs real head to head l.jpg

Experiment

2003 NFL Season

ProbabilitySports.com Online football forecasting competition

Contestants assess probabilities for each game

Quadratic scoring rule

~2,000 “experts”, plus:

NewsFutures (play $)

Tradesports (real $)

Used “last trade” prices

Results:

Play money and real money performed similarly

6th and 8th respectively

Markets beat most of the ~2,000 contestants

Average of experts came 39th (caveat)

Does money matter? Play vs real, head to head

Electronic Markets, Emile Servan-Schreiber, Justin Wolfers, David Pennock and Brian Galebach


Does money matter play vs real head to head37 l.jpg
Does money matter? Play vs real, head to head

Statistically:TS ~ NFNF >> Avg

TS > Avg


Real markets vs market games l.jpg
Real marketsvs. market games

IEM

HSX

averagelog

score

arbitrageclosure

MP1-41


Real markets vs market games39 l.jpg
Real marketsvs. market games

probabilisticforecasts

HSX

FX, F1P6

forecast source avg log score

F1P6 linear scoring -1.84

F1P6 F1-style scoring -1.82

betting odds -1.86

F1P6 flat scoring -2.03

F1P6 winner scoring -2.32

expectedvalueforecasts489 movies

MP1-42


A wisdom of crowds story l.jpg

1/7

Story

Survey

Research

A Wisdom of Crowds Story

Opinion

  • ProbabilitySports.com

  • Thousands of probability judgments for sporting events

    • Alice: Jets 67% chance to beat Patriots

    • Bob: Jets 48% chance to beat Patriots

    • Carol, Don, Ellen, Frank, ...

  • Reward: Quadratic scoring rule:Best probability judgments maximize expected score


Individuals l.jpg
Individuals

  • Most individuals are poor predictors

  • 2005 NFL Season

    • Best: 3747 points

    • Average: -944 Median: -275

    • 1,298 out of 2,231 scored below zero(takes work!)


Individuals42 l.jpg

Poorly calibrated (too extreme)

Teams given < 20% chance actually won 30% of the time

Teams given > 80% chance actually won 60% of the time

Individuals


The crowd l.jpg
The Crowd

  • Create a crowd predictor by simply averaging everyone’s probabilities

    • Crowd = 1/n(Alice + Bob + Carol + ... )

    • 2005: Crowd scored 3371 points(7th out of 2231) !

  • Wisdom of fools: Create a predictor by averaging everyone who scored below zero

    • 2717 points (62nd place) !

    • (the best “fool” finished in 934th place)


The crowd how big l.jpg
The Crowd: How Big?

More:http://blog.oddhead.com/2007/01/04/the-wisdom-of-the-probabilitysports-crowd/http://www.overcomingbias.com/2007/02/how_and_when_to.html


Can we do better ml stats l.jpg
Can We Do Better?: ML/Stats

[Dani et al. UAI 2006]

  • Maybe Not

    • CS “experts algorithms”

    • Other expert weights

    • Calibrated experts

    • Other averaging fn’s (geo mean, RMS, power means, mean of odds, ...)

    • Machine learning (NB, SVM, LR, DT, ...)

  • Maybe So

    • Bayesian modeling + EM

    • Nearest neighbor (multi-year)