wisdom of crowds in human memory reconstructing events by aggregating memories across individuals n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals PowerPoint Presentation
Download Presentation
Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals

Loading in 2 Seconds...

play fullscreen
1 / 74

Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals - PowerPoint PPT Presentation


  • 93 Views
  • Uploaded on

Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals. Mark Steyvers Department of Cognitive Sciences University of California, Irvine. Joint work with: Brent Miller, Pernille Hemmer, Mike Yi Michael Lee, Bill Batchelder , Paolo Napoletano.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals' - dyre


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
wisdom of crowds in human memory reconstructing events by aggregating memories across individuals

Wisdom of Crowds in Human Memory: Reconstructing Events by Aggregating Memories across Individuals

Mark Steyvers

Department of Cognitive Sciences

University of California, Irvine

Joint work with:

Brent Miller, Pernille Hemmer, Mike Yi

Michael Lee, Bill Batchelder, Paolo Napoletano

wisdom of crowds phenomenon
Wisdom of crowds phenomenon
  • Group estimate often performs as well as or better than best individual in the group
examples of wisdom of crowds phenomenon
Examples of wisdom of crowds phenomenon

Galton’s Ox (1907): Median of individual estimates comes close to true answer

Who wants to be a millionaire?

recollection of 9 11 event sequence altmann 2003
Recollection of 9/11 Event Sequence (Altmann, 2003)

Most frequent response (i.e, mode)

A = One plane hits the WTC

B = A second plane hits the WTC

C = One plane crashes into the Pentagon

D = One tower at the WTC collapses

E = One plane crashes in Pennsylvania

F = A second tower at the WTC collapses

Correct

research g oal a ggregating responses
Research goal: aggregating responses

ground truth

group answer

?

A B C D

=

A B C D

Aggregation Algorithm

A D B C

D A B C

B A D C

A C B D

A B D C

task constraints
Task constraints
  • No communication between individuals
  • There is always a true answer (ground truth)
  • Aggregation algorithm never has access to ground truth
    • unsupervised methods
    • ground truth only used for evaluation
i s this research part of psychology
Is this research part of psychology?
  • Yes
  • Effective aggregation of human judgments requires cognitive models
overview of talk
Overview of talk
  • Ordering problems
    • what is the order of US presidents?
  • Matching problems
    • memory for pairs: what object was paired with what person?
  • Recognition memory problems
    • what words were studied?
  • Experts in crowds
    • how to find experts in the absence of feedback
recollecting order from declarative memory
Recollecting Order from Declarative Memory

Abraham Lincoln

Ulysses S. Grant

time

Ulysses S. Grant

Rutherford B. Hayes

Rutherford B. Hayes

James Garfield

Abraham Lincoln

Andrew Johnson

James Garfield

Andrew Johnson

Place these presidents in the correct order

experiment order all 44 us presidents
Experiment: Order all 44 US presidents
  • Similar to Roediger and Crowder (1976); Healy, Havas, Parker (2000)
  • Methods
    • 26 participants (college undergraduates)
    • Names of presidents written on cards
    • Cards could be shuffled on large table
measuring performance
Measuring performance

Kendall’s Tau: The number of adjacent pair-wise swaps

= 1

= 1+1

= 2

Ordering by Individual

A B E C D

A B E CD

E

C

D

A B C D E

A B

True Order

A B C D E

empirical results
Empirical Results

(random guessing)

t

a bayesian generative approach
A Bayesian (generative) approach

shared group knowledge

A B C D

(latent random variable)

Generative Model

A D B C

D A B C

B A D C

A C B D

A B D C

bayesian models
Bayesian models
  • We extend two models:
    • Thurstone’s(1927) model
    • Estes (1972) perturbation model
bayesian thurstonian approach
Bayesian Thurstonian Approach

C

B

A

Each item has a true coordinate on some dimension

bayesian thurstonian approach1
Bayesian Thurstonian Approach

Person 1

B

A

C

… but there is noise because of encoding and/or retrieval error

bayesian thurstonian approach2
Bayesian Thurstonian Approach

Person 1

B

A

C

B

C

A

Each person’s mental representation is based on (latent) samples of these distributions

bayesian thurstonian approach3
Bayesian Thurstonian Approach

Person 1

B

A

C

Observed Ordering:

A < B < C

B

C

A

The observed ordering is based on the ordering of the samples

bayesian thurstonian approach4
Bayesian Thurstonian Approach

Person 1

B

A

C

Observed Ordering:

A < B < C

B

C

A

Person 2

B

C

A

Observed Ordering:

A < C < B

C

B

A

People draw from distributions with common means but different variances

bayesian inference problem
Bayesian Inference Problem
  • Given the orderings from individuals, infer:
    • mean for each item
    • standard deviations for each person
  • Markov Chain Monte Carlo (MCMC)
inferred distributions for 44 us presidents
Inferred Distributions for 44 US Presidents

George Washington (1)

John Adams (2)

Thomas Jefferson (3)

James Madison (4)

median and minimumsigma

James Monroe (6)

John Quincy Adams (5)

Andrew Jackson (7)

Martin Van Buren (8)

William Henry Harrison (21)

John Tyler (10)

James Knox Polk (18)

Zachary Taylor (16)

Millard Fillmore (11)

Franklin Pierce (19)

James Buchanan (13)

Abraham Lincoln (9)

Andrew Johnson (12)

Ulysses S. Grant (17)

Rutherford B. Hayes (20)

James Garfield (22)

Chester Arthur (15)

Grover Cleveland 1 (23)

Benjamin Harrison (14)

Grover Cleveland 2 (25)

William McKinley (24)

Theodore Roosevelt (29)

William Howard Taft (27)

Woodrow Wilson (30)

Warren Harding (26)

Calvin Coolidge (28)

Herbert Hoover (31)

Franklin D. Roosevelt (32)

Harry S. Truman (33)

Dwight Eisenhower (34)

John F. Kennedy (37)

Lyndon B. Johnson (36)

Richard Nixon (39)

Gerald Ford (35)

James Carter (38)

Ronald Reagan (40)

George H.W. Bush (41)

William Clinton (42)

George W. Bush (43)

Barack Obama (44)

model can predict individual performance
Model can predict individual performance

t

individual

t

distance to ground truth

s

inferred noise level for each individual

weak wisdom of crowds e ffect
(Weak) Wisdom of Crowds Effect

t

model’s ordering is as good as best individual (but not better)

extension of estes 1972 perturbation model
Extension of Estes (1972) Perturbation Model
  • Main idea:
    • item order is perturbed locally
  • Our extension:
    • perturbation noise varies between individuals and items

True order

A

B

C

D

E

A

C

B

D

E

Recalled order

inferred perturbation matrix and item accuracy
Inferred Perturbation Matrix and Item Accuracy

Abraham Lincoln

Richard Nixon

James Carter

strong wisdom of crowds effect
Strong wisdom of crowds effect

t

Perturbation

Perturbation model’s ordering is better than best individual

alternative heuristic models
Alternative Heuristic Models
  • Many heuristic methods from voting theory
    • E.g., Borda count method
  • Suppose we have 10 items
    • assign a count of 10 to first item, 9 for second item, etc
    • add counts over individuals
    • order items by the Borda count
    • i.e., rank by average rank across people
recollecting order from episodic memory
Recollecting order from episodic memory

http://www.youtube.com/watch?v=a6tSyDHXViM&feature=related

recollecting order from episodic memory1
Recollecting Order from Episodic Memory

Study this sequence of images

example calibration result for individuals
Example calibration result for individuals

t

individual

distance to ground truth

s

inferred noise level

(pizza sequence; perturbation model)

overview of talk1
Overview of talk
  • Ordering problems
    • what is the order of US presidents?
  • Matching problems
    • memory for pairs: what object was paired with what person?
  • Recognition memory problems
    • what words were studied?
  • Experts in crowds
    • how to find experts in the absence of feedback
overview of talk2
Overview of talk
  • Ordering problems
    • what is the order of US presidents?
  • Matching problems
    • memory for pairs: what object was paired with what person?
  • Recognition memory problems
    • what words were studied?
  • Experts in crowds
    • how to find experts in the absence of feedback
experiment
Experiment
  • Study list
    • 10 lists of 15 spoken words
  • Recognition memory test
    • Targets (15 items)
    • Lure (1 item)
    • Related distractors (15 items)
    • Unrelated distractors (15 items)
  • Confidence ratings
    • 5-point confidence ratings
      • 1=definitely not on list; 2 = probably not on list; 3 = not sure; 4 = probably on list; 5 = sure it was on the list
heuristic aggregation method
Heuristic Aggregation Method
  • Group confidence = mean confidence rating across individuals
problem with aggregation method
Problem with aggregation method
  • Aggregate also suffers from false memories

Confidence

overview of talk3
Overview of talk
  • Ordering problems
    • what is the order of US presidents?
  • Matching problems
    • memory for pairs: what object was paired with what person?
  • Recognition memory problems
    • what words were studied?
  • Experts in crowds
    • how to find experts in the absence of feedback
experiment1
Experiment
  • 78 participants
  • 17 ordering problems each with 10 items
    • Chronological Events
    • Physical Measures
    • Purely ordinal problems, e.g.
      • Ten Amendments
      • Ten commandments
ordering states west east
Ordering states west-east

Oregon (1)

Utah (2)

Nebraska (3)

Iowa (4)

Alabama (6)

Ohio (5)

Virginia (7)

Delaware (8)

Connecticut (9)

Maine (10)

ordering ten amendments
Ordering Ten Amendments

Freedom of speech & religion (1)

Right to bear arms (2)

No quartering of soldiers (4)

No unreasonable searches (3)

Due process (5)

Trial by Jury (6)

Civil Trial by Jury (7)

No cruel punishment (8)

Right to non-specified rights (10)

Power for the States & People (9)

question
Question
  • How many individuals do we need to average over?
how effective are small groups of experts
How effective are small groups of experts?
  • Want to find experts endogenously – without feedback
  • Approach: select individuals with the smallest estimated noise levels based on previous tasks
  • We are identifying general expertise(“Pearson’s g”)
group composition based on prior performance
Group Composition based on prior performance

# previous tasks

T = 0

T = 2

T = 8

t

Group size (best individuals first)

slide60

Endogenous no feedback

required

Exogenousselecting people based on actual performance

t

t

summary
Summary
  • Aggregation of combinatorially complex data
    • going beyond numerical estimates or multiple choice questions
  • Incorporate individual differences
    • going beyond models that treat every vote equally
    • assume some individuals might be “experts”
  • Take cognitive processes into account
    • going beyond mere statistical aggregation
that s all
That’s all

Do the experiments yourself:

http://psiexp.ss.uci.edu/

predictive rankings fantasy football
Predictive Rankings: fantasy football

Australian Football League (29 people rank 16 teams)

South Australian Football League (32 people rank 9 teams)

online experiments
Online Experiments
  • Experiment 1 (Prior knowledge)
    • http://madlab.ss.uci.edu/dem2/examples/
  • Experiment 2a (Serial Recall)
    • study sequence of still images
    • http://madlab.ss.uci.edu/memslides/
  • Experiment 2b (Serial Recall)
    • study video
    • http://madlab.ss.uci.edu/dem/
heuristic aggregation approach
Heuristic Aggregation Approach
  • Combinatorial optimization problem
    • maximizes agreement in assigning N items to N responses
  • Hungarian algorithm
    • construct a count matrix M
    • Mij = number of people that paired item i with response j
    • find row and column permutations to maximize diagonal sum
    • O( n3 )
hungarian algorithm example
Hungarian Algorithm Example

= correct

= incorrect

what are methods for finding experts
What are methods for finding experts?

1) Self-reported expertise:

  • unreliable  has led to claims of “myth of expertise”

2) Based on explicit scores by comparing to ground truth

  • but ground truth might not be immediately available

3) Endogenously discover experts

  • Use the crowd to discover experts
  • Small groups of experts can be effective
predicting problem difficulty
Predicting problem difficulty

city size rankings

t

t

distance of group answer to ground truth

ordering states

geographically

std( s )

dispersion of noise levels across individual

mean p yes
Mean p( “yes” )

note: confidence ratings were converted to yes/no judgments. Yes = rating >= 3; No = rating < 3