The Wisdom of Crowds in the Aggregation of Rankings

1. The Wisdom of Crowds in the Aggregation of Rankings Mark Steyvers Department of Cognitive Sciences University of California, Irvine Joint work with: Michael Lee, Brent Miller, Pernille Hemmer

2. Rank aggregation problem • Goal is to combine many different rank orderings on the same set of items in order to obtain a “better” ordering • Example applications • Combining voters rankings: social choice theory • Information retrieval and meta-search* • *e.g. Lebanon & Mao (2008); Klementiev, Roth et al. (2008; 2009), Dwork et al. (2001)

3. Example ranking problem in our research What is the correct chronological order? Abraham Lincoln Ulysses S. Grant time Ulysses S. Grant Rutherford B. Hayes Rutherford B. Hayes James Garfield Abraham Lincoln Andrew Johnson James Garfield Andrew Johnson

4. Aggregating ranking data ground truth group answer ? A B C D = A B C D Aggregation Algorithm A D B C D A B C B A D C A C B D A B D C

5. Generative Approach latent truth ? ? ? ? Generative Model A D B C D A B C B A D C A C B D A B D C

6. Wisdom of crowds phenomenon • Aggregating over individuals often leads to an estimate that is among the best individual estimates (or sometimes better) Galtons Ox (1907): Median of individual weight estimates came close to true answer

7. Approach • No communication between individuals • There is always a true answer (ground truth) • ground truth only used in evaluation • Unsupervised weighting of individuals* • exploit relationship between expertise and consensus • experts tend to be closer to the truth and therefore reach more similar judgments • Incorporate prior knowledge about latent truth • discount a priori bad rankings • * Klementiev, Roth et al. (2008, 2009); Dani, Madani, Pennock et al. (2006). Bayesian truth serum (Prelec et al., 2004); Cultural Consensus Theory (Batchelder and Romney, 1986)

8. Overview of talk • General knowledge tasks • reconstructing order of US presidents • Thurstonian models • Sports prediction • forecasting NBA and NCAA outcomes • Thurstonian models • Episodic memory • reconstructing order of personally experienced events • Mallows model

9. Experiment: 26 individuals order all 44 US presidents

10. Measuring performance Kendall’s Tau: The number of adjacent pair-wise swaps = 1 = 1+1 = 2 Ordering by Individual A B E C D A B E C D E C D A B C D E A B True Order A B C D E

11. Empirical Results (random guessing) t

12. Unsupervised models for ranking data • Classic models: • Thurstone (1927) • Mallows (1957); Fligner and Verducci, 1986 • Diaconis (1989) • Voting methods: e.g. Borda count (1770) • We will focus on Thurstonian and Mallows models • implemented as graphical models • MCMC inference Many models were developed for preference rankings and voting situations  no known ground truth

13. Thurstonian Model A. George Washington B. James Madison C. Andrew Jackson Each item has a true coordinate on some dimension

14. Thurstonian Model A. George Washington B. James Madison C. Andrew Jackson … but there is noise because of encoding errors

15. Thurstonian Model A. George Washington B. James Madison C. Andrew Jackson A B C Each persons mental encoding is based on a single sample from each distribution

16. Thurstonian Model A. George Washington B. James Madison C. Andrew Jackson A A < C < B B C The observed ordering is based on the ordering of the samples

17. Thurstonian Model A. George Washington B. James Madison C. Andrew Jackson A A < B < C B C The observed ordering is based on the ordering of the samples

18. Thurstonian Model A. George Washington B. James Madison C. Andrew Jackson Important assumption: across individuals, variance can vary but not the means

19. Graphical Model of Extended Thurstonian Model Latent truth Expertise of individual Mental samples Observed ordering j individuals

20. Inferred Distributions for 44 US Presidents George Washington (1) John Adams (2) Thomas Jefferson (3) James Madison (4) James Monroe (6) John Quincy Adams (5) Andrew Jackson (7) Martin Van Buren (8) William Henry Harrison (21) John Tyler (10) James Knox Polk (18) Zachary Taylor (16) Millard Fillmore (11) Franklin Pierce (19) James Buchanan (13) Abraham Lincoln (9) Andrew Johnson (12) Ulysses S. Grant (17) Rutherford B. Hayes (20) James Garfield (22) Chester Arthur (15) Grover Cleveland 1 (23) Benjamin Harrison (14) Grover Cleveland 2 (25) William McKinley (24) Theodore Roosevelt (29) William Howard Taft (27) Woodrow Wilson (30) Warren Harding (26) Calvin Coolidge (28) Herbert Hoover (31) Franklin D. Roosevelt (32) Harry S. Truman (33) Dwight Eisenhower (34) John F. Kennedy (37) Lyndon B. Johnson (36) Richard Nixon (39) Gerald Ford (35) James Carter (38) Ronald Reagan (40) George H.W. Bush (41) William Clinton (42) George W. Bush (43) Barack Obama (44) error bars = median and minimum sigma

21. Calibration of individuals t individual t distance to ground truth s inferred noise level for each individual

22. Wisdom of crowds effect t

23. Heuristic Models • Many heuristic methods from voting theory • E.g., Borda count method • Suppose we have 10 items • assign a count of 10 to first item, 9 for second item, etc • add counts over individuals • order items by the Borda count • i.e., rank by average rank across people

24. Model Comparison t Borda

25. Other ordering tasks Ten Commandments Ten Amendments

26. Overview of talk • General knowledge tasks • reconstructing order of US presidents • Sports prediction • forecasting NBA and NCAA outcomes • Episodic memory • reconstructing order of personally experienced events • New directions

27. Human forecasting experiment • Forecast end-of-season rankings for 15 NBA teams • Eastern conference • Western conference • Participants were college undergraduates • heterogeneous population regarding basketball expertise • 172 individuals for Eastern conference • 156 individuals for Western conference • Experiment conducted Feb 2010 • teams have played about a bit over half of games in regular season

28. Model predictions for Eastern conference Borda Boston Cleveland Orlando Miami Detroit Chicago Philadelphia Atlanta New York New Jersey Indiana Washington Toronto Charlotte Milwaukee Actual outcome Cleveland Orlando Atlanta Boston Miami Milwaukee Charlotte Chicago Toronto Indiana New York Detroit Philadelphia Washington New Jersey Thurstonian Model Cleveland Boston Orlando Miami Atlanta Chicago Detroit Charlotte Toronto Philadelphia Washington Indiana New York Milwaukee New Jersey

29. East 73% t 93% West t 87% 94%

30. Calibration Results East West t t s s

31. Heuristics: who will win more games? Chicago Bulls Charlotte Bobcats vs Won 0 championships Team in existence for 6 years Won 6 championships Team in existence for 44 years Related to work on “fast and frugal heuristics” by Gigerenzer et al.

32. Heuristic ranking by #championships won Actual outcome Cleveland Orlando Atlanta Boston Miami Milwaukee Charlotte Chicago Toronto Indiana New York Detroit Philadelphia Washington New Jersey #championships Boston Chicago Philadelphia Detroit Indiana New York New Jersey Atlanta Washington Milwaukee Miami Orlando Cleveland Toronto Charlotte

33. Informative Priors on Expertise • Individuals who closely follow heuristic orderings are probably not experts • Set hyperparameters of variance prior based on distance to heuristic ordering prior for individual who closely follows heuristic ordering s

34. Graphical Model j individuals

35. East 73% 93% t 96% West t 87% 94% 96%

36. Forecasting NCAA tournament (March Madness) • 64 US college basketball teams are placed in a set of four seeded brackets, and play an elimination tournament. • Midwest bracket:

37. Data • Predictions from 16,718 Yahoo users • Each individual predicts the winner of all games • We use the predictions for the first four rounds (60 games total) • Two scoring systems • Number of correct predictions • Points: • 1 point per correct winner in 1st round • 2 points in 2nd • 4 points in 3rd • 8 points in 4rd

38. Data and Results of Heuristic Strategies majority rule 73% priorseeding 61% Obama 83% #correct predictions majority rule 71% priorseeding 66% Obama 47% points individuals

39. Thurstonian Model Team A Team B Team C • Each team has a mean on a single “strength” dimension • Each person has single variance

40. Thurstonian Model Team A Team B Team C A B wins over A B • The probability a person will choose team A over team B is the probability their strength for team A will be sampled above team B

41. Thurstonian Model Team A Team B Team C B C wins over B C • The probability a person will choose team A over team B is the probability their strength for team A will be sampled above team B

42. Modeling Results Thurstonian modelinform. priors 90% Thurst model 83% majority rule 73% priorseeding 61% #correct predictions Thurst. modelinform. priors 81% Thurst. model 78% majority rule 71% priorseeding 66% points individuals

43. Overview of talk • General knowledge tasks • reconstructing order of US presidents • Sports prediction • forecasting NBA and NCAA outcomes • Episodic memory • reconstructing order of personally experienced events

44. Recollecting Order from Episodic Memory Study this sequence of images

45. How good is your memory? Place the images in the correct sequence (by reading order) B C D A F H E G J I

46. Problem • What if we have only a small number of individuals? • How can we guard against individuals with poor memory? • Idea: “smooth” the inferred group ordering with a prior

47. Approach • Empirically measure the prior orderings over events • Experiment: a separate group of individuals orders the images without seeing original video • Use this data to construct a prior on the group ordering

48. Mallows Model Kendall tau distance ω latent truth θj expertise for person j observed ranking for person j yj (memory data)

49. Mallows Model with an informative prior on the latent truth ωo prior on orderings θ* ω latent truth θoj θj expertise for person j observed ranking for person j yoj yj (prior knowledge data) (memory data)

50. Results when picking K worst “witnesses” uniform prior informative prior t Number of “witnesses” (K)