1 / 27

Mutual Information Scheduling for Ranking

Mutual Information Scheduling for Ranking. Hamza Aftab Nevin Raj Paul Cuff Sanjeev Kulkarni Adam Finkelstein. Applications of Ranking. Pair-wise Comparisons. Query: A > B ? Ask a voter whether candidate I is better than candidate J Observe the outcome of a match. Scheduling.

nanda
Download Presentation

Mutual Information Scheduling for Ranking

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Mutual Information Scheduling for Ranking HamzaAftab Nevin Raj Paul Cuff SanjeevKulkarni Adam Finkelstein

  2. Applications of Ranking

  3. Pair-wise Comparisons • Query: • A > B ? • Ask a voter whether candidate I is better than candidate J • Observe the outcome of a match

  4. Scheduling • Design queries dynamically, based on past observations.

  5. Example: Kitten Wars

  6. Example: All Our Ideas(Matthew Salganik – Princeton)

  7. Select Informative Matches • Assume matches are expensive but computation is cheap • Previous Work (Finkelstein) • Use Ranking Algorithm to make better use of information • Select matches by giving priority based on two criterion • Lack of information: Has a team been in a lot of matches already? • Comparability of the match: Are the two teams roughly equal in strength? • Our innovation • Select matches based on Shannon’s mutual information

  8. Related Work • Sensor Management (tracking) • Information-Driven • [Manyika, Durrant-Whyte 1994] • [Zhao et. al. 2002] – Bayesian filtering • [Aoki et. al. 2011] – This session • Learning Network Topology • [Hayek, Spuckler 2010] • Noisy Sort

  9. Ranking Algorithms – Linear Model • Each player has a skill level µ • The probability that Player I beats Player J is a function of the difference µi - µj • Transitive • Use Maximum Likelihood • Thurstone-Mosteller Model • Q function • Performance has Gaussian distribution about the mean µ • Bradley-Terry Model • Logistic function

  10. Examples • Elo’s chess ranking system • Based on Bradley-Terry model • Sagarin’s sports rankings

  11. Mutual Information • Mutual Information: • Conditional Mutual information

  12. Entropy • Entropy: • Conditional Entropy High entropy Low entropy

  13. Mutual Information Scheduling • Let R be the information we wish to learn • (i.e. ranking or skill levels) • Let Ok be the outcome of the kth match • At time k, scheduler chooses the pair (ik+1, jk+1):

  14. Why use Mutual Information? • Additive Property • Fano’s Inequality • Related entropy to probability of error • For small error: • Continuous distributions: MSE bounds differential entropy

  15. Greedy is Not Optimal • Consider Huffman codes---Greedy is not optimal

  16. Performance (MSE)

  17. Performance (Gambling Penalty)

  18. Identify correct ranking

  19. Find strongest player

  20. Find strongest player

  21. Evaluating Goodness-of-Fit 1 4 1 2 3 4 • Ranking: • Inversions • Skill Level Estimates: • Mean squared error (MSE) • Kullback-Leibler (KL) divergence (relative entropy) • Others • Betting risk • Sampling inconsistency 3 2

  22. Numerical Techniques • Calculate mutual information • Importance sampling • Convex Optimization (tracking of ML estimate)

  23. Summary of Main Idea • Get the most out of measurements for estimating a ranking • Schedule each match to maximize • (Greedy, to make the computation tractable) • Flexible • S is any parameter of interest, discrete or continuous • (skill levels; best candidate; etc.) • Simple design---competes well with other heuristics

  24. Ranking Based on Pair-wise Comparisons • Bradley Terry Model: • Examples: • A hockey team scores Poisson- goals in a game • Two cities compete to have the tallest person • is the population

  25. Computing Mutual Information • Importance Sampling: • Multidimensional integral • Probability distributions • Skill level estimates Skill level of player 2 Skill level of player 1 • Why is it good for estimating skill levels? • Faster than convex optimization • Efficient memory use

  26. (for a 10 player tournament and100 experiments) Results

  27. Visualizing the Algorithm Outcomes Scheduling A B C D ?

More Related