Only Valuable Experts Can be Valued

1. Only Valuable Experts Can be Valued Moshe Babaioff, Microsoft Research, Silicon Valley LiadBlumrosen, Hebrew U, Dept. of EconomicsNicolas Lambert, Stanford GSB Omer Reingold, Microsoft Research, Silicon Valley and Weizmann.

2. Probabilities of Events • Often, estimating probabilities of future events is important. • Examples: • Weather: probability of rain tomorrow • Online advertising: what is the click probability of the next visitor on our web-site? • What % of Toyota cars is defective? • Many applications in financial markets.

3. Contracts and Screening A decision maker (Alice) An expert (Bob) Uncertain about the probability of a future event. Claimshe knows this probability. Averse to uncertainty, is willing to pay $$$ to reduce it. May be uninformed and pretend to be informed to get $$$…

4. Contracts and Screening • Goal: screen experts. • That is, design contracts such that: • Informed experts will: • accept the contract • reveal the true probability. • Uninformed experts will reject the contract. • Contracts can be based on outcomes only: True probabilities are never revealed.

5. This work • We characterize settings where Alice can separate good experts from bad experts. • We discuss what is a “valuable” expert, and its relation to screening of experts.

6. Outline • Model • With prior: • An easy impossibility result • A positive result • No priors • Extensions

7. Model (1/4) • Ω - Finite set of outcomes. • p- A (true) distribution over Ω. • Unknown to Alice • Φ- The set of possible distributions. • Φ may be restricted, examples to come… • Bayesian assumptions: • Priorfon Φ. • fis known to Alice, Bob. No Prior on Φ. Later…. In the beginning of this talk

8. Model (2/4) Reported probability Realized outcome • Contract: π(q,ω) payment to Bob when reporting q when outcome is ω. • Bob is risk neutral. • Bob’s expected payment depends on what he knows: • Informed: π(q,p) = Eω~p [ π(q,ω) ] • Uninformed: Ep~f Eω~p [ π(q,ω) ] = π(q,E[p]) • Notation: E[payment] upon reporting q when the true probability is p.

9. Model (3/4) • Bob δ-accepts the contract if he has a report q with payment > δ • Otherwise, we say that Bob δ-rejects. • For avoiding handling ties, we aim that for δa> δr : • an informed Bob will δa -accept • an uninformed Bob will δr –reject δa 0 δr

10. Model (4/4) • We actually study a more general model: • experts are ε-informed. • Mixed strategies are allowed. • This talk: perfectly informed, pure strategies.

11. Example • A binary event: Ω = { , } Pr( ) = α p  Φ = { (α , 1-α) | α  [0,1] } • Alice does not know p. • Knows, however, that α ~ U[0,1] . • Sees an a-priori probability of ½. • Bob claims he knows the realization of p. • Contract: Bob reports α. Is paid according to or.

12. Main message • The ability to screen experts closely relates to the structure of Φ • Roughly, on whether Φis convex or not. • If all possible experts are “valuable” to Alice, then screening is possible.

13. An Easy Impossibility • Proposition: screening is impossible. • Proof: • Informed experts always accept the contract π.That is, for all p, we have π(p,p) > δa. • Then, an uninformed agent can get >δa by reporting E[p]:Ep~f Eω~p [π(E[p],ω)] =π(E[p],E[p]) > δa • Reason: when the true probability is E[p] a true expert accepts the contract.

14. Valuable Experts • So screening is impossible when E[p]  Φ. • But experts knowing that the true distribution is E[p] are not really valuable to Alice. • In the binary-outcome example: • Alice’s prior is U[0,1], so she believes that Pr() = ½. • An expert knowing that p=½ is not that helpful… • What if all experts are “valuable”?

15. Possibility result • Theorem: when E[p] is not in Φ(and Φis closed), screening is possible. • When all experts are valuable, we can screen... E[p] Φ • Immediate questions: • Is non-convex Φnatural? • Can we expect Φ not to contain E[p]?

16. Non-convex Φ: examples • Example 1: a coin which is either fair (p=1/2) or biased (p=3/4) • For any (non-trivial) prior, E[p] not in Φ. • Example 2: many standard distributions are not closed under mixing. • E.g., uniform, normal, etc. 1/2 3/4

17. Non-convex Φ: examples • Example 3: The binary outcome example. • But now, Alice observes two samples. • For example, we wish to know the failure rate in cars, and we thoroughly check 2 random Toyotas.

18. Non-convex Φ: examples • Example 3: The binary outcome example. But now, Alice observes two samples. Ω = { ( , ), ( , ), ( , ) , ( , ) } Φ = { ( α2, α(1- α), (1-α)α, (1-α)2) } α[0,1] Φ is not convex! • Moreover, for every p,p’, the convex combination of the above vectors is not in Φ.  For every prior, E[p] is not in Φ, and thus screening is possible.

19. Possibility result • Theorem: when E[p] is not in Φ(and Φis closed), screening is possible. Φ • One definition before proceeding to the proof…

20. Scoring rules • Scoring rules: • Contracts that elicit distributions from experts. • S(q,ω) = payment for an expert reporting q when the realized outcome is ω. • A scoring rule is strictly proper if the expert is always strictly better off by reporting the true distribution. • Strictly proper scoring rules are known to exist [Brier ‘50, , Good ’52, Savage ‘71,…] • We want, in addition, to screen good experts from bad.

21. Possibility result • Proof: • Let s be some strictly proper scoring rule. • On the full probability space • The following contracts screens experts: • Theorem: when E[p] is not in Φ(and Φis closed), screening is possible. Φ We need to show: 1. An informed expert δa-accepts. • 2. An uninformed expert δr-rejects.

22. Possibility result • Proof: • Let s be some strictly proper scoring rule. • On the full probability space • The following contracts screens experts: • Theorem: when E[p] is not in Φ(and Φis closed), screening is possible. Φ ≥1 An informed expert reporting the truth p gains:

23. Possibility result • Proof: • Let s be some strictly proper scoring rule. • The following contracts screens experts: • Theorem: when E[p] is not in Φ(and Φis closed), screening is possible. Φ Since s is strictly proper, for every q: An uninformed expert will gain:

24. Outline • Model • With prior: • An easy impossibility result • A positive result • No priors • Extensions

25. Related work • [Olszewski & Sandroni 2007] studied a similar model: • A binary event with unknown probability p. • No priors: • An uninformed expert accepts a contract if it is good in the worst case. • Theorem [O&S]: • Use min-max theorems. • The probability space is convex. All informed experts accept a contract An uninformed expert also accepts it

26. Valuable Experts: No Prior • We claim: invaluable agents are also behind this impossibility. • But what is a valuable agent without priors? • What are Alice’s utility function and actions? • A(p) : Alice’s action when she knows p. • U( A(p),p ): utility maximizing actions. Theorem: Φis convex (and closed) There exists psuch thatU(A(p),p)=U(A(“reject”),p) no prior on Φ • Interpretation: if Φis convex then some informed expert is not valuable.

27. No prior: positive result • We have an analogues positive result for the no-prior case:non-convexΦscreening is possible. • (we use the with-prior positive result in the proof)

28. Outline • Model • With prior: • An easy impossibility result • A positive result • No priors • Extensions

29. Extension: forecasting • A related line of researchis forecasting: • An unbounded sequence of events. • An expert provides a forecast before each event occurs. • Goal: test the expert.

30. Forecasting: related work • Negative results are known: • Informed experts pass the test  uninformed experts can do it too.[e.g., Foster & Vohra ‘97, Fudenberg & Levine ‘99] • When forecasting is possible, decisions can be delayed arbitrarily.[Olszewski & Sandroni ‘09] • Some works around this impossibility: • [Olszewski & Sandroni ‘09] show a counter-example by constructing non-convex set of distributions. • [Al-Najjar & Sandroni & Smorodinsky & Weinstein ‘10] Describe a class of distribution such that decisions can be made in time. The relevant class of distributions also admits non-convexities.

31. Extension: forecasting • We extend our approach to forecasting settings. • In the works. • We characterize conditions on the set of distributions that allow expert testing. • Analysis is more involved, but the ideas are similar. • Results relate to the convexity of Φ. • For example: two samples at each period enable testing.

32. Summary • A decision maker want to hire an expert. • For learning the probability of some future event. • The expert may be a charlatan. • Can the decision maker separate good experts from bad ones? • We characterize the settings where such screening is possible. • With or without priors on Φ. • We design screening contracts.

33. Thanks!