1 / 32

Economics and Search

Economics and Search. Hal Varian SIGIR, August 16, 1999 http://www.sims.berkeley.edu/~hal. Three points of contact. 1. Value of information 2. Estimating degree of relevance 3. Optimal search behavior. 1. Value of information. Economic value of information

dacian
Download Presentation

Economics and Search

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Economics and Search Hal Varian SIGIR, August 16, 1999 http://www.sims.berkeley.edu/~hal

  2. Three points of contact • 1. Value of information • 2. Estimating degree of relevance • 3. Optimal search behavior

  3. 1. Value of information • Economic value of information • More information helps us make better decisions • Economic value of information = value of best decision with information - value of best decision without the information • Increase in expected utility due to the better decision, or decrease in expected cost

  4. Properties • Information has non-negative private value (because it can be ignored) • Information is valuable only when it is “new” -- when it changes a decision • Example • financial information gets quickly incorporated into stock prices • subsequent “news” may not move prices • “buy on the rumor, sell on the news”

  5. Relevance to search? • Information is valuable when it is “new” • “Relevance” captures only part of information value since a document may be relevant but not “new” • Example • repeated occurrence of documents • many similar documents

  6. How to handle? • Post-retrieval clustering • often-proposed strategy • for disambiguation • organization • possible additional motivation • maximize the “information content” in each new document cluster • may allow for more effective search

  7. 2. Estimating relevance • Estimate probability of relevance as function of characteristics of document and query • E.g., logistic regression a la Bill Cooper • Why logistic form? • Formerly data-poor environment • Had to assume functional form • Now that we have a data-rich environment, can use nonparametric methods

  8. Example with TREC dat • 100,102 WSJ doc-query pairs for fitting • 173,330 WSJ doc-query pairs for extrapolation • One explanatory variable: x=terms in common (after stemming, etc.) • (Thanks to Aito Chen and Fred Gey for data)

  9. Outline of estimation • Maximum likelihood (classical procedure) • Calculate frequencies of relevance as a function of terms-in-common • fit by logistic transformation • fit by nonparametric regression • Compare shapes of fitted functions

  10. Frequency of relevance • Look at all document-query pairs with 1 word-in-common • See what fraction of these are relevant • Repeat for 2, 3, 4 … words in common • generates a histogram with words-in-common on horizontal axis, frequency of relevance on vertical axis

  11. ML-fitted logit and freqs

  12. Direct estimate of logit • Logit • p(x) = exb/(1+exb) • p(x)/(1-p(x)) = exb • Regression • log [fi/(1-fi)] = xb • Note: have to censor observations fi = 0 or 1

  13. Results

  14. Nonparametric regression • Find monotone function that minimizes sum of squared residuals between observations and fitted expression • PAV = “pool adjacent violators” algorithm doesn’t require solving minimization problem directly

  15. Nonparametric results

  16. Further smoothing

  17. Extrapolation to other data

  18. Further work • Add another variable, e.g., • query length/ document length • “inverse document frequency” • Look at other collections • Note: since there is only one variable, recall-precision is same for all estimators

  19. 3. Search behavior • Economic model: search for lowest price or highest wage • With or without “recall” (revisit stores) • Results do not cumulate, care only about the max • May or may not be natural in IR context • Of course, can generalize to k-best choices

  20. Example • Marty Weitzman’s “Pandora problem” • “Optimal Search for the Best Alternative”, Econometrica, May 1979 • n boxes • reward in box i is random with cdf Fi(x) • costs ci to open a box, time discount factor d<1 • payoff is maximum value found up to point when you stop opening

  21. IR story • You work at airport book store • people are in a hurry (d < 1) • mental effort to examining books (c > 0) • will only take one book with them • you have an idea of how likely it is that person will like the book (Fi(x)) • Problem: in what order to show them books?

  22. Analysis • State is summarized by maximum reward so far • Question is whether to open next box • Can be solved by dynamic programming

  23. Nature of solution • Assign a “score” to each box • depends only on that box • can be computed “easily” • Selection rule: if you open a box, open that box with the highest score • Stopping rule: stop searching when the maximum sampled reward exceeds the score of every closed box

  24. Riskiness and search order • Score is not expected value • “Other things being equal, it is optimal to sample first from distributions that are more spread out or riskier in hopes of striking it rich early and ending the search.” • “Low-probability, high-payoff situations should be prime candidates for early investigation…”

  25. Simple example • Box S: gives 6 for sure • Box R: equally likely to give 10 or 0 • Note: expected value of S > expected value of R

  26. Open box S first • Have 6 for sure, should you continue? • 1/2 of time get 10d -c • 1/2 of time get -c • expected payoff from continuing is 5d - c • this is less than 6 • Conclusion • if open box S first, get payoff of 6 and will not continue

  27. Open box R first • 1/2 of time get 10 • can’t do any better, so stop • 1/2 of time get 0 • continue if 6d-c > 0 (1) • expected payoff = 5 +3d - c/2 • opening R first is best strategy if • 5 + 3d - c/2 > 6, or • 6d - c > 2 [if this is true (1) is true]

  28. Summary • If 6d - 2 < c, open S first and stop • If 6d -2 > c, open R first • if get 10, stop • if get 0, open S • small search cost and small time preference implies open risky box first

  29. Airport bookstore • Customer runs in says “I want a travel guide to Borneo.” • S = Fodors, R = Lonely Planet • Which do you show first? • If only time for one book, show Fodors • If time for two books, show Lonely Planet • Why: may be able to stop search early and get higher payoff

  30. Risk and search • Don’t necessarily want to order search by expected payoff • Want some high-variance choices early to reduce search costs/time • Generalization • Want to sample from high-variance populations (if they have similar means) • Result depends on time-value, search cost, utility is maximum of choices

  31. Estimation of value? • From a Bayesian perspective, forecast relevance (or value) is random variable • as in regressions described earlier • Can apply a Weitzman-type rule to determine optimal order • Is it worth the effort? Depends on how good an estimate of value, discount factor, search cost we have...

  32. Summary • Information has economic value since it helps make better decisions • Nonlinear estimation (which requires lots of data) may be useful in prediction • Risk and search cost are important factors for determining optimal search order and stopping rule

More Related