1 / 26

Effective Time Ratio: A measure for Web search engine with document snippet

Effective Time Ratio: A measure for Web search engine with document snippet. Outline. Introduction Effective Time Ratio Experiment and Results Conclusion. Existing IR Measures. search engine quality. doc relevance. Snippets. Snippets ： False Positive.

dima
Download Presentation

Effective Time Ratio: A measure for Web search engine with document snippet

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Effective Time Ratio:A measure for Web searchengine with document snippet

  2. Outline • Introduction • Effective Time Ratio • Experiment and Results • Conclusion

  3. Existing IR Measures search engine quality doc relevance

  4. Snippets

  5. Snippets：False Positive Waste time examining irrelevant document!

  6. Snippets：False Negative Miss really relevant document!

  7. Snippets Affect the Search Engine Quality!

  8. Outline • Introduction • Effective Time Ratio • Experiment and Results • Conclusion

  9. IntuitionReduce time wasted on examining irrelevant information

  10. Effective Time Ratio (ETR) of getting relevant information of total search

  11. Precision:a special case of ETR • Assumption: constant time(T) on each doc P@5 = 0.4 of getting rel information = 2T of total search = 5T ETR@5 = 0.4

  12. User Behavior Assumptions • Examination Assumption[Dupret+08,Craswell+08, Guo+09a,Guo+09b] • Cascade Assumption[Craswell+08, Guo+09a,Guo+09b, Chapelle+09] • Examination Time Assumption • Snippet Examination Time: T1 • Doc Examination Time: T2 Position i Position (i+1)

  13. User Behavior

  14. ETR for Search Engine with Snippets of getting rel information T1 + T2 of total search 5T1 + 3T2

  15. Factors affecting ETR • Retrieval System Performance(P@N) • Snippet Quality • First Type Error(False Positive) • Second Type Error(False Negative) P1 = Pr( | ) P2 = Pr( | )

  16. Expected ETR • C = T1/T2

  17. Properties • It helps (increases ETS) by using perfect snippets (with no first/second type error)

  18. Properties • ETR can reflect snippet generation/retrieval system quality

  19. Outline • Introduction • Effective Time Ratio • Experiment and Results • Conclusion

  20. User Study Data • 10 Users • 50 Questions from commercial query log • 25 open + 25 close • Various topics/difficulty level • For each question, 100 top results(snippet + doc) from a search engine

  21. User Study • Each user examines the results to answer the question • Write the answer • Report the satisfaction degree • Three assessors judges each snippet/doc as relevant/irrelevant

  22. Measure Validation • Correlation[Huffman+07] between • User reported satisfaction • Measure value based on judgments • Higher correlation indicates the measure can reflect user satisfaction better

  23. Measures • Traditional IR measures • P@N, RR, DCG, CP • Version of measure including snippet quality [Turpin+09] • P’@N, RR’, DCG’, CP’ • ETR and its cumulated version • ETR and CETR

  24. Results • Up-bound: 0.706 (avg correlation between users) • Traditional Measures • Measures including snippet quality[Turpin+09] • ETR and its cumulated version

  25. Results • Open Questions • Close Questions

  26. Conclusion • Search engine quality is also affected by snippets • ETR can reflect both retrieval system and snippet generation algorithm quality • ETR can reflect user satisfaction better than existing measures

More Related