1 / 18

Evaluation of IR Performance

Evaluation of IR Performance. Dr. Bilal IS 530 Fall 2006. Searching for Information. Imprecise Incomplete Tentative Challenging. IR Performance. Precision Ratio = the number of relevant documents retrieved the total number of documents retrieved. IR Performance. Recall Ratio =

gay-berry
Download Presentation

Evaluation of IR Performance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation of IR Performance Dr. Bilal IS 530 Fall 2006

  2. Searching for Information • Imprecise • Incomplete • Tentative • Challenging

  3. IR Performance Precision Ratio = the number of relevant documents retrieved the total number of documents retrieved

  4. IR Performance Recall Ratio = the number of relevant documents retrieved the total number of relevant documents

  5. Why Some Items Are Not Retrieved? • Indexing errors • Wrong search terms • Wrong database • Language variations Other (to be answered by students)

  6. Why Do We Get Unwanted Items or Results? • Indexing errors • Wrong search terms • Homographs • Incorrect term relations Other (to be answered by students)

  7. Boolean Operators • OR increases recall • AND increases precision • NOT increases precision by elimination

  8. Recall and Precision in Practice • Inversely related • Search strategies designed for high precision or high recall (or medium) • Needs of users dictate search strategy towards recall or precision • Practice helps changing queries to favor recall or precision

  9. Recall and Precision 1.0 Recall 1.0 Precision

  10. Relevance • A match between a query and information retrieved • A judgment • Can be judged by anyone who is informed of the query and views the retrieved information

  11. Relevance • Judgment is dynamic • Documents can be ranked by likely relevance • In practice, not easy to measure • Not focused on user needs

  12. Pertinence • Based on information need rather than a match between a query and retrieved documents • Can only be judged by user • May differ from relevance judgment

  13. Pertinence • Transient, varies with many factors • Not often used in evaluation • May be used as a measure of satisfaction • User-based, as opposed to relevance

  14. High Precision Search • Use these strategies, as appropriate: • Controlled vocabulary • Limit feature (e.g., specific fields, major descriptors, date(s), language, as appropriate) • AND operator • Proximity operators carefully • Truncation carefully

  15. High Recall Search • Use these strategies, as appropriate: • OR logic • Keyword searching • No or minimal limit to specific field(s) • Truncate • Broader terms

  16. Relevance Judgment • Users base it on: • Topicality • Aboutness • Utility • Pertinence • Satisfaction

  17. Improving IR Performance • Good mediation of search topic before searching • User presence during search, if possible • Preliminary search judged by user • Evaluation during search (by searcher or by searcher and user) • Refinement of search strategies • Searcher evaluation of final results • User evaluation of final results

  18. Improving IR Performance • Better system design • Better indexing and word parsing • Better structure of thesauri • Better user interface (e.g., more effective help feature) • Better error recovery feedback • User-centered design

More Related