accurately interpreting clickthrough data as implicit feedback n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Accurately Interpreting Clickthrough Data as Implicit Feedback PowerPoint Presentation
Download Presentation
Accurately Interpreting Clickthrough Data as Implicit Feedback

Loading in 2 Seconds...

play fullscreen
1 / 22

Accurately Interpreting Clickthrough Data as Implicit Feedback - PowerPoint PPT Presentation


  • 140 Views
  • Uploaded on

Accurately Interpreting Clickthrough Data as Implicit Feedback. Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05. Introduction. Adapt a retrieval system to users and or collections Manual adaptation - time consuming or even impractical

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Accurately Interpreting Clickthrough Data as Implicit Feedback' - avalbane


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
accurately interpreting clickthrough data as implicit feedback

Accurately Interpreting Clickthrough Data as Implicit Feedback

Joachims, Granka, Pan, Hembrooke, Gay

Paper Presentation: Vinay Goel

10/27/05

introduction
Introduction
  • Adapt a retrieval system to users and or collections
  • Manual adaptation - time consuming or even impractical
  • Explore and evaluate implicit feedback
  • Use clickthrough data in WWW search
user study
User Study
  • Record and evaluate user actions
  • Provide insight into the decision process
  • Record users’ eye movements : Eye tracking
two phases of the study
Two Phases of the study
  • Phase I
    • 34 participants
    • Start search with Google query, search for answers
  • Phase II
    • Investigate how users react to manipulations of search results
    • Same instructions as phase I
    • Each subject assigned to one of three experimental conditions
      • Normal, Swapped, Reversed
explicit relevance judgments
Explicit Relevance Judgments
  • Collected explicit relevance judgments for all queries and results pages
  • Inter-judge agreements
analysis of user behavior
Analysis of user behavior
  • Which links do users view and click?
  • Do users scan links from top to bottom?
  • Which links do users evaluate before clicking?
which links do users view and click
Which links do users view and click?
  • Almost equal frequency of 1st and 2nd link, but more clicks on 1st link
  • Once the user has started scrolling, rank appears to become less of an influence
do users scan links from top to bottom
Do users scan links from top to bottom?
  • Big gap before viewing 3rd ranked abstract
  • Users scan viewable results thoroughly before scrolling
which links do users evaluate before clicking
Which links do users evaluate before clicking?
  • Abstracts closer above the clicked link are more likely to be viewed
  • Abstract right below a link is viewed roughly 50% of the time
analysis of implicit feedback
Analysis of Implicit Feedback
  • Does relevance influence user decisions?
  • Are clicks absolute relevance judgments?
does relevance influence user decisions
Does relevance influence user decisions?
  • Yes
  • Use the “reversed” condition
    • Controllably decreases the quality of the retrieval function and relevance of highly ranked abstracts
  • Users react in two ways
    • View lower ranked links more frequently, scan significantly more abstracts
    • Subjects are much less likely to click on the first link, more likely to click on a lower ranked link
clicks absolute relevance judgments
Clicks = absolute relevance judgments?
  • Interpretation is problematic
  • Trust Bias
    • Abstract ranked first receives more clicks than the second
      • First link is more relevant (not influenced by order of presentation) or
      • Users prefer the first link due to some level of trust in the search engine (influenced by order of presentation)
trust bias
Trust Bias
  • Hypothesis that users are not influenced by presentation order can be rejected
  • Users have substantial trust in search engine’s ability to estimate relevance
quality bias
Quality Bias
  • Quality of the ranking influences the user’s clicking behavior
    • If relevance of retrieved results decreases, users click on abstracts that are on average less relevant
    • Confirmed by the “reversed” condition
are clicks relative relevance judgments
Are clicks relative relevance judgments?
  • An accurate interpretation of clicks needs to take into consideration
    • User’s trust into quality of search engine
    • Quality of retrieval function itself
  • Difficult to measure explicitly
  • Interpret clicks as pairwise preference statements
strategy 1
Strategy 1
  • Takes trust and quality bias into consideration
  • Substantially and significantly better than random
  • Close in accuracy to inter judge agreement
strategy 2
Strategy 2
  • Slightly more accurate than Strategy 1
  • Not a significant difference in Phase II
strategy 3
Strategy 3
  • Accuracy worse than Strategy 1
  • Ranking quality has an effect on the accuracy
strategy 4
Strategy 4
  • No significant differences compared to Strategy 1
strategy 5
Strategy 5
  • Highly accurate in the “normal” condition
  • Misleading
    • Aligned preferences probably less valuable for learning
    • Better results even if user behaves randomly
  • Less accurate than Strategy 1 in the “reversed” condition
conclusion
Conclusion
  • Users’ clicking decisions influenced by search bias and quality bias
  • Strategies for generating relative relevance feedback signals
  • Implicit relevance signals are less consistent with explicit judgments than the explicit judgments among each other
  • Encouraging results