referee an open framework for practical testing of recommender systems using researchindex
Download
Skip this Video
Download Presentation
REFEREE: An open framework for practical testing of recommender systems using ResearchIndex

Loading in 2 Seconds...

play fullscreen
1 / 31

REFEREE: An open framework for practical testing of recommender systems using ResearchIndex - PowerPoint PPT Presentation


  • 100 Views
  • Uploaded on

REFEREE: An open framework for practical testing of recommender systems using ResearchIndex. Dan Cosley , Steve Lawrence, David M Pennock. Proceedings of the 28 th VLDB Conference Hong Kong, China, 2002. Presenter: Raghav Karumur Date: 4/11/2011

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'REFEREE: An open framework for practical testing of recommender systems using ResearchIndex' - bayle


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
referee an open framework for practical testing of recommender systems using researchindex

REFEREE: An open framework for practical testing of recommender systems using ResearchIndex

Dan Cosley, Steve Lawrence, David M Pennock

Proceedings of the 28th VLDB Conference Hong Kong, China, 2002

Presenter: Raghav Karumur

Date: 4/11/2011

Course: [CSCI 8735] Advanced Database Systems

Department of Computer Science and Engineering

University of Minnesota, Twin Cities

Spring 2011

background
Background

REcommenderFramework and Evaluator for REsEarchIndex

An autonomous citation indexing system that indexes a large fraction of the computer science literature available on the web.

  • It locates papers, downloads them, and automatically extracts information such as abstracts, authors, and citations.
  • It provides citation links both to and from a document.
  • It computes similarity: “users who viewed x also viewed y”.
  • What is REFEREE?
  • What is ResearchIndex?

Advanced Database Systems Raghav Karumur Spring 2011

motivation
Motivation

Currently we have lot of automated recommender systems!!

Advanced Database Systems Raghav Karumur Spring 2011

motivation1
Motivation

Problem:

No standard, public, real-world test-bed for evaluating the recommendation algorithms.

How this paper addresses this issue?

This paper built one such test-bed REFEREE using ResearchIndex database so that anyone can develop , deploy or evaluate recommender systems relatively easily and quickly.

Why did they choose ResearchIndex?

Because it is : (author, title, abstract, fulltext), collaborative filtering information (user behaviors), as well as linkage info(citations).

content rich

Advanced Database Systems Raghav Karumur Spring 2011

rest of the paper
Rest Of the Paper:

Types of Recommenders

A brief history of CF

Hybrid systems

Why use Research Index?

REFEREE

PD-Live: An example recommender

Results and Discussion

Conclusion

References

Advanced Database Systems Raghav Karumur Spring 2011

types of recommenders
Types of Recommenders

alone uses several recommender systems:

  • Non-personalized:

Ex: “Users who bought Men Are From Mars, Women Are From Venus also bought Space

Invaders From Pluto!”.

  • Users post comments about an item. Customers read the comments before they buy.
  • Personalized recommendations based on user’s ratings (explicit or implicit).

(An example of a pure collaborative filtering (CF) system that computes recommendations based only on the ratings (explicit or implicit) given by users, ignoring product attributes.)

As a result, CF systems are completely domain independent.

Advanced Database Systems Raghav Karumur Spring 2011

collaborative filtering
Collaborative Filtering
  • Earlier collaborative filtering systems used to be manual. User had to create filters, know the product, etc.
  • Automated collaborative filtering addresses this problem, users assign ratings to items. System uses this rating to suggest users with similar interests.
  • Original algorithms use similarity metrics such as correlation, MS distance, vector similarity, etc or use ML algorithms such as clustering, neural networks, Bayesian networks, etc.

Advanced Database Systems Raghav Karumur Spring 2011

collaborative filtering1
Collaborative Filtering

However, CF systems can fail when

  • Data is too sparse,
  • When recommending new items, cold start problem.
  • or when recommending to new users.

More sophisticated recommender systems combine user ratings with domain-specific descriptive information about the items.

Advanced Database Systems Raghav Karumur Spring 2011

hybrid systems
Hybrid Systems
  • content-based filtering + collaborative filtering
  • Standard databases such as EachMovie dataset of movie ratings, exist for testing pure CF algorithms.
  • This is not the case with hybrid systems!
  • You have to populate it with content information from databases such as IMDB.
  • Ex: filterbot, P-Tango, PD-Live etc

Advanced Database Systems Raghav Karumur Spring 2011

hybrid systems1
Hybrid Systems

“Makes recommendations from all 1,000,000 documents in <200msec”

  • PD-Live is a hybrid recommender they have built in order to evaluate the performance of this testbed.
  • It uses content based filtering to select the documents and collaborative filtering to rank them.
  • They claim that PD-live performs reasonably well compared to other recommenders in ResearchIndex.

Advanced Database Systems Raghav Karumur Spring 2011

why use researchindex as a testbed
Why use ResearchIndex as a testbed?

The algorithm designers of this paper have also taken performance into consideration as >1000 users access this database of >1,000,000 papers per hour.

Research papers have both objective ("this work is related to mine") and taste ("I think this work is interesting") components, thus can help for both content-based and collaborative filtering.

Recommender system developers can install a local copy of ResearchIndex, and use a rich API to access the database.

ResearchIndex also has a large, active system, which will allow recommender systems developers to ensure that their algorithms scale well to systems which demand quick response time while servicing thousands of requests per hour, over a database of hundreds of thousands of users and items.

Advanced Database Systems Raghav Karumur Spring 2011

researchindex a peek inside
ResearchIndex: A peek inside !

The basic entities in ResearchIndex: users, documents, citations, and citation groups.

Users : People who make queries to access documents and citations. ResearchIndex makes it easy to observe a user’s behavior, automatically generating & assigning user IDs and stores them in cookies.

Documents: (research papers) each is assigned a document ID (DID).

Citations: The actual text of citations extracted from the documents. Each citation receives a unique citation ID (CID).

Citation groups: relate individual citations that cite the same document, with each citation group receiving a group ID (GID).

User actions (interaction with a document or citation/issue a query) are called “events”. There are about 30 kinds of events; some of the events most likely to interest recommender systems developers are shown in Table 1.

Advanced Database Systems Raghav Karumur Spring 2011

researchindex a peek inside1
ResearchIndex: A peek inside !

Table1: A sample of user actions in ResearchIndex. Most actions involve a user interacting with a document or citation in the database.

Advanced Database Systems Raghav Karumur Spring 2011

researchindex a peek inside2
ResearchIndex: A peek inside !

REFEREE has no built-in notion of how important or interesting an event is.

Recommender system developers have to infer what an event signifies about a user’s preferences. This is also known as using implicit ratings, as opposed to explicit ratings, where the user effectively says “I like item xthis much”.

Implicit ratings are useful in ecommerce recommenders, as it is a burden on users to ask for explicit ratings.

ResearchIndex does provide a way for users to give a rating on a 1 to 5 scale!

Advanced Database Systems Raghav Karumur Spring 2011

researchindex built in recommenders
ResearchIndex: Built in recommenders

Sentence Overlap: recommends docs with significant text identical to current document.

Cited By: documents that site current document.

Active Bibliography: documents that make similar citations.

Users who viewed: documents that userbase has seen.

Co-citation: documents sited together with current document.

On Same site: documents found on same webpage as current document.

Advanced Database Systems Raghav Karumur Spring 2011

referee
REFEREE

click-through rates and download rates, purchase rates

Purposes:

  • To help recommender systems communicate with ResearchIndex.
  • To minimize the amount of work recommender system developers must do in order to implement and test working systems.
  • REFEREE allows anyone to quickly implement, and evaluate recommenders for documents in ResearchIndex.
  • Want to evaluate recommenders using: user behavior
  • A Recommender Framework Daemon (RFD) runs on the ResearchIndex site and fetches user events to the recommenders.
  • It also requests recommendations of the form “user x is looking at document y: what three documents might x want to look at next?”
  • The system generally issues between one and five recommendation requests per second, and usually requires recommenders to respond within 200 milliseconds.
  • Figure 2 shows the REFEREE architecture.

Advanced Database Systems Raghav Karumur Spring 2011

referee architecture
REFEREE: Architecture

Logs

recs

Requests to join, recs upon request

Engine

RFD

events

RFD Client

Broadcast events, request recs from specific engines

recs

Events,

rec requests

Research

Index

Figure 2: An overview of the REFEREE architecture. Recommender engine writers need only write the Engine portion, making use of the shaded components which REFEREE provides.

Advanced Database Systems Raghav Karumur Spring 2011

the process
The Process

In order to develop a REFEREE-based recommender, a researcher first contacts ResearchIndex for access to the framework, codebase, etc.

Then he develops a recommender & tests it locally against his local instance.

Once the recommender is ready, the researcher changes the client to point at the actual ResearchIndex site, and it then goes live.

The recommender can run on any machine, as long as the network latency plus computation time required to make recommendations is within the deadline which ResearchIndex imposes.

Advanced Database Systems Raghav Karumur Spring 2011

metrics and accuracy
Metrics and Accuracy

How accurate are the metrics? MAE : user ratings Vs system ratings

User ratings are often not precise: depend on user’s mood.

Also often they are not explicit; system must infer from user behavior.

So, do we need absolute prediction accuracy?

- In most cases, we don’t!

Metrics are there to measure whether user accepted a recommendation.

This paper distinguishes between user looking at a document’s detail page and a user actually downloading it.

Instead of Click-Through and Buy-Now, they have Click-Soon and Buy Soon.

Advanced Database Systems Raghav Karumur Spring 2011

what should can a recommender system do
What should(can) a recommender system do?

Neither of these important cases can be captured by evaluating recommenders using accuracy metrics on a static dataset!!

Help the user make a decision about what item to choose.

Actually change user behavior.

By suggesting a set of items, a recommender provides a user with a set of choices which the user might not otherwise have made.

influence which items users consume (and thus can rate) and might even influence the actual ratings users give (“the system recommended it, so it has to be good”).

static datasets of what use are they
Static Datasets: Of what use are they?

Provide a fixed and unchanging world. Then what use????

.

.

.

Valuable tool for evaluating recommender algorithms

Helpful in removing unsuitable algorithms

Increase effectiveness of online experiments.

In any case, ResearchIndex captures raw data: it logs user events, recommendation requests, and documents recommended.

Advanced Database Systems Raghav Karumur Spring 2011

pd live an example hybrid recommender
PD-Live : An example hybrid recommender

PD-Live : variant of Personality Diagnosis algorithm of Pennock.

Maintains an in-memory sparse-matrix representation of users, items and ratings

Remembers its state between runs

Computes most recommendations in under 200 milliseconds

The authors of this paper chose PD because they had a reasonably fast working offline version, and the algorithm has been shown to be more accurate than standard correlation methods on EachMovie and ResearchIndex data.

Advanced Database Systems Raghav Karumur Spring 2011

considering context in hybrid recommenders
Considering context in hybrid recommenders

A user viewing recipes should not be suggested jokes.

Most CF recommenders do not consider context explicitly, they draw recommendations from all the items they know about.

They get some implicit context from the fact that people tend to like similar types of items (e.g., if Dan likes mostly science fiction movies, a recommender will tend to match him up with other sci-fi fans and recommend science fiction rather than romances).

Issues with ResearchIndex:

Scalability: Computing a recommendation from among 500,000 items

Second, researchers typically have several areas of interest. Documents with the highest predicted scores may not be what the user is currently interested in.

Advanced Database Systems Raghav Karumur Spring 2011

two phase approach for determining context similarity
Two Phase Approach for determining context similarity

Citation links and other similarity measures form a directed graph with documents as nodes and similarity relationships as edges.

PD-Live does a BFS from the document a user is currently looking at, to select a candidate set of documents.

It then uses the PD algorithm to generate predictions for these documents and conducts a lottery biased toward higher-scoring documents to select which ones to recommend.

If the user is new, or no neighbors can be found who have rated a document, the system falls back to non-personalized average ratings for the documents.

If the document is new and no similarity information is available, the system chooses the set of recommendable documents randomly.

Advanced Database Systems Raghav Karumur Spring 2011

two phase approach advantages
Two Phase Approach: Advantages
  • Easy and fast to access and use.
  • Using content to select a set of plausible documents and then collaborative information to rank them is also a reasonable model of how people often make decisions: eliminate obviously wrong alternatives and then choose the best among them.
  • This approach fits well with the notion of relevance adopted by the IR community.
  • Their recommender starts with the impersonal, binary notion of relevance, then uses CF to include personal notions of relevance.

Advanced Database Systems Raghav Karumur Spring 2011

two phase approach drawbacks
Two Phase Approach: Drawbacks
  • Consumes a lot of memory.
  • It also reduces the number of documents considered by only recommending documents that have at least five ratings, since documents rated by only a few users would seldom be recommended anyway.
  • User’s session is updated only every 15 minutes.
  • PD-Live also makes use of only some user actions: those where the user is interacting with a document already stored in the system.

Advanced Database Systems Raghav Karumur Spring 2011

results and discussion
Results and Discussion

Four metrics produce the same rank ordering of the recommenders.

The best PD-Live-based recommender places in the middle of the pack, outperforming three of ResearchIndex’s built-in recommenders (On Same Site, Co-citation, and Users Who Viewed) while lagging behind three others (Sentence Overlap, Cited By, and Active Bibliography).

The percentages for click-thru do not add up to 100% because there are many other actions that users can take besides following a recommendation link.

The Sentence Overlap and Cited By recommenders do much better than the rest.

These recommenders often produce documents related to the current document and that are more recent (and often, most detailed).

Cited By helps researchers find newer/more papers related to area of interest.

Advanced Database Systems Raghav Karumur Spring 2011

results and discussion1
Results and Discussion
  • REFEREE uses metrics that evaluate how the recommender system affects user behavior.
  • PD-Live exploits both content-based (to limit the population of documents to recommend)and collaborative similarity measures (to order them).
  • PD-Live has great flexibility and corresponds well to the problem of recommending items to a user in context.
  • The prototype version makes reasonably good recommendations compared to several content-based recommenders, has good performance, and appears to have much potential for improvement.

Advanced Database Systems Raghav Karumur Spring 2011

future research
Future Research
  • REFEREE has the potential to stimulate research in at least three areas:
  • Recommender Systems that merge content and collaborative data
  • Systems that must make tradeoffs to sacrifice accuracy and elegance for speed and memory savings, and
  • Systems that recommend based on current user actions (context)

Advanced Database Systems Raghav Karumur Spring 2011

thank you
Thank You!

Questions?

Advanced Database Systems Raghav Karumur Spring 2011

ad