Collaborative Filtering & Recommender Systems

1. Junaid Alam Arish Joyo Imran Allawala Anum Mazhar Collaborative Filtering & Recommender Systems

2. How Often Do We See The Following

3. Youtube.com

4. Download.cnet.com

5. Amazon.com

6. The Definition • Collaborative filtering (CF) is the process of filtering for information or patterns using techniques involving collaboration among multiple agents, viewpoints, data sources, etc. “The Collaborative Filtering system is without a doubt the lifeblood of the social web”.

7. The Idea • “People who viewed/liked/bought X also viewed/liked/bought Y” • Personal tastes are correlated: • If Alice and Bob both like X and Alice likes Y then Bob is more likely to like Y • especially (perhaps) if Bob knows Alice

8. The Underlying Principle “We all have some things in common amongst us and we tend to agree on similar choices or recommendations.” We generally tend to buy: • Things recommended by people we agree with • Products similar to what we already like.

9. It is a method of making automatic predictions (filtering) about the interests of a user by collecting taste information from many users (collaborating)

10. Example • A collaborative filtering system for television tastes could make predictions about which television show a user should like given a partial list of that user's tastes (likes or dislikes). • These predictions are specific to the user, but use information gleaned from many users.

11. Collaborative filtering (CF) is a common Web technique for generating personalized recommendations. Examples of its use include  • Amazon,  • iTunes,  • Netflix, • LastFM,  • StumbleUpon, • Delicious.

12. Basic Principles There are two basic principles involved in Collaborative Filtering: • The Wisdom of Crowds • Law of Large Numbers

13. The Wisdom of Crowds  • The principle suggests that: As communities grow, not only does a large community make better decisions but the larger a community gets, the better its decisions will be. • Therefore, we can create a Collaboratively Filtered newspaper, television channel, radio station, etc. which would be better (for the community) rather than any other arbitrarily selected medium. • Digg, YouTube, and Last.fm are doing this.

14. Law of Large Numbers • This principle suggests that: In any large community, with enough data on individual participants and on how the individual participants collaborate/correlate with each other, we can make predictions about what these users will like in the future based on what their tastes have been in the past. • Assumption: • People's interests, preferences, and ideologies don't change too drastically over time.

15. The need of Collaborative Filtering: • If you need to choose between a variety of options with which you do not have any experience, you will often rely on the opinions of others who do have such experience. • However, when there are thousands or millions of options, like in the Web, it becomes practically impossible for an individual to locate reliable experts that can give advice about each of the options.

16. The Mechanism • Uses a collective method of recommendation (problem becomes more manageable). • Determines an "average opinion" for the group. • You would rather like to hear the opinions of those people who have interests similar to your own. • However, it ignores your particular interests, which may be different from those of the "average person".

17. The basic mechanism behind collaborative filtering systems is the following: • A large group of people's preferences are registered • Using a similarity metric, a subgroup of people is selected whose preferences are similar to the preferences of the person who seeks advice • A weighted average of the preferences for that subgroup is calculated • The resulting preference function is used to recommend options on which the advice-seeker has expressed no personal opinion as yet.

18. How Netflix knows what you like? • Video

19. Types Of CF systems: Collaborative filtering systems have many forms, but many common systems can be reduced to two steps: • User-based collaborative filtering • Logic-based collaborative filtering

20. User-based collaborative filtering • Step 1: Look for users who share the same rating patterns with the active user (the user whom the prediction is for). • Step 2:Use the ratings from those like-minded users found in step 1 to calculate a prediction for the active user • A specific application of this is the user-based Nearest Neighbor algorithm.

21. Logic-based collaborative filtering. This form of CF is based on implicit observations of normal user behavior (as opposed to the artificial behavior imposed by a rating task). • Step 1: You observe what a user has done together with what all users have done. • Step 2: Use that data to predict the user's behavior in the future. • Step 3: These predictions then have to be filtered through business logic to determine how these predictions might affect what a business system ought to do (e.g. not suggest things that you own already)

22. User-data-profiling: • (CF) system collects information on what kind of content and commentary you like and dislike, and based on your submission and voting habits, it does user-data-profiling. • This user profile helps the site recommend content that has been submitted by users you generally agree with and find interesting, as well as topics that you usually vote up and tend to comment on. • The system finds the content and deliver it to you rather than it requiring you to scout for it. • Furthermore, the more you use the recommendation system and vote up or down, the better it becomes with its recommendations.

23. Recommender System • A recommender system suggests the items expected to be preferred by the users. • Recommender systems use collaborative filtering (CF) to recommend items by summarizing the preferences of people who have tendencies similar to the user preference. Mechanism: • Traditional CF algorithms adopted the Semantic Differential (SD) method, in which preferences are measured using an n-point-scale on which extremes are represented by antonyms. • New CF algorithms adopt the ranking method. In the ranking method, the preferences are represented by orders, which are sorted item sequences according to the users' preferences.

24. Related Concepts • Folksonomy is a system of classification derived from the practice and method of collaboratively creating and managing tags to annotate and categorizecontent; this practice is also known as collaborative tagging, social classification, social indexing, and social tagging. • Social bookmarking is a method for Internet users to organize, store, manage and search for bookmarks of resources online. Unlike file sharing, the resources themselves aren't shared, merely bookmarks that reference them

25. Do Social Media tools constrict or expand us? • Are all these social media tools (blogs, tags, collaborative filtering, social networking sites, etc.) harming us as a society because by their very nature they help us find people and content that reinforces what we already believe or like, rather than introducing us to new and unexpected ideas ???

26. If all the music you ever purchased were related to what you currently like, how would you ever be exposed to amazingly different artists?  • Do You care that You are not reading conservative blogs also? Do You miss important ideas because your reading is limited?

27. Our opinion: • CF systems expand us far more. • Forums focused on a particular narrow topic inevitably contain discussions that are far-reaching and broadening. • Tags bring together certain items but also lead to unexpected connections. • And of course the best source of new ideas that expand us and challenge us is other people. • Friends and relative strangers that we are connected to via Twitter, Facebook, YouTube, and other sites generally happen because of some kind of connection we have with them: work, interest area, belief, social connection. But of course those relationships are never confined to what we share in common. The joy of social networking is the exposure to differences between us, to new ideas and knowledge that expand our minds.