1 / 22

Text-based User-kNN: Measuring user similarity based on text reviews

This paper explores a text-based user-kNN approach for measuring user similarity by analyzing text reviews, improving the performance of collaborative filtering algorithms and making accurate recommendations.

gnolte
Download Presentation

Text-based User-kNN: Measuring user similarity based on text reviews

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Text-based User-kNN: Measuring user similarity based on text reviews Maria Terzi, Matthew Rowe, Maria-Angela Ferrario, Jon Whittle School of Computing & Communications, Infolab21 Lancaster University, United Kingdom

  2. Recommender systems Information filtering systems: • seek to predict the interest of a user for an item or a social element that had not yet considered • at the right time and in the right context • by measuring similarity between users and items/social elements Text-based user-kNN: Measuring user similarity based on text reviews 1/19

  3. User-based k-Nearest Neighbors Collaborative Filtering Matches similar users based on the similarity of their ratings to predict ratings and make recommendations to a target user. How it works: a. weight all users with respect to similarity with the target user b. select k users that have the highest similarity (neighbors) c. compute a prediction value from a weighted average of the selected neighbors. Text-based user-kNN: Measuring user similarity based on text reviews 2/19

  4. Similarity reflection problems Ratings cannot fully reflect the similarity between users… • ratings do not capture the rationale behind a user’s rating • there is a high probability (p=0.8) that two ratings with the same value on the same item will be given for different reasons. ..new sources of knowledge are required! Text-based user-kNN: Measuring user similarity based on text reviews 3/19

  5. Importance of Online User Text Reviews Why do they share? 46% feel they can be brutally honest on the Internet. 38% aim to influence others when they express their preferences online (Harris Poll, April 2010) 34% have turned to social media to air their feelings about a company. 26% to express dissatisfaction, 23% to share companies or products they like. (Harris Poll, April 2010) 90% of customers say Buying Decisions are influenced by online reviews (Dimensional Research, April 2013) 97% who made a purchase based on an online review found the review to be accurate (Comscore/The Kelsey Group, Oct. 2007) Users trust more text reviews than ratings Make an ideal source of knowledge for recommender systems and measuring similarity between users. .. fake, computationally expensive, noisy and hard to quantify Text-based user-kNN: Measuring user similarity based on text reviews 4/19

  6. How existing work used text-reviews in RecSys Matrix Factorization CF approaches use text reviews to define a regularizer: opinion score: feature extraction and sentiment analysis (Pero et al., 2013) sentiment score (Singh et al., 2011) review quality score (Raghavan et al., 2011) ratings & features (MacAuley & Leskovec, 2013) ..Not focused on improving the performance of neighborhood based models, such as user-kNN Using the sentiment of text reviews instead of ratings in user-kNN similar to a rating, the sentiment says how much a person liked an item.. but misses the reason why ..The reasons behind a user’s rating remain unexploited Text-based user-kNN: Measuring user similarity based on text reviews 5/19

  7. How existing work used text-reviews in RecSys Use text reviews to construct user preference profiles for CF: profiles : set of item features (such as “plot”) extracted from text reviews used to match user profiles (Chen and Wang, 2013) ..not focused on predicting ratings or reviews similar to the profile when making recommendations in Topic Profile CF (TPCF) (Musat et al.,2013) ..TPCF does not form neighborhoods of similar users “the overall number of opinions regarding a certain item feature reveals how important that feature is to the item” .. Do not distinguish user preferences across domains Text-based user-kNN: Measuring user similarity based on text reviews 6/19

  8. Contributions A text-based user-kNN approach An extensive evaluation: a) investigation of the performance of various text similarity measures in the user-kNN approach. -> significant improvement of semantic similarity measures over text similarity measures b) a comparison of text-based user-kNN with a range of rating based approaches in a ratings prediction task over two datasets -> consistently higher accuracy of text-based user-kNN Novelty of Text-based User-kNN Text-based user-kNN calculates directly the similarity of text reviews to measure the similarity between users, form neighborhoods of similar users and predict ratings. Text-based user-kNN: Measuring user similarity based on text reviews 7/19

  9. How text-based User-kNN works Text-based user-kNN measures the similarity between users by applying a text similarity measure over the content of the reviews for each of their co-reviewed items, instead of statistical measures on ratings. • set of users u∈U, a set of items i∈I and a set of quadruples D, (u,i,r,c)∈D, u,v: user, i:item, r: rating, c: content of text review • UserItemAverage baseline Text-based user-kNN: Measuring user similarity based on text reviews 8/19

  10. Short text Similarity measures Identify “similar reviews” : reviews that use semantically similar wordings to review an item. word overlap: produces a similarity score based on the number of words that occur in both segments. ..cannot identify semantic similarity WordNet: structure of synsets; sets of terms with synonymous meanings. Why WordNet? • WordNet offers two ways to measure similarity between two terms • WordNet based measures had the higher correlation with human users & outperformed LSA in paraphrase detection tasks. • Does not require to build a semantic space like LSA • Does not require a corpus like LSA, LDA Text-based user-kNN: Measuring user similarity based on text reviews 9/19

  11. Short text Similarity measures based on the path length • path: count of links between two terms in the taxonomy. (path of synonymous=0) • Leacock & Chodorow, Wu & Palmer based on the information content (IC) • is a measure of specificity based on the frequency counts of a word in a WordNet tagged corpus i.e high IC -> mouse, lower IC-> animal • Resnik, Lin, Jiang & Conrath All similarity measures are publicly provided by Pedersen et al, 2004. 10/19

  12. Short text Similarity measures Great storyline! ...very interesting plot All similarity measures are publicly provided by Pedersen et al, 2004. 11/19

  13. Short text Similarity measures Great storyline! ...very interesting plot • Remove Stop Words • Reviews similarity = Average similarity between all words All similarity measures are publicly provided by Pedersen et al, 2004. 11/19

  14. Experimental Setup - Datasets MyMediaLite 3.07 Datasets: {itemid, userid, score, text, timestamp} RottenTomatoes: Critics reviews, collected using the API ( 0.85 cosine similarity with standard users) all reviews of the Top-100 movies for the years 2001 to 2010 normalized to 0-to-1 scale AudioCDs - Amazon Product Reviews: (Jindal and Liu, 2008) used by related work (Pero et al., 2013, Raghavan et al.,2013) Three-way dataset splitting method: 64.64% for training, 1.36% for validation, 1.36% for 25 test sets • time ordering is preserved • allows for significance testing Text-based user-kNN: Measuring user similarity based on text reviews 12/19

  15. Dataset splitting method Three-way dataset splitting method: 64.64% for training, 1.36% for validation, 1.36% for 25 test sets • time ordering is preserved • allows for significance testing • Netflix Prize : training 95.9%, validation 1.36%, testing 1.36% .. no significance testing! Why significance testing is important? • due to the marginal increases in performance often performed in the literature • in assessing for the chance involvement • to be more confident in any improvement we find Why preserving time ordering? • resembles, most closely, the situation of a recsys in a real system • Cross Validation (CV) approaches introduce bias in a model by training on future results. Text-based user-kNN: Measuring user similarity based on text reviews 13/19

  16. Experimental Setup - Metrics We evaluate the approaches in a ratings prediction task Measure: Root Mean Square Error (RMSE) between the actual ratings and the predictions averaged over the 25 test-folds Significance testing using the Sign Test Text-based user-kNN: Measuring user similarity based on text reviews 14/19

  17. Experimental Setup - Ratings based approaches Modified ratings-based approaches are usually compared: to : the ratings equivalent (Pero et al.,2013) a non-personalized baseline (Chen et al.,2013, Musat et al., 2013) Ratings-based approaches used in our experiment: • Non-personalised Baseline: User Item Average • Memory-based Neighborhood algorithms : • user-kNN with Pearson and Cosine Correlation Coefficients Similarity • item-kNN with Pearson and Cosine Correlation Coefficients Similarity • Matrix Factorization Methods: • Singular Value Decomposition Plus Plus (SVD++) : SVD + asymmetric SVD • Biased Matrix Factorization (BMF) : MF + user and item biases Text-based user-kNN: Measuring user similarity based on text reviews 15/19

  18. Training the text-based user-kNN approaches Text-based user-kNN approaches perform better when • using the most proximate user in terms of sharing similar views on items (k=1) • when using a weighted average of ratings of a large amount of users (k>=200). .. resembling a real life scenario Text-based user-kNN: Measuring user similarity based on text reviews 16/19

  19. Results • Text-based user-kNN approaches perform consistently and significantly (p <0.0001) better than the rating based approaches • Semantic similarity measures using IC (Lin, Resnik, J&C) perform better than path-based approaches Mean RMSE over the 25 test folds (lower is better) Even a small improvement in a rating prediction error can affect the ordering of items and have significant impact on the quality of the top few presented recommendations. (Koren, 2008) Text-based user-kNN: Measuring user similarity based on text reviews 17/19

  20. Conclusions • Measuring the similarity between users based on text reviews can help to overcome similarity reflection problems in user-kNN. • Text-based user-kNN performs significantly better than the rating-based approaches used in this experiment. • Semantic measures (based on WordNet) tend to perform better than simple lexical matching measures. Particularly those using the IC. Text-based user-kNN: Measuring user similarity based on text reviews 18/19

  21. Future Work Provide evidence of how significant the results are to the users • text-based user-k vs other text-based approaches in an items prediction task (top-N) / user study How to enhance the measurement of similarity… • combinations of text, ratings and sentiment analysis • identify hidden similarity in text reviews using Open Linked Data Text-based user-kNN: Measuring user similarity based on text reviews 19/19

  22. Thank You contact details: m.terzi@lancaster.ac.uk Questions ?

More Related