1 / 30

Graph-based Analysis of Semantic Drift in Espresso-like Bootstrapping Algorithms

Mamoru Komachi † , Taku Kudo ‡ , Masashi Shimbo † and Yuji Matsumoto † † Nara Institute of Science and Technology, Japan ‡ Google Inc. Graph-based Analysis of Semantic Drift in Espresso-like Bootstrapping Algorithms. EMNLP 2008. Bootstrapping and semantic drift.

luther
Download Presentation

Graph-based Analysis of Semantic Drift in Espresso-like Bootstrapping Algorithms

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Mamoru Komachi†, Taku Kudo‡, Masashi Shimbo† and Yuji Matsumoto† †Nara Institute of Science and Technology, Japan ‡Google Inc. Graph-based Analysis of Semantic Drift in Espresso-like Bootstrapping Algorithms EMNLP 2008

  2. Bootstrapping and semantic drift • Bootstrapping grows small amount of seed instances Seed instance Co-occurrence pattern Extracted instance iPhone MacBook Air Buy # at Apple Store iPod car iPod # for sale house Generic patterns = Patterns which co-occur with many (irrelevant) instances

  3. Main contributions of this work • Suggest a parallel between semantic drift in Espresso-like bootstrapping and topic drift in HITS (Kleinberg, 1999) • Solve semantic drift by graph kernels used in link analysis community 3

  4. Espresso Algorithm [Pantel and Pennacchiotti, 2006] • Repeat • Pattern extraction • Pattern ranking • Pattern selection • Instance extraction • Instance ranking • Instance selection • Until a stopping criterion is met

  5. Espresso uses pattern-instance matrix Afor ranking patterns and instances |P|×|I|-dimensional matrix holding the (normalized) pointwise mutual information (pmi) between patterns and instances instance indices pattern indices [A]p,i= pmi(p,i) / maxp,i pmi(p,i)

  6. Pattern/instance ranking in Espresso p = pattern score vector i = instance score vector A = pattern-instance matrix Reliable instances are supported by reliable patterns, and vice versa ... pattern ranking ... instance ranking |P| = number of patterns |I| = number of instances normalization factors to keep score vectors not too large

  7. Espresso Algorithm [Pantel and Pennacchiotti, 2006] • Repeat • Pattern extraction • Pattern ranking • Pattern selection • Instance extraction • Instance ranking • Instance selection • Until a stopping criterion is met For graph-theoretic analysis, we will introduce 3 simplifications to Espresso

  8. First simplification • Compute the pattern-instance matrix • Repeat • Pattern extraction • Pattern ranking • Pattern selection • Instance extraction • Instance ranking • Instance selection • Until a stopping criterion is met Simplification 1 Remove pattern/instance extraction steps Instead, pre-compute all patterns and instances once in the beginning of the algorithm

  9. Second simplification • Compute the pattern-instance matrix • Repeat • Pattern ranking • Pattern selection • Instance ranking • Instance selection • Until a stopping criterion is met Simplification 2 Remove pattern/instance selection steps which retain only highest scoring k patterns / m instances for the next iteration i.e., reset the scores of other items to 0 Instead, retain scores of all patterns and instances

  10. Third simplification • Compute the pattern-instance matrix • Repeat • Pattern ranking • Instance ranking • Until a stopping criterion is met Simplification 3 No early stopping i.e., run until convergence Until score vectors p and i converge

  11. Simplified Espresso Input • Initial score vector of seed instances • Pattern-instance co-occurrence matrix A Main loop Repeat Until i and p converge Output Instance and pattern score vectors i and p ... pattern ranking ... instance ranking

  12. Simplified Espresso is HITS Simplified Espresso = HITS in a bipartite graph whose adjacency matrix is A Problem • No matter which seed you start with, the same instance is always ranked topmost • Semantic drift (also called topic drift in HITS) The ranking vector i tends to the principal eigenvector of ATA as the iteration proceeds regardless of the seed instances!

  13. How about the original Espresso? Original Espresso has heuristics not present in Simplified Espresso • Early stopping • Pattern and instance selection Do these heuristics really help reduce semantic drift?

  14. Experiments on semantic drift Does the heuristics in original Espresso help reduce drift?

  15. Word sense disambiguation task of Senseval-3 English Lexical Sample Predict the sense of “bank” … the financial benefits of the bank (finance) 's employee package ( cheap mortgages and pensions, etc ) , bring this up to … Training instances are annotated with their sense In that same year I was posted to South Shields on the south bank (bank of the river) of the River Tyne and quickly became aware that I had an enormous burden Predict the sense of target word in the test set Possibly aligned to water a sort of bank(???) by a rushing river.

  16. Word sense disambiguation by Original Espresso Seed instance = the instance to predict its sense System output = k-nearest neighbor(k=3) Heuristics of Espresso • Pattern and instance selection • # of patterns to retain p=20 (increase p by 1 on each iteration) • # of instance to retain m=100 (increase m by 100 on each iteration) • Early stopping Seed instance

  17. Convergence process of Espresso Heuristics in Espresso helps reducing semantic drift (However, early stopping is required for optimal performance) Original Espresso Simplified Espresso Most frequent sense (baseline) Semantic drift occurs (always outputs the most frequent sense) Output the most frequent sense regardless of input

  18. Learning curve of Original Espresso:per-sense breakdown Most frequent sense # of most frequent sense predictions increases Recall for infrequent senses worsens even with original Espresso Other senses

  19. Summary: Espresso and semantic drift Semantic drift happens because • Espresso is designed like HITS • HITS gives the same ranking list regardless of seeds Some heuristics reduce semantic drift • Early stopping is crucial for optimal performance Still, these heuristics require • many parameters to be calibrated • but calibration is difficult

  20. Main contributions of this work • Suggest a parallel between semantic drift in Espresso-like bootstrapping and topic drift in HITS (Kleinberg, 1999) • Solve semantic drift by graph kernels used in link analysis community 20

  21. Q. What caused drift in Espresso? A. Espresso's resemblance to HITS HITS is an importance computation method (gives a single ranking list for any seeds) Why not use a method for another type of link analysis measure - which takes seeds into account? "relatedness" measure (it gives different rankings for different seeds)

  22. The regularized Laplacian kernel • A relatedness measure • Takes higher-order relations into account • Has only one parameter A: adjacency matrix of the graph D: (diagonal) degree matrix Graph Laplacian Regularized Laplacian matrix β: parameter Each column of Rβ gives the rankings relative to a node

  23. Evaluation of regularized Laplacian Comparison to (Agirre et al. 2006)

  24. WSD on all nouns in Senseval-3 Espresso needs optimal stopping to achieve an equivalent performance Outperforms other graph-based methods

  25. Conclusions • Semantic drift in Espresso is a parallel form of topic drift in HITS • The regularized Laplacian reduces semantic drift • inherently a relatedness measure (importance measure)

  26. Future work • Investigate if a similar analysis is applicable to a wider class of bootstrapping algorithms (including co-training) • Try other popular tasks of bootstrapping such as named entity extraction • Selection of seed instances matters

  27. Label prediction of “bank” (Fmeasure) Espresso suffers from semantic drift (unless stopped at optimal stage) The regularized Laplacian keeps high recall for infrequent senses

  28. Regularized Laplacian is mostly stable across a parameter

  29. Pattern/instance ranking in Espresso Score for pattern p Score for instance i p: pattern i: instance P: set of patterns I: set of instances pmi: pointwise mutual information max pmi: max of pmi in all the patterns and instances 30

More Related