1 / 52

Distance Functions and IE - 4? William W. Cohen CALD Announcements

This announcement provides current statistics on unscheduled student talks, projects due date, and additional requirements. It also discusses different string distance metrics and their suitability for various problems, with a focus on the SecondString toolkit.

ebolden
Download Presentation

Distance Functions and IE - 4? William W. Cohen CALD Announcements

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Distance functions and IE – 4? William W. Cohen CALD

  2. Announcements • Current statistics: • days with unscheduled student talks: 6 • students with unscheduled student talks: 4 • Projects are due: 4/28 (last day of class) • Additional requirement: draft (for comments) no later than 4/21

  3. The data integration problem

  4. String distance metrics so far... • Term-based (e.g. TF/IDF as in WHIRL) • Distance depends on set of words contained in both s and t – so sensitive to spelling errors. • Usually weight words to account for “importance” • Fast comparison: O(n log n) for |s|+|t|=n • Edit-distance metrics • Distance is shortest sequence of edit commands that transform s to t. • No notion of word importance • More expensive: O(n2) • Other metrics • Jaro metric & variants • Monge-Elkan’s recursive string matching • etc? • Which metrics work best, for which problems?

  5. Jaro metric

  6. Winkler-Jaro metric

  7. String distance metrics so far... • Term-based (e.g. TF/IDF as in WHIRL) • Distance depends on set of words contained in both s and t – so sensitive to spelling errors. • Usually weight words to account for “importance” • Fast comparison: O(n log n) for |s|+|t|=n • Edit-distance metrics • Distance is shortest sequence of edit commands that transform s to t. • No notion of word importance • More expensive: O(n2) • Other metrics • Jaro metric & variants • Monge-Elkan’s recursive string matching • etc? • Which metrics work best, for which problems?

  8. So which metric should you use? SecondString (Cohen, Ravikumar, Fienberg): • Java toolkit of string-matching methods from AI, Statistics, IR and DB communities • Tools for evaluating performance on test data • Exploratory tool for adding, testing, combining string distances • e.g. SecondString implements a generic “Winkler rescorer” which can rescale any distance function with range of [0,1] • URL – http://secondstring.sourceforge.net • Distribution also includes several sample matching problems.

  9. SecondString distance functions • Edit-distance like: • Levenshtein – unit costs • untuned Smith-Waterman • Monge-Elkan (tuned Smith-Waterman) • Jaro and Jaro-Winkler

  10. Results - Edit Distances Monge-Elkan is the best on average....

  11. Edit distances

  12. SecondString distance functions • Term-based, for sets of terms S and T: • TFIDF distance • Jaccard distance: • Language models: construct PS and PT anduse

  13. SecondString distance functions • Term-based, for sets of terms S and T: • TFIDF distance • Jaccard distance • Jensen-Shannon distance • smoothing toward union of S,T reduces cost of disagreeing on common terms • unsmoothed PS, Dirichlet smoothing, Jelenik-Mercer • “Simplified Fellegi-Sunter”

  14. Results – Token Distances

  15. SecondString distance functions • Hybrid term-based & edit-distance based: • Monge-Elkan’s “recursive matching scheme”, segmenting strings at token boundaries (rather than separators like commas) • SoftTFIDF • Like TFIDF but consider not just tokens in both S and T, but tokens in S “close to” something in T (“close to” relative to some distance metric) • Downweight close tokens slightly

  16. Results – Hybrid distances

  17. Results - Overall

  18. Prospective test on two clustering tasks

  19. An anomolous dataset

  20. An anomalous dataset: census

  21. An anomalous dataset: census Why?

  22. Other results with SecondString • Distance functions over structured data records (first name, last name, street, house number) • Learning to combine distance functions • Unsupervised/semi-supervised training for distance functions over structured data

  23. Combining Information Extraction and Similarity Computations 2) Krauthammer et al 1) Bunescu et al

  24. Experiments • Hand-tagged 50 abstracts for gene/protein entities (pre-selected to be about human genes) • Collected dictionary of 40,000+ protein names from on-line sources • not complete • example matching is not sufficient • Approach: use hand-coded heuristics to propose likely generalizations of existing dictionary entries. • not hand-coded or off-the-shelf similarity metrics

  25. Example name generalizations

  26. Basic idea behind the algorithm original dictionary carefully-tuned heuristics (aka hacks) similar (but not identical process) applied to word n-grams from text to do IE: extract if n-gram -> CD

  27. Example: canonicalizing “short names” (different procedure for “full names” and “one-word” names)

  28. Example: canonicalizing “short names” (different procedure for “full names” and “one-word” names) NF<n> Nf<n> NF => CD (from <x><n>) NF-25 in OD NF in CD? (<x><g><l>) NF<n> Recognize: “... NF-kappa B...” NF<g><l>

  29. Results • Why is precision less than 100%? • When should you use “similarity by normalization”? • Could a simpler algorithm do as well? • Is there overfitting? (50 abstracts, <750 proteins)

  30. ...

  31. Combining Information Extraction and Similarity Computations 2) Krauthammer et al 1) Bunescu et al

  32. Background • Common task in proteomics/genomics: • look for (soft) matches to a query sequence in a large “database” of sequences. • want to find subsequences (genes) that are highly similar (and hence probably related) • want to ignore “accidental” matches • possible technique is Smith-Waterman (local alignment) • want char-char “reward” for alignment to reflect confidence that the alignment is not due to chance

  33. Background • Common task in proteomics/genomics: • look for (soft) matches to a query sequence in a large “database” of sequences. • want to find subsequences (genes) that are highly similar (and hence probably related) • want to ignore “accidental” matches • possible technique is Smith-Waterman (local alignment) • want char-char “reward” for alignment to reflect confidence that the alignment is not due to chance

  34. c o h e n d o r f m 0 0 0 0 0 0 0 0 0 c 1 0 0 0 0 0 0 0 0 c 0 0 0 0 0 0 0 0 0 o 0 2 1 0 0 0 2 1 0 h 0 1 4 3 2 1 1 1 0 n 0 0 3 3 5 4 3 2 1 s 0 0 2 2 4 4 3 2 1 k 0 0 1 1 3 3 3 2 1 i 0 0 0 0 2 2 2 2 1 dist=5 Smith-Waterman distance

  35. In general “peaks” in the matrix scores indicate highly similar substrings.

  36. Background • Common task in proteomics/genomics: • look for (soft) matches to a query sequence in a large “database” of sequences. • possible technique is Smith-Waterman (local alignment) • want char-char “reward” for alignment to reflect confidence that the alignment is not due to chance • based on substitutability theory for amino acids • doesn’t scale well • BLAST and FASTA: fast approximate S-W

  37. BLAST/FASTA ideas • Find all char n-grams (“words”) in the query string. • FASTA: • Use inverted indices to find out where these words appear in the DB sequence • Use S-W only near DB sections that contain some of these words

  38. BLAST/FASTA ideas • Find all char n-grams (“words”) in the query string. • BLAST: • Generate variations of these words by looking for changes that would lead to strong similarities • Discard “low IDF” words (where accidental matches are likely) • Use expanded set of n-grams to focus search

  39. query string words and expansions

  40. BLAST/FASTA ideas • Find all char n-grams (“words”) in the query string. • BLAST: • Generate variations of these words by looking for changes that would lead to strong similarities • Discard “low IDF” words (where accidental matches are likely) • Use expanded set of n-grams to focus search • The BLAST program: • Widely used, • Fast implementation, • Supports asking multiple queries against a database at once... • Can one use it find soft matches of protein names (from a dictionary) in text?

  41. Protein database Query strings Proposed alignment (query->database) Query algorithm: BLAST Biomedical paper Protein name dictionary Extracted protein name (dict. entry->text) IE system: dictionaries+BLAST (optimized for this problem) Basic idea:

  42. 1) Mapping text to DNA sequences(Q: what sort of char similarity is this?)

  43. 2) Optimizing blast • Split protein-name database into several parts (for short, medium-length, long protein names) • Require space chars before and after “short” protein names. • Manually search (grid search?) for better settings for certain key parameters for each protein-name subdatabase • With what data? • Evaluate on one review article, 1162 protein names • inter-annotator agreement not great (70-85%)

  44. 2) Optimizing blast

  45. 2) Optimizing blast

More Related