1 / 21

Bootstrapping

Bootstrapping. April 3 2007 William Cohen. Prehistory.

gustav
Download Presentation

Bootstrapping

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Bootstrapping April 3 2007 William Cohen

  2. Prehistory Karl Friedrich Hieronymus, Freiherr von Münchhausen (11 May1720 – 22 February1797) was a Germanbaron who in his youth was sent to serve as page to Anthony Ulrich II, Duke of Brunswick-Lüneburg and later joined the Russian military. He served until 1750, in particular taking part in two campaigns against the Turks. Returning home, Münchhausen supposedly told a number of outrageous tall tales about his adventures. The Baron was born in Bodenwerder and died there as well. According to the stories, as retold by others, the Baron's astounding feats included riding cannonballs, travelling to the Moon, and escaping from a swamp by pulling himself up byhis own hair. … In later versions he was using his own boot straps to pull himself out of the sea. [Wikipedia]

  3. Prehistory “Bob Wilson is desperately trying to finish his doctoral thesis and has locked himself in his room in a marathon attempt to do so. His typewriter jams, and as he unjams it he hears someone say "Don't bother, it's hogwash anyway." The thesis, in fact, deals with time travel. The interloper is a man who seems strangely familiar, and might be recognizable without the two-day growth of beard and the black eye. …” “In computing, bootstrapping refers to a process where a simple system activates another more complicated system that serves the same purpose. It is a solution to the Chicken-and-egg problem of starting a certain system without the system already functioning. The term is most often applied to the process of starting up a computer, in which a mechanism is needed to execute the software program that is responsible for executing software programs …” [Wikipedia]

  4. Some more recent history - 1 • Idea: write some specific patterns that indicate A is a kind of B: • … such NP as NP (“at such schools as CMU, students rarely need extensions”) • NP, NP, or other NP (“William, Carlos or other machine learning professors”) • NP including NP (“struggling teams including the Pirates”) • NP, especially NP (prestigious conferences, especially NIPS) [Coling 1992] Results: 8.6M words of Grolier’s encyclopedia  7067 pattern instances  152 relations Many were not in WordNet.

  5. Some history – 2a • Idea: exploit “pattern/relation duality”: • Start with some seed instances of (author,title) pairs (“Isaac Asimov”, “The Robots of Dawn”) • Look for occurrences of these pairs on the web. • Generate patterns that match the seeds. • - URLprefix, prefix, middle, suffix • Extract new (author, title) pairs that match the patterns. • Go to 2. [some workshop, 1998] Unlike Hearst, Brin learned the patterns; and learned very high-precision, easy-to-match patterns. Result: 24M web pages + 5 books  199 occurrences  3 patterns  4047 occurrences + 5M pages  3947 occurrences  105 patterns  … 15,257 books *with some manual tweaks

  6. Some history – 2b Patterns Occurrences Instances • Idea: exploit “pattern/relation duality”: • Start with some seed instances of (author,title) pairs (“Isaac Asimov”, “The Robots of Dawn”) • Look for occurrences of these pairs on the web. • Generate patterns that match the seeds. • - URLprefix, prefix, middle, suffix • Extract new (author, title) pairs that match the patterns. • Go to 2. Result: 24M web pages + 5 books  199 occurrences  3 patterns  4047 occurrences + 5M pages  3947 occurrences  105 patterns  … 15,257 books *with some manual tweaks

  7. Some history – 3 [COLT 98]

  8. Some history – 3b Instances/Occurrences Instances Patterns Patterns Occurrences How to filter out “bad” instances, occurrences, patterns?

  9. Bootstrapping Hearst ‘92 Deeper linguistic features, free text… BM’98 Learning, semi-supervised learning, dual feature spaces… Brin’98 Scalability, surface patterns, use of web crawlers…

  10. Bootstrapping Hearst ‘92 Deeper linguistic features, free text… Boosting-based co-train method using content & context features; context based on Collins’ parser; learn to classifythree types of NE Collins & Singer ‘99 BM’98 Learning, semi-supervised learning, dual feature spaces… Brin’98 Scalability, surface patterns, use of web crawlers…

  11. Bootstrapping Hearst ‘92 Deeper linguistic features, free text… Riloff & Jones ‘99 Hearst-like patterns, Brin-like bootstrapping (+ “meta-level” bootstrapping) on MUC data Collins & Singer ‘99 BM’98 Learning, semi-supervised learning, dual feature spaces… Brin’98 Scalability, surface patterns, use of web crawlers…

  12. Bootstrapping Hearst ‘92 Deeper linguistic features, free text… Riloff & Jones ‘99 Collins & Singer ‘99 BM’98 Learning, semi-supervised learning, dual feature spaces… EM like co-train method with context & content both defined by character-level tries Cucerzan & Yarowsky ‘99 Brin’98 Scalability, surface patterns, use of web crawlers…

  13. Bootstrapping Hearst ‘92 Deeper linguistic features, free text… Stevenson & Greenwood 2005 Riloff & Jones ‘99 … Collins & Singer ‘99 Rosenfeld and Feldman 2006 BM’98 Learning, semi-supervised learning, dual feature spaces… Etzioni et al 2005 … Cucerzan & Yarowsky ‘99 Brin’98 Scalability, surface patterns, use of web crawlers… De-emphasize duality, focus on distance between patterns.

  14. Stevenson & Greenwood Instances/Occurrences Patterns Patterns Pattern-pattern-from is semantic similarity (Wordnet) Flow from pattern-pattern depends on empirical similarity (i.e. overlapping occurrences in corpus)

  15. Bootstrapping Hearst ‘92 Deeper linguistic features, free text… Stevenson & Greenwood 2005 Riloff & Jones ‘99 … Collins & Singer ‘99 Rosenfeld and Feldman 2006 BM’98 Learning, semi-supervised learning, dual feature spaces… Etzioni et al 2005 … Cucerzan & Yarowsky ‘99 Brin’98 Scalability, surface patterns, use of web crawlers… Clever idea for learning relation patterns & strong experimental results

  16. Instances  Occurrences as before. Vary “positive” occurrences to get near-miss “negative” occurrences, using asymmetry, disjointness, etc. Learn patterns in a (moderately) expressive but easy-to-match language (NPs from OpenNLP). Rosenfeld & Feldman

  17. Know It All

  18. Architecture Set of predicates to consider + two names for each ~= [H92]

  19. Architecture

  20. Bootstrapping - 1 • Submit the queries & apply the rules  initial seeds. • Evaluate each seed with each discriminator U: e.g., compute PMI stats like: |hits(“city Boston”)| / |hits(“Boston”)| • Take the top seeds from each class and call them POSITIVE and use disjointness, etc to find NEGATIVE seeds. • Train a NaiveBayes classifier using thresholded U’s as features.

  21. Bootstrapping - 2

More Related