Passage Retrieval for Information Extraction using Distant Supervision

1. Wei Xu, Ralph Grishman, Le Zhao (CMU) xuwei@cs.nyu.edu New York University Novmember 24, 2011 Passage Retrieval for Information Extraction using Distant Supervision

2. About Me • New York University (since 2007) • Tsinghua University • Microsoft Research • Amazon.com • ETS • France Telecom • Hong Kong Polytechnic University • Information Retrieval • Information Extraction • Spelling Correction • Summarization

3. Task • Knowledge Base Population (KBP) slot filling task (Ji et al. 2010) at TAC 2010: • find attributes of a specified person or organization from a large corpus • person:employee_of • organization:headquarters • Etc.

4. Motivation • No time to perform in-depth extraction • Little annotated data • Blend QA and IR with IE

5. Motivation (cont.) • Like QA, begin with passage retrieval to find passages relevant to the person or organization and its attributes • Unlike QA, there is a fixed set of relations which allows more specialized learning methods

6. Previous Work • Passage Retrieval • Hand-selected keywords (Surdeanu et al. 2010, Nguye et al. 2010) • Expected named entity types (Chrupala et al. 2010) • Relevance based on the similarity between terms (Fang et al. 2010)

7. Previous Work (cont.) • Distant Supervision • Use noisy training data generated automatically from a related, but different, type of dataset to solve problem on another type of data • Rich-featured logistic regression model (Mintz et al. 2009)

8. Approach • Use Lemur to retrieve passages containing the given entity • Re-rank the retrieved passages preferring those that contain: • the target entity or pronouns • named entity of the expected type • relevant terms of the target relation

9. Relevant Terms • Baseline: hand selected keywords + 635 common title words (person-employer) • IKFB: weighted keyword list learned by distant supervision

10. Learning Indicative Keywords by Distant Supervision FREEBASE Steve Jobs – Apple Inc. WIKIPEDIA – Steve Jobs * “Heis co-founder, chairman, and former chief executive officer of Apple Inc.” * “On August 24, 2011, Jobs announced his resignation from his role as Apple's CEO.” * “In March 1998, to concentrate Apple's efforts on returning to profitability, Jobs terminated a number of projects, such as Newton, Cyberdog, and OpenDoc.” * “Even though Jobs earned only $1 a year as CEO of Apple, he holds 5.426 million Apple shares, as well as 138 million shares in Disney (which he had received in exchange for Disney's acquisition of Pixar).”

11. Learning Indicative Keywords by Distant Supervision (cont.)

12. Test Data • KBP 2010 training data • human annotated, incomplete keys • 1.8 million documents • 67 person entities • 54 instances of person-employer

13. Data for Distant Supervision • 10702 person:employee_of in Freebase • 4497 are found in Wikipedia articles • 6574 positive 93756 negative sentences • 2436 indicative keywords

14. Evaluation • Coverage: the proportion of relation instances that can be found within top n retrieved passages • For 31.5% of relation instances, only one passage had to be examined using IKFB, 16.7% for baseline, 9.3% for Lemur

15. Evaluation • Redundancy: the average number of passages within the top n ranks that contain a correct answer • 18.7% of the top 10 ranked passages by IKFB contain correct answers, 5.93% for baseline

16. Sample Results • Employer of ‘John Dewey’ – 1stRanked Passage • An institution that sees itself as an uncon-ventional alternative to other colleges, the New School was founded in 1919 by a group of pro-fessors, including the philosopher and education reformer John Dewey, who had resigned in pro-test from Columbia. They could not abide by a stance taken by Columbia's president at the time, Nicholas Murray Butler, that faculty members had to support America's entry into World War I.