1 / 42

Type Systems, Interoperability and Database Population

Type Systems, Interoperability and Database Population. Eric Nyberg, CMU Shilpa Arora, CMU Lance Ramshaw, BBN. Outline. Annotation sample analysis emergent type systems ongoing issues / clarification questions Data interoperability Database population CMU’s Annotations DB OntoNotes

robbin
Download Presentation

Type Systems, Interoperability and Database Population

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Type Systems,Interoperabilityand Database Population Eric Nyberg, CMUShilpa Arora, CMULance Ramshaw, BBN

  2. Outline • Annotation sample analysis • emergent type systems • ongoing issues / clarification questions • Data interoperability • Database population • CMU’s Annotations DB • OntoNotes • Possible architecture for interoperability withUIMA annotators • Issues for Discussion

  3. Task • Analyze sample outputs from different annotation groups • Formalize annotation type system (UML object model) for each sample • Generate clarification questions • Work toward a unified type system • Work toward interoperability architecture In progress, not finished Not started

  4. For each annotation sample: • Overview of what we received • Brief example annotation • Type system analysis • Issues / Questions

  5. Whats in the bin ? 5

  6. CMU/Columbia Belief Annotation • Annotation Manual: • Davis et. al., “Annotating belief in Communication: Manual” • Annotation Units: Propositions identified by PropBank and NomBank 6

  7. CMU/Columbia CMU Belief Annotation • Three categories: • Committed belief: Belief expressed in utterance • Can be a proposition about present or future • E.g. (1) I know Mark and Sandra have eloped. (2) The sun will rise again. (Future) • Non-committed belief: Not a strong belief • Can be a proposition about present or future • E.g. (1) Mark and Sandra may have eloped. (2) John may return tomorrow. • Not application: Not a belief • E.g. (1) I wish Mark and Sandra would finally elope. 7

  8. CMU/Columbia Belief Annotation • Five Classes: • Committed Belief • Committed Belief Future • Non-Committed Belief • Non-Committed Belief Future • Not Applicable 8

  9. CMU/Columbia Belief Annotation:Type System (1) 9

  10. CMU/Columbia Belief Annotation:Type System (2) 10

  11. CMU/Columbia Belief Annotation: Type System (3) 11

  12. Follow up questions • Extensions: • What extensions do we expect to the annotation scheme? • How best we can tailor the type system towards expected future changes • Requirements from application domain? • Do we have a set of requirements from the application side? 12

  13. Ed Hovy’s group • Annotations: • Annotated with OntoNotes for Noun senses • 205 nouns, one file for each noun, sense + location in files for each noun is stored • Sample annotations: • eng/AFGP-2002-600175-Trans.txt 427 4 position-n@0.0 3 Mon Dec 3 02:31:27 2007 • eng/AFGP-2002-602187-Trans.txt 25 6 position-n@0.0 2 Mon Dec 3 02:31:27 2007 • Noun="position", sense=3; file= AFGP-2002-600175-Trans.txt, position = “427 4” • Noun="position", sense=3; file=AFGP-2002-602187-Trans.txt, position=“25 6” 13

  14. TypeSystem (Ed Hovy et. al. Annotation) 14

  15. BBN BBN TTO-3 Temporal Ordering Annotation BBN Name Annotations: named entities – org, date, per etc BBN-Coref-Annotation: entity (with type) and entity mentions etc BBN-complex-coref-annotation 15

  16. Temporal Relationship Assignment ID TT TP TR 11/28 1 DS 2 A Arrived 2 EP 0 B yesterday 3 DS 2 C told 4 SP 2 B Visiting 5 EUN 4 A left 6 EP 4 A Return 7 EF 2 A Monday 8 DS 7 C is 9 BC 0 C Return 10 EF 9 A day 11 DU 10 C 16

  17. Type System (BBN Temporal Ordering Annotation) 17

  18. BBN Name Annotations (Type system) 18

  19. BBN-complex-coref-annotation Annotations: • Relations between entities • Member • Member Base • Subset • Subset Size (future type system) • Other annotations - Attributes of a mention • Reference type • Syntactic Context 19

  20. Type System for BBN (Complex) coreference annotation 20

  21. Type System for BBN (Complex) coreference annotation (contd…) 21

  22. UMBC Modality Annotations TMR – Text Meaning Representation or Concepts annotated Main Annotation – Modality. It has three main attributes: TYPE, VALUE, SCOPE & ATTRIBUTED-TO TMRs can be nested i.e. attributes or relation can refer to other TMRs 22

  23. UMBC Modality Annotations 23

  24. Interoperability: Data • Common data model • Multiple implementations • based on the same underlying schema(formal object model) • meet different goals / requirements • Implementation Criteria: • support effective run-time annotation(e.g. UIMA type system) • Support effective user interface, query/update(e.g. OntoNotes) • Support on-the-fly schema extension (e.g. CMU’s AnnotationsDB)

  25. Interoperability: Data [2] • Formal object model is mapped to: • UIMA type system definition (create) • OntoNotes RDBMS schema (extend) • CMU’s Annotations DB (extend) • Annotated data can be represented in any format that implements the formal model • “Have your cake and eat it too”

  26. CMU’s Annotations Database • MySQL implementation • Java APIs (SQL connection API and simple object access API) • Fully integrated with UIMA • Used on DTO and DARPA projects • PRO: tag types can be extended at run time by the application (schema supports open-ended type definition) • CON: interactive tools are currently limited

  27. * * * Annotations Database tag typevalueparent In an interview with Defense News, Indian Defence Research and Development Organization (DRDO) scientists said India was launching a comprehensive plan to develop a wide range of modern nuclear missiles. Within two years, India would develop an intercontinental ballistic missile (ICBM), ... <entity type=org offset=21 length=12 /> <entity type=org offset=35 length=59 /> <entity type=gpe offset=111 length=5 source=bbn ref=#INDIA /> <entity type=gpe offset=223 length=5 source=bbn ref=#INDIA /> <entity type=fac offset=231 length=41 /> * span offset length passage text document datetimedocno doctype

  28. An Integrated Annotation DB in OntoNotes Sameer Pradhan, Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel http://www.bbn.com/NLP/OntoNotes

  29. Goals • Capture multiple layers of annotation and modeling • Syntax • Propositions • Word sense • Ontology • Coreference • Names • Using an integrated relational database representation • Enforces consistency across the different annotations • Supports integrated models that can combine evidence from different layers

  30. Unified Representation • Provide a bare-bones representation independent of the individual semantics that can • Efficiently capture intra- and inter- layer semantics • Maintain component independence • Provide mechanism for flexible integration • Integrate information at the lowest level of granularity • A Relational Database

  31. Corpus Senses Trees Propositions Coreference Names Unified Relational Representation

  32. Example: DB Representation of Syntax • Treebank tokens (stored in the Token table) provide the common base • The Tree table stores the recursive tree nodes, each with its span • Subsidiary tables define the sets of function tags, phase types, etc.

  33. Advantages of an Integrated Representation • Each layer translates into a common representation • Clean, consistent layers • Resolve the inconsistencies and problems that this reveals • Well defined relationships • Database schema defines the merged structure efficiently • Original representations available as predefined views • Treebank, PropBank, etc. • SQL queries can extract examples based on multiple layers or define new views • Python Object-oriented API allows for programmatic access to tables and queries

  34. major reductions and realignments of troops in central Europe S NP SYNTAX NP PP PP IN NP JJ NNS CC NNS IN NP NNS JJ NNP ... major reductions and realignments of troops in central Europe – ... Syntax Layer • Identifies meaningful phrases in the text • Lays out the structure of how they are related Concerns about the pace of the Vienna talks -- which are aimed at the destruction of some 100,000 weapons , as well as major reductions and realignments of troops in central Europe – also are being registered at the Pentagon .

  35. S NP ARG2 NP PP PP ARGM-LOC ARG1 IN NP JJ NNS CC NNS IN NP NNS JJ NNP ... major reductions and realignments of troops in central Europe – ... Propositional Structure • Tells who did what to whom • For both verbs and nouns Concerns about the pace of the Vienna talks -- which are aimed at the destruction of some 100,000 weapons , as well as major reductions and realignments of troops in central Europe – also are being registered at the Pentagon . Concerns about the pace of the Vienna talks -- which are aimed at the destruction of some 100,000 weapons , as well as major reductions and realignments of troops in central Europe – also are being registered at the Pentagon .

  36. Predicate Frames • Predicate frames define the meanings of the numbered arguments Concerns about the pace of the Vienna talks -- which are aimed at the destruction of some 100,000 weapons , as well as major reductions and realignments of troops in central Europe – also are being registered at the Pentagon . Predicate Frames Predicate Frames aim reduction aim.01 – Plan reduce.01 – Make less reduce.01 – Make less ARG0 – Agent ARG1 – Thing falling ARG2 – Amount fallen ARG3 – Starting point ARG4 – Ending point ARG0 – Aimer ARG1 – Action aim.02 – Directed motion aim.02 – Directed motion ARG0 – Aimer ARG1 – Thing in motion ARG2 – Target

  37. Word Sense Word Sense aim register • Point or direct object, weapon, at something ... • Wish, purpose or intend to achieve something • Enter into an official record • Be aware of, enter into someone’s conciousness • Indicate a measurement • Show in one’s face Word Sense and Ontology • Meaning of nouns and verbs are specified • All the senses are annotatable at 90% inter-annotator agreement • Catalog of possible meanings supplied in the sense inventory files • Ontology links (currently being added) will capture similarities between related senses of different words Concerns about the pace of the Vienna talks -- which are aimed at the destruction of some 100,000 weapons , as well as major reductions and realignments of troops in central Europe – also are being registered at the Pentagon . Concerns about the pace of the Vienna talks -- which are aimed at the destruction of some 100,000 weapons , as well as major reductions and realignments of troops in central Europe – also are being registered at the Pentagon . • Enter into an official record • Wish, purpose or intend to achieve something

  38. President Bush conventional arms talk Vienna talks – which are aimed at the destruction of some 100,000 weapons , as well as major reductions and realignments of troops the Pentagon in central Europe the Pentagon He Pentagon e0 e0 e0 Coreference • Identifies different mentions of the same entity in text – especially links definite, referring noun phrases, and pronouns in text • Two types – Identity as well as Attributive coreference tagged. Concerns about the pace of the Vienna talks -- which are aimed at the destruction of some 100,000 weapons , as well as major reductions and realignments of troops in central Europe – also are being registered at the Pentagon .

  39. Example of DB Query Function What is the distribution of named entities that are ARG0s of the predicate “say”? if (proposition.lemma == “say”): for a_proposition in a_proposition_bank: if(a_proposition.lemma != "say"): arg_in_p_q = "select * from argument where proposition_id = '%s';" % (a_proposition.id) a_cursor.execute(arg_in_p_query) argument_rows = a_cursor.fetchall() for a_argument_row in argument_rows: a_argument_id = a_argument_row["id"] a_argument_type = a_argument_row["type"] if(a_argument_type != "ARG0"): n_in_arg_q = "select * from argument_node where argument_id = '%s';" % (a_argument_id) a_cursor.execute(n_in_arg_q) argument_node_rows = a_cursor.fetchall() for a_argument_node_row in argument_node_rows: a_node_id = a_argument_node_row["node_id"] a_ne_node_query = "select * from name_entity where subtree_id = '%s';" % (a_node_id) a_cursor.execute(a_ne_node_query) ne_rows = a_cursor.fetchall() for a_ne_row in ne_rows: a_ne_type = a_ne_row["type"] ne_hash[a_ne_type] = ne_hash[a_ne_type] + 1 a_tree = a_tree_document.get_tree(a_tree_id) a_node = a_tree.get_subtree(a_node_id) for a_child in a_node.subtrees(): a_ne_subtree_query = "select * from name_entity where subtree_id = '%s';" % (a_child.id) subtree_ne_rows = a_cursor.execute(a_ne_subtree_query) ne_subtree_rows = a_cursor.fetchall() for a_ne_subtree_row in ne_subtree_rows: a_subtree_ne_type = a_ne_subtree_row["type"] ne_hash[a_subtree_ne_type] = ne_hash[a_subtree_ne_type] + 1 query = “select * from argument where proposition_id = '%s';” .. if (argument_type == "ARG0"): for child in node.subtrees():

  40. Conclusion • Integrating the annotation layers using a relational schema • Improves consistency • Allows predictive features that combine evidence from multiple layers • Easily Accessible • Through Python API • SQL queries

  41. Interoperability: Components OntoNotes A shared, formal type systemallows multiple data formats tobe combined effectively OntoNotesCollectionReader OntoNotesCASConsumer XCASCollectionReader key UIMAAnalysisEngine XML Customer’sannotators Filestorage XCASCASConsumer ADBCollectionReader ADBCASConsumer RDBMSstorage File SystemCollectionReader TXT AnnotationsDB New UIMAwrapper ExistingUIMA wrapper

  42. Issues for Discussion • Persistence formats optimize for different concerns • RDBMS – relational querying, update • XCAS – fast deserialization of run-time objects • Consider extending schema to hold XML serialization of document annotations

More Related