ANALYSIS OF INTER-ANNOTATOR AGREEMENT (TEXT MINING & REG. ANNOTATION) RegCreative Jamboree , Friday, December, 1st,

1. TEXT MINING & REG. ANNOTATION ANALYSIS OF INTER-ANNOTATOR AGREEMENT (TEXT MINING & REG. ANNOTATION) RegCreative Jamboree , Friday, December, 1st, (2006) MARTIN KRALLINGER, 2006

2. TEXT MINING & REG. ANNOTATION • MAIN ASPECTS • Explore annotation overlap • Discuss variability in annotation • Text mining and regulatory element annotation: needs, limits, tasks MARTIN KRALLINGER, 2006

3. TEXT MINING & REG. ANNOTATION SOCIOLOGY OF GENOME ANNOTATION (Lincoln Stein 2001) • Models of annotation • Museum model: small group of specialized curators • Jamboree model: a group of biologists and bioinformaticians come together for a short intensive annotation workshop • Cottage industry: decentralized effort of annotators among therecruited community • Factory model: highly automated methods (Elsik et al, 2006) MARTIN KRALLINGER, 2006

4. TEXT MINING & REG. ANNOTATION WHY PRE-JAMBOREE QUEUE? • Get familiar with annotation system (before jamboree)! • Understand content and annotation strategy of Oreganno • Detect aspects which require improvements such as incompleteness, ambiguity or wrong structures in annotation strategy, guidelines or documentation -> active Feedback (Questionnaire and wiki) • Assess consistency of the current annotation procedures • Explore which aspects affect annotation agreement • Estimate difficulty of task (alternative interpretation, uncertainty, etc,..) MARTIN KRALLINGER, 2006

5. TEXT MINING & REG. ANNOTATION SIMILARITY MEASURES • Similarity calculation popular subject in computer science • Different entities considered: • Feature vectors: Alignment, Cosine, Dice, Euclidean, … • Strings or sequences of strings (text): averaged String Matching, TFIDF • Sets: Jaccard, Loss of Information, Resembalance • Sequences: Levensthein Edit Distance • Trees:Bottom-up/Top-down Maximum Common Subtree, Tree Edit Distance • Graphs: Conceptual Similarity, Graph Isomorphism, Subgraph Isomorphism, Maximum Common Subgraph Isomorphism, Graph Isomorphism Covering, Shortest Path • Information theory: Jiang & Conrath, Lin, Resnik • Bioinformatics: sequence similarity, structural similarity, similarity of gene expression • Here similarity between human annotations ( refer to SimPack project examples) MARTIN KRALLINGER, 2006

6. TEXT MINING & REG. ANNOTATION MEASUREMENT OF OBSERVER AGREEMENT • Assumption when independent annotators agree they are correct?! • Statistical agreement measures for categorical data • Overall proportion of agreement • Pairwise comparison; Cohen’s kappa; Pearson Chi-square • Weighted kappa for multiple categories • High accuracy implies high agreement • Kappa sometimes is inconsistent with accuracy measured as AROC Kappa coefficient Measurement of Observer agreement Kundel and Polansky, Statistical concepts Series (2003) MARTIN KRALLINGER, 2006

7. TEXT MINING & REG. ANNOTATION ANNOTATOR AGREEMENT FOR WSD • Word Sense Disambiguation (WSD) a central problem in NLP • WSD: discerning the meaning of a word in context • Two human annotators may disagree in their sense assignment • Agreement of human annotators often the baseline for evaluation of automated approaches • Case study using more than 30,000 instances of the most frequently occurring nouns and verbs in English • Sense tagged word in sentences manually by two groups of annotator to WordNet • Used the Kappa score to measure inter-annotator agreement considering effect of chance agreement • Difficult to achieve high agreement when they have to assign refined sense tags • Importance of example sentences for the usage of each word sense A case Study on Inter-Annotator Agreement for Word Sense Disambiguation, Ng et al MARTIN KRALLINGER, 2006

8. TEXT MINING & REG. ANNOTATION AGREEMENT OF SPEECH CORPORA • Phonetically annotated speech corpora • Quality of manual annotations affected by: • Implicit incoherence: labeling incoherent due to human variability in perceptual capacities and other factors • Lack of consensus on coding schema: manual annotations reflect the variability of the interpretation and application of the coding schema by the annotators • Annotator characteristics: individual characteristics of coders such as familiarity with the material, amount of former training, motivation, interest and fatigue induced errors Measuring the reliability of Manual annotations of Speech corpora, Gut and Bayerl MARTIN KRALLINGER, 2006

9. TEXT MINING & REG. ANNOTATION CHALLENGES FOR OREGANNO ANNOTATION • Complexity of gene regulation • Need of ontologies and lexical resources • Deep inference of domain expert curators • Spatial, temporal, experimental conditions • Range of entity types: genes, regulatory sequences, proteins • Gene family and individual gene member distinction • TF binding site sequence extraction and mapping to genome • TF mapping to normalized database entries (NCBI, Ensembl) • Archeology-like annotation: annotation of old papers • BUT GENE REGULATION IS ONE OF THE MAIN BIOLOGICAL INFORMATION (ANNOTATION) ASPECTS! MARTIN KRALLINGER, 2006

10. TEXT MINING & REG. ANNOTATION SOURCES FOR ANNOTATION VARIABILITY (1) • Curator background (biologist, bioinformatician,...) • Familiarity with the annotation system • Number of previously annotated papers or proteins • Prior knowledge on the regulated gene or TF • Prior knowledge (experience) on the experimental types • Sub-domain knowledge (e.g. developmental biology or OS) • Publication date (reflect the state of knowledge) MARTIN KRALLINGER, 2006

11. TEXT MINING & REG. ANNOTATION SOURCES FOR ANNOTATION VARIABILITY (2) • Nr. of papers annotated the same day (fatigue effect) • Unclear or partial documentation of certain annotation aspects • Annotation type (ontology of annotation types?, CV?) • Nr. of pages, figures, tables, references,… • Consultation of additional resources (material, databases, web) • Different degrees of granularity in annotation • Differences in the recall of manually extracted annotations (all ?) • Sequence (paper/database, strand, typos, length) MARTIN KRALLINGER, 2006

12. TEXT MINING & REG. ANNOTATION REGCREATIVE CASE STUDY: PREJAMBOREE (1) • Relatively few articles -> only exploratory examination • Annotation type: 9/11 (2071609, 10674400: RR vs. TFBS) • Considerable difference in average nr. of annotations/paper • Some only extracted a single annotation others basically every annotation mentioned in the paper • Almost perfect agreement in organism source (1 case of human and mouse disagreement), but genes correct! • Very high agreement on the gene names, only few user defined cases (which are difficult to evaluate) MARTIN KRALLINGER, 2006

13. TEXT MINING & REG. ANNOTATION REGCREATIVE CASE STUDY: PREJAMBOREE (2) • Certain disagreement in TF names, many are user defined! • Evidence class: high agreement many Transcription regulator site, and unknown • Evidence type: high agreement, some more complete than others, (again, some annotate all the types others only some of them) • Evidence sub-type: similar to evidence types, but in general a little lower agreement than for the evidence type. MARTIN KRALLINGER, 2006

14. TEXT MINING & REG. ANNOTATION Transcription names factor: PREJAMBOREE MARTIN KRALLINGER, 2006

15. TEXT MINING & REG. ANNOTATION Example case 1: TF annotation variance B A MARTIN KRALLINGER, 2006

16. TEXT MINING & REG. ANNOTATION Example case 2: TF annotation variance MARTIN KRALLINGER, 2006

17. TEXT MINING & REG. ANNOTATION Example case: difference in evidence types B A A B MARTIN KRALLINGER, 2006

18. TEXT MINING & REG. ANNOTATION Different amount of annotation extracted A A B B MARTIN KRALLINGER, 2006

19. TEXT MINING & REG. ANNOTATION REGCREATIVE CASE STUDY: JAMBOREE • Intensive annotation strategy: face to face with other curators and expert annotators • Get direct feedback and provide suggestions • Promote integration of additional aspects in the annotation structure as well as annotated information types • Populate the database with new annotation records • Explore efficient curation training strategies • Create Gold Standard collection of annotation records, maybe useful to allow example-based annotation training/evaluation • Explore demands of biologists / curators to text mining community - > where it would be useful MARTIN KRALLINGER, 2006

20. TEXT MINING & REG. ANNOTATION REGCREATIVE CASE STUDY: POST-JAMBOREE • Monitor improvements in the annotation consistency • Allow consistent community-based annotation • Promote integration of additional aspects in the annotation structure as well as annotated information types • Increase efficiency in populating the database • Construction for text mining training collection MARTIN KRALLINGER, 2006

21. TEXT MINING & REG. ANNOTATION ANNOTATION CONSISTENCY • For selection as relevant paper for curation • For the evidence class • For the evidence types • For the evidence subtypes • For the regulated genes • For the transcription factors • For cell types • How to structure comments • Other aspects: ... MARTIN KRALLINGER, 2006

22. TEXT MINING & REG. ANNOTATION TEXT MINING TASKS FOR GENE REGULATION EXTRACTION • Detection of relevant articles: abstracts or full text • Extraction of ranked list of regulated genes: mention or normalized gene (database entries) • Extraction of ranked list if TF • Extraction of ranked list of evidence type IDs together with name and text passage (sentence) • Extraction of ranked associations between these genes and TF • Extraction of associations to other controlled vocabularies or ontologies MARTIN KRALLINGER, 2006