1 / 32

Main Mono and Bilingual Tasks: Track Organisation and Results Analysis

Main Mono and Bilingual Tasks: Track Organisation and Results Analysis. Outline. CLEF Infrastructure : DIRECT. Information Hierarchy. experimental collections and the experiments are data , since they are the raw, basic elements needed for any further investigation

jerome-olson
Download Presentation

Main Mono and Bilingual Tasks: Track Organisation and Results Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Main Mono and Bilingual Tasks: Track Organisation and Results Analysis

  2. Outline

  3. CLEF Infrastructure: DIRECT

  4. Information Hierarchy • experimental collections and the experiments are data, since they are the raw, basic elements needed for any further investigation • performance measurements are information, since they are the result of computations and processing on the data, • descriptive statistics and the hypothesis tests are knowledge, since they are a further elaboration of the information carried by the performance measurements • theories, models, algorithms, and techniques are wisdom, since they provide interpretation, explanation, and formalization of the content of the previous levels.

  5. Approach to the Evaluation (1/2) • Introduce a conceptual model • it makes clear what are the entities entailed by the information space of an evaluation campaign, their features, and their relationships • logical models can be derived from it to manage and preserve the experimental data • commonly agreed data formats for exchanging information can be derived from it • Develop common metadata formats • they provide meaning to the data, and thereby enable their sharing and re-use • they allow to keep track of the lineage of the managed information • Adopt a unique identification mechanism • it allows for explicit citation and easy access to the scientific data and it supports the enrichement of the scientific data

  6. Approach to the Evaluation (2/2) • Provide common tools for statistical analyses • they allow for judging whether measured differences between retrieval methods can be considered statistically significant • a uniform way of performing statistical analyses on experiments make the analysis and assessment of the experiments comparable too • Design and develop a Digital Library System (DLS) for IR scientific data • it is well suited for managing and making accessible the scientific data and the experiments produced during the course of an evaluation campaign • it also provides tools for analyzing, comparing, and citing the scientific data of an evaluation campaign, as well as curating, preserving, annotating, enriching, and promoting the re-use of them • Give to organizations responsible for evaluation initiatives an active role in this process • they should take a leadership role in developing a comprehensive strategy for long-lived digital data collections and drive the research community through this process in order to improve the way of doing research • they should take care also of defining guiding principles, policies, best practices for making use of the scientific data produced during the evaluation campaign itself

  7. Internationalizationof the User Interface

  8. Identification: DigitalObjectIdentifiers (DOI) 10.2415/AH-BILI-X2BG-CLEF2007.JHU-APL.APLBIENBGTD4 • DOIs • allowustouniquelyidentify a digitalobject • are persistent and actionable • aimespecially at the intellectualproperty • WeassignDOIsto: • collections − prefix 10.2453 • topics − prefix 10.2452 • experiments − prefix 10.2415 • pools − prefix 10.2454 • statisticaltests − prefix 10.2455 http://www.medra.org

  9. DOI Resolution http://dx.doi.org

  10. ExperimentMetrics

  11. ExperimentStatistics

  12. ExperimentPlots

  13. Task Statistics

  14. Task Plots

  15. Appendices (1/2)

  16. Appendices (2/2)

  17. TrackOverview

  18. Participation 22participants12countries

  19. ParticipationbyCountry

  20. Tasks and Collections • Monolingual and bilingualtasks have principally offered for Central European languages: Bulgarian, Czech and Hungarian • Topics in 16 languages • Europeanlanguages: Bulgarian, Czech, English, French, Hungarian, Italian and Spanish • non-European languages (for X2EN): Amharic, Chinese, Indonesian, Oromo • Indian sub-task: Bengali, Hindi, Marathi, Tamil and Telugu

  21. Participationby Task 172 submittedruns Disappointingparticipation

  22. Runsby Source Language

  23. MonolingualTasks

  24. MonolingualBulgarian

  25. MonolingualCzech

  26. MonolingualHungarian

  27. MonolingualEnglish*

  28. ApproachestoMonolingualRetrieval LinguisticStemmers:both light and aggressive LinguisticStemmers:both light and aggressive • Mainemphasis: • stemming • morphologicalanalysis • relevance feed-back • Stemming vs 4-grams • impact on individualtopicsbutnot on average • blindrelevance feedback can bedetrimental • Stemming vs 4-grams • impact on individualtopicsbutnot on average • blindrelevance feedback can bedetrimental • Stemming vs 4-grams • impact on individualtopicsbutnot on average • blindrelevance feedback can bedetrimental MorphologicalLemmatizer • Relevance Feed-back: • probabilistic RF • mutual information RF • Relevance Feed-back: • probabilistic RF • mutual information RF • NLP techniques • NamedEntityRecognition • NLP techniques • NamedEntityRecognition • NLP techniques • NamedEntityRecognition Indexing: word-based or 4-grams Indexing: word-based or 4-grams Indexing: word-based or 4-grams word decompounding

  29. BilingualTasks

  30. Bilingual X  English

  31. ApproachestoBilingual X2EN • Mainemphasis: • bilingualdictionaries • machinetranslation • coverageoflexicons • useof pivot languages Best Bilingual English system isabout88%of the best monolingual system • bilingualdictionaries and pivot languages • queryexpansionwith RF • parallelcorpora • translationambiguityresolutionwith a graphbasedapproach • lexiconcoveragewith a pattern-basedapproach • AfaanOromostemmer • stop listcreation • bilingualOromo-Englishdictionarycreation • BilingualHungarianto English • bilingualdictionary • exploitingWikipediatoremoveimprobabletranslations

  32. Bilingual X2EN: IndianSubtask • limitedlinguisticresources • phoneme-basedtransliterationsto generate equivalent English queries • stemmers and morphologicalanalyzersifavailable • limitedlinguisticresources • phoneme-basedtransliterationsto generate equivalent English queries • stemmers and morphologicalanalyzersifavailable • bilingualdictionary • OOV using a rule-basedapproachfortransliteration and editdistances • translationdisambiguation via a page-rank style algorithm • bilingualdictionary • OOV using a rule-basedapproachfortransliteration and editdistances • translationdisambiguation via a page-rank style algorithm • bilingualdictionary • OOV using a rule-basedapproachfortransliteration and editdistances • translationdisambiguation via a page-rank style algorithm • Hindi-English and Telugu-Englishdictionariescreated in one week • TFIDF approachcombinedwithbooleanoperators • Hindi-English and Telugu-Englishdictionariescreated in one week • TFIDF approachcombinedwithbooleanoperators • statistical MT system trained on parallelalignedsentences • languagemodels • statistical MT system trained on parallelalignedsentences • languagemodels • bilingualdictionaries • stop listcreation • stemming and n-gram • bilingualdictionaries • stop listcreation • stemming and n-gram

More Related