adam kilgarriff lexical computing ltd l.
Skip this Video
Loading SlideShow in 5 Seconds..
Corpus Evaluation PowerPoint Presentation
Download Presentation
Corpus Evaluation

Loading in 2 Seconds...

play fullscreen
1 / 22

Corpus Evaluation - PowerPoint PPT Presentation

  • Uploaded on

Adam Kilgarriff Lexical Computing Ltd. Corpus Evaluation. Now Corpora to spec Choice Need to evaluate. Then Very few corpora Use what ’ s there. Intrinsic See what it looks like Extrinsic Embed in a task How well do you do at the task Better It all depends what you want it for.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Corpus Evaluation

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
adam kilgarriff lexical computing ltd
Adam Kilgarriff

Lexical Computing Ltd

Corpus evaluation

Corpus Evaluation
Corpus evaluation
  • Now
    • Corpora to spec
    • Choice
    • Need to evaluate
  • Then
    • Very few corpora
    • Use what’s there
Corpus evaluation
  • Intrinsic
    • See what it looks like
  • Extrinsic
    • Embed in a task
    • How well do you do at the task
    • Better
      • It all depends what you want it for
it all depends what you want it for but
Corpus evaluation it all depends what you want it for but
  • ‘general English (/French/Chinese/ …)’
    • Many purposes
    • Not specialist sublanguage
  • A decent construct?
    • Not sure but it has form
      • General language dictionaries
      • “how good is a corpus, for making them?”
general truths
Corpus evaluationGeneral truths
  • Duplicates bad
  • Noise bad
  • Big good
  • Diverse (good coverage of varieties within research scope, not dominated by any one variety) good
word sketch
Corpus evaluationword sketch

A corpus-derived one-page summary of a word’s grammatical and collocational behaviour

Corpus evaluation

Macmillan English Dictionary

For Advanced Learners

Ed: Rundell, 2002

Corpus evaluation
  • 11 years
    • 1999-2010
  • Feedback
    • Good but anecdotal
  • Formal evaluation
Corpus evaluationGoal

Collocations dictionary

Model: Oxford Collocations Dictionary


Ask a lexicographer

For 42 headwords

For 20 best collocates per headwords

“should we include this collocation in a published dictionary?”

sample of headwords
Corpus evaluationSample of headwords

Nouns verbs adjectives, random

High (Top 3000)‏

N space solution opinion mass corporation leader

V serve incorporate mix desire

Adj high detailed open academic

Mid (3000- 9999)‏

N cattle repayment fundraising elder biologist sanitation

V grieve classify ascertain implant

Adj adjacent eldest prolific ill

Low (10,000- 30,000)‏

N predicament adulterer bake bombshell candy shellfish

V slap outgrow plow traipse

Adj neoclassical votive adulterous expandable

Corpus evaluation

Precision and recall

  • We tested precision
  • Recall is harder
    • How do we find all the collocations that the system should have found?
four languages three families
Corpus evaluationFour languages, three families


ANW, 102m-word lexicographic corpus


UKWaC, 1.5b web corpus


JpWaC, 400m web corpus


FidaPlus, 620m lexicographic corpus

user evaluation
Corpus evaluationUser evaluation

Evaluate whole system

Will it help with my task

Eg preparing a collocations dictionary

Contrast: developer evaluation

Can I make the system better?

Evaluate each module separately

Current work

Corpus evaluationComponents


NLP tools

Segmenter, lemmatiser, POS-tagger

Sketch grammar


Corpus evaluationPracticalities


Good, Good-but

Merge to good

Maybe, Maybe-specialised, Bad

Merge to bad

For each language

Two/three linguists/lexicographers

If they disagree

Don't use for computing performance

Corpus evaluationResults

Dutch 66%

English 71%

Japanese 87%

Slovene 71%

Two thirds of a collocations dictionary can be gathered automatically

world final problem
Corpus evaluation<world, final> problem
  • Is it good?
    • Superficially no
    • Look at concordances:
      • World cup finals
  • Solution
    • ‘Commonest string’
next step
Corpus evaluationNext step
  • Recall
      • 200 collocates per headword
      • Selected from
        • All the corpora we have
        • Various parameter settings
      • Plus just-in-time evaluation for 'new' collocates
  • Then
    • For a sample of headwords
      • These are the collocations we should get
from sketches to corpora
Corpus evaluationFrom sketches to corpora
  • Hold other inputs constant
    • Just one varies
    • Evaluate that one
  • Hold tools, stats, grammar constant
    • evaluate corpora
Corpus evaluationCriteria
      • Duplicates bad
      • Noise bad
      • Big good
      • Diverse (good coverage of varieties within research scope, not dominated by any one variety) good
  • We think so
over next year
Corpus evaluationOver next year
  • Build test sets
  • Textbook cases
    • English
      • BNC vsUKWaCvs OEC vsGigaword
    • Dutch
      • ANW corpus vs web corpus
  • web crawling, deduplication
    • Which parameters give best results?
Corpus evaluation

Thank you