1 / 17

Measures of Coincidence

Measures of Coincidence. Vasileios Hatzivassiloglou University of Texas at Dallas. A study of different measures. Smadja, McKeown, and Hatzivassiloglou (1996): Translating Collocations for Bilingual Lexicons: A Statistical Approach Use aligned parallel corpora (Hansards)

Download Presentation

Measures of Coincidence

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Measures of Coincidence Vasileios Hatzivassiloglou University of Texas at Dallas

  2. A study of different measures • Smadja, McKeown, and Hatzivassiloglou (1996): Translating Collocations for Bilingual Lexicons: A Statistical Approach • Use aligned parallel corpora (Hansards) • Task: Find translation for a word group across languages

  3. Sketch of algorithm • Start with set of collocations in French • Find candidate single word translations according to association between original collocation and translation • Measure association between source collocation and pairs of candidate words • Expand iteratively to triplets, etc. by recalculating association

  4. Dice vs. SI • Dice depends on conditional probabilities only • SI depends on the marginals: logP(X|Y)-logP(X) • SI depends on how rare X is • Limit behavior

  5. Asymmetry • Many kinds of asymmetry • Between X and Y • Between X=1 and X=0 • 1-1 matches versus 0-0 matches • Adding 0-0 matches does not change Dice • Adding 0-0 matches always increases SI

  6. Effect of asymmetry • Hypothetical scenario on 100 sentences • A,B appear together twice, by themselves three times each • Dice: 2×2 / (5+5) = 0.4 • SI: log (0.02 / (0.05×0.05)) = 3 bits • MI: 0.0457 bits

  7. Reversing one and zeroes • Now replace every 1 with 0 and vice versa • New variables A′, B′ occur together 92 times, each occurs by itself three times • Dice: 2×92 / (95 + 95) = 0.9684 • MI: Unchanged (0.0457 bits) • SI: log(0.92 / (0.95×0.95)) = 0.0277 bits

  8. Explaining the behavior • Limit effect as P(X) decreases with P(X|Y) constant • P(X) eventually dominates SI • Makes SI (and MI) more sensitive to estimation errors

  9. Bounds and testing purpose • No upper bound for SI and MI • Dice is always between 0 and 1 • Easy to test SI/MI for independence • Easy to test Dice for correlation

  10. Empirical comparison • How to compare without redoing the entire experiment? • Solution: Use competing measure in the last round • Test cases where the correct solution is available • Provide lower bound on competitor error

  11. Empirical results • 45 French collocations • 2 did not produce any candidate translation • Dice resulted in 36 correct, 7 incorrect translations • SI resulted in 26 correct, 17 incorrect translations

  12. Re-examining contingency tables • Ted Dunning, “Accurate Methods for the Statistics of Surprise and Coincidence”, Computational Linguistics, 1993. • Problem: Asymptotic normality assumptions • How much data is enough? • Are researchers aware of the need for statistical validity analysis?

  13. Rarity of words • Empirical counts on words show that 20–30% of words appear less than 1 in 50,000 words • Estimating binomial as normal: Good as long as np(1-p) > 5 • Significance overestimated by 20% for np=1,40 for np=0.1, 1020 for np=0.01

  14. Likelihood in parameter spaces • Parametric model (known except for parameter values) • Likelihood function H(ω;k) • Hypothesis represented by a point ω0

  15. Likelihood ratio • Test statistic: -2logλ • Rapidly approaches χ2 distribution for binomial H

  16. Comparing to chi-square • Leads to same formula as Pearson’s chi-square statistic when approximating with normal distribution • Diverges significantly from chi-square for low np • Closely follows chi-square distribution

  17. Experimental results • 32,000 words of financial text from Switzerland • Find highly correlated word pairs • Observe top-ranked entries for log-likelihood and chi-square • Chi-square leads to huge scores for rare pairs • 2,682 of 2,693 bigrams violate assumptions

More Related