1 / 40

Machine Learning for Textual Information Access: Results from the SMART project

This paper presents the results from the SMART project, which investigates methods for cross-language information retrieval using dictionary adaptation and latent semantic methods based on Canonical Correlation Analysis.

engle
Download Presentation

Machine Learning for Textual Information Access: Results from the SMART project

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Machine Learning for Textual Information Access: Results from the SMART project Nicola Cancedda, Xerox Research Centre Europe First Forum for Information Retrieval Evaluation Kolkata, India, December 12th-14th, 2008 TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAAAAAAAAA

  2. The SMART Project • Statistical Multilingual Analysis for Retrieval and Translation (SMART) • Information Society Technologies Programme • Sixth Framework Programme, “Specific Target Research Project” (STReP) • Start date: October 1, 2006 • Duration: 3 years • Objective: bring Machine Learning researchers to work on Machine Translation and CLIR

  3. The SMART Consortium

  4. The SMART Consortium

  5. Premise and Outline • Two classes of methods for CLIR investigated in SMART • Methods based on dictionary adaptation for the cross-language extension of the LM approach in IR • Latent semantic methods based on Canonical Correlation Analysis • Initial plan (reflected in abstract): to present both • ...but it would take too long, so: • Outline: • (Longish) introduction to state of the art in Canonical Correlation Analysis • A number of advances obtained by the SMART project • For lexicon adaptation methods: check out deliverable D 5.1 from the project website!

  6. Background: Canonical Correlation Analysis

  7. Canonical Correlation Analysis • Abstract view: • Word-vector representations of documents (or queries, or whatever text span) are only superficial manifestations of a deeper vector representation based on concepts. • Since they cannot be observed directly, these concepts are latent • If two spans are the translation of one another, their deep representation in terms of concepts is the same. • Can we recover (at least approximately) the latent concept space? Can we learn to map text spans from their superficial word appearance into their deep representation? • CCA: • Assume mapping from deep to superficial representation is linear • Estimate mapping from empirical data

  8. Five documents in the world of concepts 1 3 2 4 5

  9. The same five documents in two languages 1 3 2 3 5 1 4 2 4 5

  10. 5’ 3’ 4’’ 2’’ 1’ 1’’ 5’’ 3’’ 4’ 2’ Finding the first Canonical Variates 1 3 2 3 5 1 4 2 4 5

  11. Finding the first Canonical Variates Maximal covariance to work back the rotation • Find the two directions, one for each language, such that projections of documents are maximally correlated. • Assuming data matrices X and Y are (row-wise) centered: Normalization by the variances to adjust for “stretched” dimensions C1 expressed in the basis of X and Y resp.

  12. Complexity: Finding the first Canonical Variate • Find the two directions, one for each language, such that projections of documents are maximally correlated • Turns out equivalent to finding the largest eigen-pair in a Generalized Eigenvalue Problem (GEP):

  13. Finding further Canonical Variates • Assume we already found i-1 pairs of Canonical Variates: • Turns out equivalent to finding the other eigen-pairs in the same GEP

  14. Examples from the Hansard Corpus

  15. Kernel CCA • Cubic complexity in the number of dimensions becomes soon intractable, especially with text • Also, it could be better to use similarity measures other than inner product of document (possibly weighted) vectors •  Kernel CCA: from primal to dual formulation, since it can be proved that the wxi (resp. wyi) is in the span of the columns of X (resp. Y)

  16. Complexity: Kernel CCA • The computation is again done by solving a GEP:

  17. Unit variances Unit variances Overfitting • Problem: if m · nx and m · ny then there are (infinite) trivial solutions with perfect correlation : OVERFITTING • E.g. two (centered) points in R2: Given an arbitrary direction in the first space... ...we can find one with perfect correlation in the second 1 1 2 2 Perfect correlation... for no matter what direction! Unit covariance

  18. Regularized Kernel CCA • We can regularize the objective function by trading correlation against good account of variance in the two spaces:

  19. 5 1 3 2 2 1 4 3 4 5 1 3 3 4 2 2 5 1 4 5 Multiview CCA • (K)CCA can take advantage of the “mutual information” between two languages... • ...but what if we have more than two? Can we benefit from multiple views? Also known as Generalised CCA.

  20. . Multivariate Eigenvalue Problem Multiview CCA • There are many possible ways to combine pairwise correlations between views (e.g. sum, product, min, ...). • Chosen approach: SUMCOR [Horst-61]. With a slightly different regularization than above, this is:

  21. Multiview CCA • Multivariate Eigenvalue Problems (MEP) are much harder to solve then GEPs: • [Horst-61] introduced an extension to MEPs of the standard power method for EPs, for finding the set of first canonical variates only • Naïve implementations would be quadratic in the number of ducuments, and scale up to no more than a few thousands

  22. Innovations from SMART

  23. Innovations from SMART • Extensions of the Horst algorithm [Rupnik and Shawe-Taylor] • Efficient implementation linear in the number of documents • Version for finding many sets of canonical variates • New regression-CCA framework for CLIR [Rupnik and Shawe-Taylor] • Sparse KCCA [Hussain and Shawe-Taylor]

  24. Efficient Implementation of Horst algorithm • Horst algorithm starts with a random set of vectors: • then iteratively multiplies and renormalizes until convergence: Inner loop: k2 matrix-vector multiplications, each O(m2) • Extension (1): exploiting the structure of the MEP matrix, one can refactor computation and save a O(k) factor in the inner loop. The inner loop can be made O(kms) instead of O(k2m2) • Extension (2): exploiting sparseness of the document vectors, one can replace each (vector) multiplication with a kernel matrix (O(m2)) with two multiplications with the document matrix (O(ms) each, where s is the max number of non-zero components in document vectors). • Leveraging this same sparsity, kernel inversions can be replaced by cheaper numerical linear system resolutions.

  25. Extended Horst algorithm for finding many sets of canonical variates • Horst algorithm only finds the first set of k canonical variates • Extension (3): maintain projection matrices Pit that project ¯k,t’s at each iteration onto the subspace orthogonal to all previous canonical variates for space i. Finding d sets of canonical variates can be done in O(d2mks). This scales up!

  26. MCCA: Experiments • Experiments: mate retrieval with Europarl • 10 languages, • 100,000 10-ways aligned sentences for training • 7873 10-ways aligned sentences for testing • Document vectors: uni-, bi- and tri-grams (~200k features for each language). TF*IDF weighting and length normalization. • MCCA used to extract d = 100-dimensional subspaces • Baseline alternatives for selecting new basis: • k-means clustering centroids on concatenated multi-lingual document vectors • CL-LSI, i.e. LSI on concatenated vectors

  27. Some example latent vectors

  28. MCCA experiment results • Measure: recall in Top 10, averaged over 9 languages

  29. MCCA experiment results • More realistic experiment: now pseudo-queries formed with top 5 TF*IDF scoring components in each sentence

  30. Extension (4): Regression - CCA • Given a query q in one language, find the target language vector w which is maximally correlated to it: • Solution: • Given this “query translation” we can then find the closest target documents using the standard cosine measure • Promising initial results on CLEF/GIRT dataset: better then standard CCA, but cannot take thesaurus into account, so MAP still not competitive with the best

  31. Extension (5): Sparse - KCCA • Seeking sparsity in dual solution: first canonical variates expressed as linear combinations of only relatively few documents • Improved efficiency • Alternative regularization Same set of indices i

  32. Sparse - KCCA • For a fixed set of indices i: • But how do we select i ?

  33. Algorithm 2 • Set i to the index of the top d values of • Solve GEP for index set i Sparse – KCCA: Algorithms • Algorithm 1 • initialize • for i = 1toddo Deflate kernel matrices • end for • Solve GEP for index set i Deflation consists in transforming the matrices to reflect a projection onto the space orthogonal to the current basis in feature space

  34. Sparse – KCCA: Mate retrieval experiments Europarl, English-Spanish KCCA Train: 24693 sec. Test: 27733 sec. SKCCA (1) Train: 5242 sec. Test: 698 sec. SKCCA (2) Train: 1873 sec. Test: 695 sec.

  35. SMART - Website D 5.1 on lexicon-based methods and D 5.2 on CCA • Project presentation and deliverables • http://www.smart-project.eu

  36. SMART - Dissemination and Exploitation • Platforms for showcasing developed tools:

  37. Thank you!

  38. Shameless plug • Cyril Goutte, Nicola Cancedda, Marc Dymetman and George Foster, eds: Learning Machine Translation, MIT Press, to appear in 2009.

  39. References

  40. Self-introduction Machine Learning (kernels for text) Text Categorization Grammar Learning (Statistical) Machine Translation ca. 2004 Natural Language Generation

More Related