1 / 34

Toric Modification on Machine Learning

Toric Modification on Machine Learning. Keisuke Yamazaki & Sumio Watanabe Tokyo Institute of Technology. Agenda. Learning theory and algebraic geometry Two forms of the Kullback divergence Toric modification Application to a binomial mixture model Summary. Agenda.

brennanj
Download Presentation

Toric Modification on Machine Learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Toric Modificationon Machine Learning Keisuke Yamazaki & Sumio Watanabe Tokyo Institute of Technology

  2. Agenda • Learning theory and algebraic geometry • Two forms of the Kullback divergence • Toric modification • Application to a binomial mixture model • Summary

  3. Agenda • Learning theory and algebraic geometry • Two forms of the Kullback divergence • Toric modification • Application to a binomial mixture model • Summary

  4. What is the generalization error? The true model Predictive model Generalization Error MLE MAP Bayes i.i.d. Learning model Data

  5. Algebraic geometry connected to learning theory in the Bayes method. • The formal definition of the generalization error. The true model Predictive model Data Learning model

  6. Algebraic geometry connected to learning theory in the Bayes method. • The formal definition of the generalization error. The true model Predictive model Another Kullback divergence : Data Learning model

  7. Algebraic geometry connected to learning theory in the Bayes method. • The formal definition of the generalization error. The true model Predictive model Another Kullback divergence : Data Learning model Algebraic Geometry Machine Learning / Learning Theory

  8. h(w) The zeta function has an important role for the connection. Analytic func. Zeta function : C-infinity func. with compact support h(w) holomorphic function Algebraic Geometry Machine Learning / Learning Theory

  9. The zeta function has an important role for the connection. Zeta function : Kullback divergence Prior distribution holomorphic function Algebraic Geometry Machine Learning / Learning Theory

  10. The largest pole of the zeta function determines the generalization error. Asymptotic Bayes generalization error [ Watanabe 2001] Algebraic Geometry Machine Learning / Learning Theory

  11. Agenda • Learning theory and algebraic geometry • Two forms of the Kullback divergence • Toric modification • Application to a binomial mixture model • Summary

  12. Resolution of singularities Calculation of the zeta function requires well-formed H(w). : Factorized form : Polynomial form

  13. Blow-ups The resolution of singularities with blow-ups is an iterative method.

  14. Blow-ups The resolution of singularities with blow-ups is an iterative method. Kullback divergence in ML is complicated and has high dimensional w :

  15. Blow-ups The bottleneck is the iterative method. Applicable function Any function Computational cost Large

  16. Blow-ups Toric Modification Toric modification is a systematic method for the resolution of singularities. Applicable function Any function Computational cost Large

  17. Agenda • Learning theory and algebraic geometry • Two forms of the Kullback divergence • Toric modification • Application to a binomial mixture model • Summary

  18. Toric Modification Newton diagram is a convex hull in the exponent space

  19. Toric Modification The borders determine a set of vectors in the dual space.

  20. Toric Modification Add some vectors subdividing the spanned area.

  21. Toric Modification Selected vectors construct the resolution map.

  22. Toric Modification Non-degenerate Kullback divergence • The condition to apply the toric modification to the Kullback divergence For all i :

  23. Toric Modification Blow-ups Toric modification is “systematic”. Blow-up The search space will be large. We cannot know how many iterations we need. The number of vectors is limited.

  24. Blow-ups Toric Modification Toric modification can be an effective plug-in method. Non-degenerate function Applicable function Any function Computational cost Limited Large

  25. Agenda • Learning theory and algebraic geometry • Two forms of the Kullback divergence • Toric modification • Application to a binomial mixture model • Summary

  26. An application to a mixture model • Mixture of binomial distributions The true model: Learning model:

  27. The Newton diagram of the mixture

  28. The resolution map based on the toric modification

  29. The generalization error of the mixture of binomial distributions : Polynomial form : Factorized form The zeta function: : Generalization error

  30. Agenda • Learning theory and algebraic geometry • Two forms of the Kullback divergence • Toric modification • Application to a binomial mixture model • Summary

  31. Summary • The Bayesian generalization error is derived on the basis of the zeta function. • Calculation of the coefficients requires the factorized form of the Kullback divergence. • Toric modification is an effective method to find the factorized form. • The error of a binomial mixture is derived as the application.

  32. Thank you !!! Toric Modificationon Machine Learning Keisuke Yamazaki & Sumio Watanabe Tokyo Institute of Technology

  33. Asymptotic form of G.E. has been derived in regular models. Fisher information matrix Essential assumption:

More Related