Create Presentation
Download Presentation

Download Presentation
## Information Geometry -- Manifolds of Probability Distributions

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Algebraic Statistic: Penn State U Introduction to**Information GeometryShun-ichi AmariRIKEN Brain Science Institute**Information GeometryA Unifying FrameworkStatistical**Inference,Convex Analysis,Optimization,Machine learning, Signal Processing,Computer Vision**Information Geometry**-- Manifolds of Probability Distributions**Information Geometry**Systems Theory Information Theory Statistics Neural Networks Combinatorics Physics Information Sciences Math. AI Vision Riemannian Manifold Dual Affine Connections Optimization Manifold of Probability Distributions**Information Geometry ?**Gaussian distributions**Invariance**Invariant under different representation**Two Geometrical Structures**Riemannian metric affine connection --- geodesic Fisher information Orthogonality: innner product**Tangent space**Spanned by scores**AffineConnection**covariant derivative; parallel transport straight line**Duality: two affine connections**Y X Y X Riemannian geometry:**Dual Affine Connections**e-geodesic m-geodesic**Mathematical structure of**-connection : dually coupled**Divergence:**M Y Z positive-definite**Kullback-Leibler Divergence**quasi-distance**divergence**KL-divergence**Metric and Connections Induced by Divergence**(Eguchi) Riemannian metric affine connections**Dually flat manifoldexponential family; mixture family; x:**discrete**Manifold with Convex Function**: coordinates : convex function negative entropy energy mathematical programming, control systems physics, engineering, vision, economics**Riemannian metric and flatness**(affine structure) Bregman divergence Flatness (affine) : geodesic (not Levi-Civita)**Legendre Transformation**one-to-one**Two affine coordinate systems**: geodesic (e-geodesic) : dual geodesic (m-geodesic) “dually orthogonal”**Pythagorean Theorem**(dually flat manifold) Euclidean space: self-dual**Projection Theorem**m-geodesic e-geodesic**Projection Theorem**Q = m-geodesic projection of P to M Q’ = e-geodesic projection of P to M**Information Geometry**Dually flat manifold; curved submanifold convex potential functions Euclidean space : self-dual Probability distributions Exponential family : : negentropy**Two Types of DivergenceInvariant divergence (Chentsov,**Csiszar) f-divergence: Fisher- structureFlat divergence (Bregman) – convex functionKL-divergence belongs to both classes: flat and invariant**KL-divergence**divergence : space of probability distributions invariance dually flat space Flat divergence invariant divergence convex functions Bregman F-divergence Fisher inf metric Alpha connection**Space of positive measures :**vectors, matrices, arrays f-divergence Bregman divergence α-divergence**structure**-Entropy-- Tsallis Shannon entropy Generalized log**conformal transformation**-Fisher information**Applications of Information GeometryStatistical**InferenceMachine Learning and AIComputer VisionConvex ProgrammingSignal Processing (ICA; Sparse)Information Theory, Systems TheoryQuantum Information Geometry**Applications to Statistics**curved exponential family: : estimator**High-Order Asymptotics**:Cramér-Rao: linear theory quadratic approximation :**Semiparametric Statistical Model**y linear relation x mle, least square, total least square**semiparametric**Statistical Model