Entropy of Hidden Markov Processes. Or Zuk 1 Ido Kanter 2 Eytan Domany 1 Weizmann Inst. 1 Bar-Ilan Univ. 2. Overview. Introduction Problem Definition Statistical Mechanics approach Cover&Thomas Upper-Bounds Radius of Convergence Related subjects Future Directions.
Or Zuk1 Ido Kanter2 Eytan Domany1
Weizmann Inst.1 Bar-Ilan Univ.2
H is difficult to compute, given as a Lyaponov Exponent (which is hard to compute generally.) [Jacquet et al 04]
p -> 0 , p -> ½ ( fixed)
-> 0 , -> ½ (p fixed)
[Ordentlich&Weissman 04] study several regimes.
We concentrate on the ‘small noise regime’ -> 0.
Solution can be given as a power-series in :
First, observe the Markovian Property :
Perform Change of Variables :
Summing, we get :
Computing the Entropy (low-temperature/high-field expansion) :
It is known (Cover & Thomas 1991) :
been expected ! For n (K+3)/2 they become constant.
We therefore have :
When is our approximation good ?
Instructive : Compare to the I.I.D. model
For HMP, the limit is unknown. We used the fit :
Exponentially decaying with n.