Discrete time Markov Chain. G.U. Hwang Next Generation Communication Networks Lab. Department of Mathematical Sciences KAIST. Definition.
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Discrete time Markov Chain
G.U. Hwang
Next Generation Communication Networks Lab.
Department of Mathematical Sciences
KAIST
for any ik2 S, k=0,1,,n-1 and i, j 2 S.
Next Generation Communication NetworksLab.
pij¸ 0 and j2 S pij = 1
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
pij(n+m) = k2 S pik(n) pkj(m)
proof:
Next Generation Communication NetworksLab.
i.e., the n-th power of the one step transition matrix P is, in fact, the n step transition matrix.
Next Generation Communication NetworksLab.
sol:
Next Generation Communication NetworksLab.
A sample path of a DTMC
transient
period
stationary period
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
0
1
2
Next Generation Communication NetworksLab.
0
1
2
3
…..
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
i
k
h
j
…..
g
f
…..
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
fji (n) = P{ (i)=n | X0 = j }
fji = P{the DTMC ever visits state i | X0 = j }
= n = 1 1 fji (n)
if P{(i) < 1 |X0 = i} = 1, and transient otherwise.
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
P{the DTMC visits state i n times} = (fii)n (1- fii ), n¸ 1
i.e., the number of visits to state i is according to a geometric distribution.
Next Generation Communication NetworksLab.
= n=11 pii(n) = 1
Next Generation Communication NetworksLab.
Proof: Since i $ j, there exist n, m ¸ 0 s.t. pij(n) > 0 and pji(m) > 0. Then,
which comes from the recurrence of state i. This completes the proof.
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
proof: Let (i) and (j) be two stationary measures by using state i and j, respectively. Since both are stationary measures, there exists a constant c (< 1) such that (j) = c (i). So, By summing over all elements in both sides, we get
(j) e = c (i) e < 1.
Hence, state j is also positive recurrent.
Next Generation Communication NetworksLab.
proof. Let Ni be the total number of visits to state i. Since i Ni should be 1 and the state space S is finite, for at least one state, say k, we have Nk = 1, which means state k is recurrent. Consequently, all states are recurrent because of the irreducibility of the DTMC.
Now, since the stationary measure has a vector of finite size, the sum ii should be finite, i.e., all states are positive recurrent.
Next Generation Communication NetworksLab.
h:S-{i} ! R satisfying
h(j) = k i pjk h(k), j i.
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
i = limn!1 P{Xn = i|X0 = j}
P{X2k+1 = 0} = 0
which means that no limiting probability exists.
Next Generation Communication NetworksLab.
d(i) = g.c.d. {n¸ 1 | pii(n) > 0}.
Next Generation Communication NetworksLab.
Time average
T
Ensemble average
= E[f(X(T))]
T
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
which are the limiting probabilities of the DTMC.
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
the DTMC is periodic with period 2,
but it has the stationary distribution
Next Generation Communication NetworksLab.
Next Generation Communication NetworksLab.
(et = the transpose of e)
= et(P+E-I)-1.
Next Generation Communication NetworksLab.