- 122 Views
- Uploaded on
- Presentation posted in: General

Discrete time Markov Chain

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Discrete time Markov Chain

G.U. Hwang

Next Generation Communication Networks Lab.

Department of Mathematical Sciences

KAIST

- The sequence of R.V.s X0, X1, X2, with a countable state space S is said to be a discrete time Markov chain (DTMC) if it satisfies the Markov Property:
for any ik2 S, k=0,1,,n-1 and i, j 2 S.

- Time homogeneous DTMC : P{Xn+1 = j | Xn = i} is independent of n.

Next Generation Communication NetworksLab.

- One step transition probability matrix P
- P = (pij) where pij = P{Xn+1 = j | Xn = i }
- The matrix P is nonnegative and stochastic, i.e.,
pij¸ 0 and j2 S pij = 1

- n step transition probability matrixP(n) = (p(n)ij)
- pij(n) = P{Xn = j | X0 = i}

Next Generation Communication NetworksLab.

- For a DTMC, the initial distribution and the matrix P uniquely determine the future behavior of the DTMC because

Next Generation Communication NetworksLab.

- Let Xi be i.i.d. R.V.s with P{X1 = j} = aj, j=0,1,.
- Let S0 = 0, Sn = k=1n Xk. Then {Sn, n¸ 1} is a DTMC because

Next Generation Communication NetworksLab.

- Chapman - Kolmogorov's theorem
pij(n+m) = k2 S pik(n) pkj(m)

proof:

Next Generation Communication NetworksLab.

- Using the chapman-Kolmogorov’s theorem we get
i.e., the n-th power of the one step transition matrix P is, in fact, the n step transition matrix.

Next Generation Communication NetworksLab.

- Find the distribution of X4 where {Xn} (Xn2 S = {1,2}) forms a DTMC with initial distribution P{X0 = 1}=1 and one step transition probability P as follows:
sol:

Next Generation Communication NetworksLab.

- When a communication system can be modeled by a DTMC with P and S = {0,1,2}, what happens?

A sample path of a DTMC

transient

period

stationary period

Next Generation Communication NetworksLab.

- The state space S ={0,1, }
- The stationary probability vector (distribution)
- a row vector = (0, 1, ) is called a stationary probability vector of a DTMC with transition matrix P if it satisfies

Next Generation Communication NetworksLab.

- Does the stationary distribution always exist?

0

1

2

Next Generation Communication NetworksLab.

- Does the stationary distribution always exist?

0

1

2

3

…..

Next Generation Communication NetworksLab.

- The key question:
- When does the stationary vector exist?

- to answer the question, we need to classify DTMCs according to its probabilistic properties as
- irreducibility
- recurrence
- positive recurrence and null recurrence
- periodic and aperiodic

Next Generation Communication NetworksLab.

- state i can reach state j if there exists n¸ 0 s.t. pij(n) > 0.
- In this case we write i ! j.
- If i! j and j! i, then we say i and j communicate and write i $ j.

i

k

h

j

…..

g

f

…..

Next Generation Communication NetworksLab.

- $ is an equivalent relation, that is,
- i $ i
- i $ j iff j $ i
- if i $ j and j $ k, then i $ k

- Definition of irreducibility
- A DTMC is irreducible if its state space consists of a single equivalent class, i.e., for any i, j 2 S we have i $ j.

Next Generation Communication NetworksLab.

- A closed set
- a set A of states is closed if no one step transition is possible from any state in A to any state in AC, i.e., for every pair of states i 2 A and j 2 AC, pij = 0

- An absorbing state
- A single state which alone form a closed set is called an absorbing state
- if state i is an absorbing state, pii = 1.

Next Generation Communication NetworksLab.

- The hitting time (i) of state i:
- For a state i 2 S, (i) = inf {n¸ 1| Xn = i}, i.e., (i) is the first visiting time of the DTMC {Xn} to state i.
- When no such n exists, (i) = 1 by convention.

- The number Ni of visits to state i:
- Ni = n=11I{Xn=i} where IA is an indicator function which is defined by 1 if the event A occurs and by 0 otherwise.
- Clearly, {Ni > 0} = {(i) < 1}.

Next Generation Communication NetworksLab.

- Define
fji (n) = P{ (i)=n | X0 = j }

- Then
fji = P{the DTMC ever visits state i | X0 = j }

= n = 1 1 fji (n)

- Definition of recurrence of state i
- state i is said to be recurrent
if P{(i) < 1 |X0 = i} = 1, and transient otherwise.

- that is, state i is recurrent if fii =1, and transient if fii <1.

- state i is said to be recurrent

Next Generation Communication NetworksLab.

- Once the DTMC revisits a recurrent state i starting from state i, by the Markov property it has the same probabilistic behavior as before. Hence, state i is visited infinitely.
- Further, for a recurrent state i and X0 = i a.s. successive visits to state i can be viewed as renewals and {fii (n) | n¸ 1} is the p.m.f. of the inter-renewal times.

Next Generation Communication NetworksLab.

- For a transient state i, starting from state i the DTMC revisits state i with probability fii (< 1) and it never enters state i with probability 1- fii. Therefore, by Markov property we have
P{the DTMC visits state i n times} = (fii)n (1- fii ), n¸ 1

i.e., the number of visits to state i is according to a geometric distribution.

Next Generation Communication NetworksLab.

- The following are equivalent:
- state i is recurrent
- Ni = 1 with probability 1 provided that X0 = i
- E[Ni|X0 = i] = E[n=11 I{Xn = i}|X0 = i]
= n=11 pii(n) = 1

- The following are equivalent:
- state i is transient
- Ni < 1 with probability 1 provided that X0 = i
- E[Ni|X0 = i] = n=11 pii(n) < 1

Next Generation Communication NetworksLab.

- If i $ j and i is recurrent, then j is also recurrent.
Proof: Since i $ j, there exist n, m ¸ 0 s.t. pij(n) > 0 and pji(m) > 0. Then,

which comes from the recurrence of state i. This completes the proof.

Next Generation Communication NetworksLab.

- For a recurrent state i,
- if E[(i) | X0 = i] < 1, state i is called positive recurrent.
- if E[(i) | X0 = i] = 1, then state i is called null recurrent.

- Note that, E[(i) | X0 = i] = 1 for a transient state i.

Next Generation Communication NetworksLab.

- a vector q = (q0, q1,) is called a stationary measure of a M.C. with transition matrix P if
- q 0
- all qi are finite, i.e., qi < 1
- q P = q

Next Generation Communication NetworksLab.

- Let i2 S be a recurrent state. Then a stationary measure (q0, q1,) can be defined by

Next Generation Communication NetworksLab.

- The stationary vector q defined above depends on the chosen state i.
- However, it can be shown that the stationary measure (q0, q1,) is unique up to a constant multiplication.

Next Generation Communication NetworksLab.

- Note that
- So, if state i is positive recurrent, by normalizing the stationary measure (q0, q1,) , we have a stationary distribution (p0, p1,) for the DTMC {Xn}

Next Generation Communication NetworksLab.

- The existence of the stationary distribution
- If the DTMC is irreducible and positive recurrent, the stationary distribution exists and is given by
- Note that the above equation is not used in numerical computation. In fact, we use
- For numerical algorithms, we will see them shortly.

Next Generation Communication NetworksLab.

- If i $ j and i is ㅔpositive recurrent, then j is also positive recurrent.
proof: Let (i) and (j) be two stationary measures by using state i and j, respectively. Since both are stationary measures, there exists a constant c (< 1) such that (j) = c (i). So, By summing over all elements in both sides, we get

(j) e = c (i) e < 1.

Hence, state j is also positive recurrent.

Next Generation Communication NetworksLab.

- An irreducible DTMC with a finite state space S is always positive recurrent.
proof. Let Ni be the total number of visits to state i. Since i Ni should be 1 and the state space S is finite, for at least one state, say k, we have Nk = 1, which means state k is recurrent. Consequently, all states are recurrent because of the irreducibility of the DTMC.

Now, since the stationary measure has a vector of finite size, the sum ii should be finite, i.e., all states are positive recurrent.

Next Generation Communication NetworksLab.

- Suppose that the DTMC is irreducible and let i be some fixed state.
- Then the chain is transient if and only if there is a bounded non-zero real valued function
h:S-{i} ! R satisfying

h(j) = k i pjk h(k), j i.

Next Generation Communication NetworksLab.

- (Pakes' lemma) Let a DTMC {Xn} be irreducible and aperiodic with state space S={0,1,}. Then {Xn} is positive recurrent if the following are satisfied:
- |E[Xn+1-Xn|Xn=i]| < 1 for i=0,1,2,
- limsupi!1 E[Xn+1-Xn|Xn = i] < 0, i.e., there exist positive numbers and N such that E[Xn+1-Xn|Xn = i] < - for all i¸ N

Next Generation Communication NetworksLab.

- The definition of the limiting probabilities of {Xn}
i = limn!1 P{Xn = i|X0 = j}

- when does the limiting probabilities exist?
- consider a DTMC with transition matrix P
- For state 0, {Xn} can visits state 0 only at slots with even numbers, i.e.,
P{X2k+1 = 0} = 0

which means that no limiting probability exists.

Next Generation Communication NetworksLab.

- Definition of the periodicity
- For state i, the span d(i) of state i is defined by
d(i) = g.c.d. {n¸ 1 | pii(n) > 0}.

- If d(i) = 1, we say state i is an aperiodic state.

- For state i, the span d(i) of state i is defined by
- If i $ j, then d(i) = d(j).
- If pii > 0 for some state i in an irreducible DTMC, the chain is aperiodic.

Next Generation Communication NetworksLab.

- The concept of ergodicity

Time average

T

Ensemble average

= E[f(X(T))]

T

Next Generation Communication NetworksLab.

- Time average of a DTMC
- For a state i, recall that fii(n), n¸ 1 form the p.m.f. for the length between consecutive visits to state i (i.e., renewals), and (i) is the length of a renewal.
- When X0 =i, Ni (n) denotes the number of visits to state i (i.e., renewals) in [1,n]. Then by the elementary renewal theorem,

Next Generation Communication NetworksLab.

- A DTMC {Xn} is said to be ergodic if it is irreducible and all the states are positive recurrent and aperiodic.
- In an irreducible and aperiodic Markov chain, there always exist the limits
which are the limiting probabilities of the DTMC.

Next Generation Communication NetworksLab.

- When state i is transient or null recurrent, the limit i = 0.
- When state i is positive recurrent (and hence all the states are positive recurrent), the limiting distribution is, in fact, the stationary distribution of the DTMC. Therefore, the limiting distribution also satisfies = P, i2 Si = 1 (or e = 1), where e is a column vector all of whose elements are equal to 1.
- Since i = limn!1 P{Xn = i|X0 = j} = limn!1 pji(n), we have Pn! e.

Next Generation Communication NetworksLab.

- consider a DTMC with transition matrix P

the DTMC is periodic with period 2,

but it has the stationary distribution

Next Generation Communication NetworksLab.

- Iterative algorithm
- (0) be a given initial distribution
- (n) = (0) Pn, the distribution of Xn.
- Then ¼(n) if |(n) - (n-1)| < for a sufficiently small >0.

- Pn! e
- Eigenvector of P
- is, in fact, the eigenvector of the matrix P corresponding to an eigenvalue 1.

Next Generation Communication NetworksLab.

- When the transition matrix P is of dimension k.
- Let E be a square matrix of dimension k with all the elements equal to 1. Note that E = et.
(et = the transpose of e)

- from = P we have (P+E-I) = et where I denotes the identity matrix of dimension k.
- since the matrix P+E-I is invertible,
= et(P+E-I)-1.

- Note that the solution of the above equation automatically satisfies the normalizing condition e = 1.

- Let E be a square matrix of dimension k with all the elements equal to 1. Note that E = et.

Next Generation Communication NetworksLab.