This presentation is the property of its rightful owner.
1 / 41

# Discrete time Markov Chain PowerPoint PPT Presentation

Discrete time Markov Chain. G.U. Hwang Next Generation Communication Networks Lab. Department of Mathematical Sciences KAIST. Definition.

Discrete time Markov Chain

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

## Discrete time Markov Chain

G.U. Hwang

Next Generation Communication Networks Lab.

Department of Mathematical Sciences

KAIST

### Definition

• The sequence of R.V.s X0, X1, X2,  with a countable state space S is said to be a discrete time Markov chain (DTMC) if it satisfies the Markov Property:

for any ik2 S, k=0,1,,n-1 and i, j 2 S.

• Time homogeneous DTMC : P{Xn+1 = j | Xn = i} is independent of n.

Next Generation Communication NetworksLab.

### Transition Probability Matrices

• One step transition probability matrix P

• P = (pij) where pij = P{Xn+1 = j | Xn = i }

• The matrix P is nonnegative and stochastic, i.e.,

pij¸ 0 and j2 S pij = 1

• n step transition probability matrixP(n) = (p(n)ij)

• pij(n) = P{Xn = j | X0 = i}

Next Generation Communication NetworksLab.

• For a DTMC, the initial distribution and the matrix P uniquely determine the future behavior of the DTMC because

Next Generation Communication NetworksLab.

### Example: A general random walk

• Let Xi be i.i.d. R.V.s with P{X1 = j} = aj, j=0,1,.

• Let S0 = 0, Sn = k=1n Xk. Then {Sn, n¸ 1} is a DTMC because

Next Generation Communication NetworksLab.

### Chapman - Kolmogorov's equation

• Chapman - Kolmogorov's theorem

pij(n+m) = k2 S pik(n) pkj(m)

proof:

Next Generation Communication NetworksLab.

• Using the chapman-Kolmogorov’s theorem we get

i.e., the n-th power of the one step transition matrix P is, in fact, the n step transition matrix.

Next Generation Communication NetworksLab.

### Example

• Find the distribution of X4 where {Xn} (Xn2 S = {1,2}) forms a DTMC with initial distribution P{X0 = 1}=1 and one step transition probability P as follows:

sol:

Next Generation Communication NetworksLab.

### Analysis of a DTMC

• When a communication system can be modeled by a DTMC with P and S = {0,1,2}, what happens?

A sample path of a DTMC

transient

period

stationary period

Next Generation Communication NetworksLab.

### The stationary probabilities

• The state space S ={0,1, }

• The stationary probability vector (distribution) 

• a row vector  = (0, 1, ) is called a stationary probability vector of a DTMC with transition matrix P if it satisfies

Next Generation Communication NetworksLab.

• Does the stationary distribution always exist?

0

1

2

Next Generation Communication NetworksLab.

• Does the stationary distribution always exist?

0

1

2

3

…..

Next Generation Communication NetworksLab.

• The key question:

• When does the stationary vector exist?

• to answer the question, we need to classify DTMCs according to its probabilistic properties as

• irreducibility

• recurrence

• positive recurrence and null recurrence

• periodic and aperiodic

Next Generation Communication NetworksLab.

### Irreducible DTMC

• state i can reach state j if there exists n¸ 0 s.t. pij(n) > 0.

• In this case we write i ! j.

• If i! j and j! i, then we say i and j communicate and write i \$ j.

i

k

h

j

…..

g

f

…..

Next Generation Communication NetworksLab.

• \$ is an equivalent relation, that is,

• i \$ i

• i \$ j iff j \$ i

• if i \$ j and j \$ k, then i \$ k

• Definition of irreducibility

• A DTMC is irreducible if its state space consists of a single equivalent class, i.e., for any i, j 2 S we have i \$ j.

Next Generation Communication NetworksLab.

• A closed set

• a set A of states is closed if no one step transition is possible from any state in A to any state in AC, i.e., for every pair of states i 2 A and j 2 AC, pij = 0

• An absorbing state

• A single state which alone form a closed set is called an absorbing state

• if state i is an absorbing state, pii = 1.

Next Generation Communication NetworksLab.

### Recurrence

• The hitting time (i) of state i:

• For a state i 2 S, (i) = inf {n¸ 1| Xn = i}, i.e., (i) is the first visiting time of the DTMC {Xn} to state i.

• When no such n exists, (i) = 1 by convention.

• The number Ni of visits to state i:

• Ni = n=11I{Xn=i} where IA is an indicator function which is defined by 1 if the event A occurs and by 0 otherwise.

• Clearly, {Ni > 0} = {(i) < 1}.

Next Generation Communication NetworksLab.

### Probability mass function of (i)

• Define

fji (n) = P{ (i)=n | X0 = j }

• Then

fji = P{the DTMC ever visits state i | X0 = j }

= n = 1 1 fji (n)

• Definition of recurrence of state i

• state i is said to be recurrent

if P{(i) < 1 |X0 = i} = 1, and transient otherwise.

• that is, state i is recurrent if fii =1, and transient if fii <1.

Next Generation Communication NetworksLab.

### Observations

• Once the DTMC revisits a recurrent state i starting from state i, by the Markov property it has the same probabilistic behavior as before. Hence, state i is visited infinitely.

• Further, for a recurrent state i and X0 = i a.s. successive visits to state i can be viewed as renewals and {fii (n) | n¸ 1} is the p.m.f. of the inter-renewal times.

Next Generation Communication NetworksLab.

• For a transient state i, starting from state i the DTMC revisits state i with probability fii (< 1) and it never enters state i with probability 1- fii. Therefore, by Markov property we have

P{the DTMC visits state i n times} = (fii)n (1- fii ), n¸ 1

i.e., the number of visits to state i is according to a geometric distribution.

Next Generation Communication NetworksLab.

• The following are equivalent:

• state i is recurrent

• Ni = 1 with probability 1 provided that X0 = i

• E[Ni|X0 = i] = E[n=11 I{Xn = i}|X0 = i]

= n=11 pii(n) = 1

• The following are equivalent:

• state i is transient

• Ni < 1 with probability 1 provided that X0 = i

• E[Ni|X0 = i] = n=11 pii(n) < 1

Next Generation Communication NetworksLab.

• If i \$ j and i is recurrent, then j is also recurrent.

Proof: Since i \$ j, there exist n, m ¸ 0 s.t. pij(n) > 0 and pji(m) > 0. Then,

which comes from the recurrence of state i. This completes the proof.

Next Generation Communication NetworksLab.

### Positive recurrence

• For a recurrent state i,

• if E[(i) | X0 = i] < 1, state i is called positive recurrent.

• if E[(i) | X0 = i] = 1, then state i is called null recurrent.

• Note that, E[(i) | X0 = i] = 1 for a transient state i.

Next Generation Communication NetworksLab.

### A stationary measure

• a vector q = (q0, q1,) is called a stationary measure of a M.C. with transition matrix P if

• q 0

• all qi are finite, i.e., qi < 1

• q P = q

Next Generation Communication NetworksLab.

• Let i2 S be a recurrent state. Then a stationary measure (q0, q1,) can be defined by

Next Generation Communication NetworksLab.

• The stationary vector q defined above depends on the chosen state i.

• However, it can be shown that the stationary measure (q0, q1,) is unique up to a constant multiplication.

Next Generation Communication NetworksLab.

• Note that

• So, if state i is positive recurrent, by normalizing the stationary measure (q0, q1,) , we have a stationary distribution (p0, p1,) for the DTMC {Xn}

Next Generation Communication NetworksLab.

### The stationary distribution

• The existence of the stationary distribution

• If the DTMC is irreducible and positive recurrent, the stationary distribution exists and is given by

• Note that the above equation is not used in numerical computation. In fact, we use

• For numerical algorithms, we will see them shortly.

Next Generation Communication NetworksLab.

• If i \$ j and i is ㅔpositive recurrent, then j is also positive recurrent.

proof: Let (i) and (j) be two stationary measures by using state i and j, respectively. Since both are stationary measures, there exists a constant c (< 1) such that (j) = c (i). So, By summing over all elements in both sides, we get

(j) e = c (i) e < 1.

Hence, state j is also positive recurrent.

Next Generation Communication NetworksLab.

• An irreducible DTMC with a finite state space S is always positive recurrent.

proof. Let Ni be the total number of visits to state i. Since i Ni should be 1 and the state space S is finite, for at least one state, say k, we have Nk = 1, which means state k is recurrent. Consequently, all states are recurrent because of the irreducibility of the DTMC.

Now, since the stationary measure has a vector  of finite size, the sum ii should be finite, i.e., all states are positive recurrent.

Next Generation Communication NetworksLab.

### Criterion for recurrence

• Suppose that the DTMC is irreducible and let i be some fixed state.

• Then the chain is transient if and only if there is a bounded non-zero real valued function

h:S-{i} ! R satisfying

h(j) = k i pjk h(k), j i.

Next Generation Communication NetworksLab.

### Criteria for positive recurrence

• (Pakes' lemma) Let a DTMC {Xn} be irreducible and aperiodic with state space S={0,1,}. Then {Xn} is positive recurrent if the following are satisfied:

• |E[Xn+1-Xn|Xn=i]| < 1 for i=0,1,2,

• limsupi!1 E[Xn+1-Xn|Xn = i] < 0, i.e., there exist positive numbers  and N such that E[Xn+1-Xn|Xn = i] < - for all i¸ N

Next Generation Communication NetworksLab.

### Limiting distribution

• The definition of the limiting probabilities of {Xn}

i = limn!1 P{Xn = i|X0 = j}

• when does the limiting probabilities exist?

• consider a DTMC with transition matrix P

• For state 0, {Xn} can visits state 0 only at slots with even numbers, i.e.,

P{X2k+1 = 0} = 0

which means that no limiting probability exists.

Next Generation Communication NetworksLab.

### Periodicity

• Definition of the periodicity

• For state i, the span d(i) of state i is defined by

d(i) = g.c.d. {n¸ 1 | pii(n) > 0}.

• If d(i) = 1, we say state i is an aperiodic state.

• If i \$ j, then d(i) = d(j).

• If pii > 0 for some state i in an irreducible DTMC, the chain is aperiodic.

Next Generation Communication NetworksLab.

### Ergodicity of a DTMC

• The concept of ergodicity

Time average

T

Ensemble average

= E[f(X(T))]

T

Next Generation Communication NetworksLab.

• Time average of a DTMC

• For a state i, recall that fii(n), n¸ 1 form the p.m.f. for the length between consecutive visits to state i (i.e., renewals), and (i) is the length of a renewal.

• When X0 =i, Ni (n) denotes the number of visits to state i (i.e., renewals) in [1,n]. Then by the elementary renewal theorem,

Next Generation Communication NetworksLab.

• A DTMC {Xn} is said to be ergodic if it is irreducible and all the states are positive recurrent and aperiodic.

• In an irreducible and aperiodic Markov chain, there always exist the limits

which are the limiting probabilities of the DTMC.

Next Generation Communication NetworksLab.

### The properties of an irreducible and aperiodic DTMC

• When state i is transient or null recurrent, the limit i = 0.

• When state i is positive recurrent (and hence all the states are positive recurrent), the limiting distribution is, in fact, the stationary distribution of the DTMC. Therefore, the limiting distribution also satisfies  =  P, i2 Si = 1 (or  e = 1), where e is a column vector all of whose elements are equal to 1.

• Since i = limn!1 P{Xn = i|X0 = j} = limn!1 pji(n), we have Pn! e.

Next Generation Communication NetworksLab.

### stationary vs. limiting Distribution

• consider a DTMC with transition matrix P

the DTMC is periodic with period 2,

but it has the stationary distribution

Next Generation Communication NetworksLab.

### Computation of the limiting distribution

• Iterative algorithm

• (0) be a given initial distribution

• (n) = (0) Pn, the distribution of Xn.

• Then ¼(n) if |(n) - (n-1)| <  for a sufficiently small >0.

• Pn! e

• Eigenvector of P

•  is, in fact, the eigenvector of the matrix P corresponding to an eigenvalue 1.

Next Generation Communication NetworksLab.

• When the transition matrix P is of dimension k.

• Let E be a square matrix of dimension k with all the elements equal to 1. Note that  E = et.

(et = the transpose of e)

• from  =  P we have  (P+E-I) = et where I denotes the identity matrix of dimension k.

• since the matrix P+E-I is invertible,

 = et(P+E-I)-1.

• Note that the solution of the above equation automatically satisfies the normalizing condition  e = 1.

Next Generation Communication NetworksLab.