1 / 41

# Discrete time Markov Chain - PowerPoint PPT Presentation

Discrete time Markov Chain. G.U. Hwang Next Generation Communication Networks Lab. Department of Mathematical Sciences KAIST. Definition.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about ' Discrete time Markov Chain' - kayo

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### Discrete time Markov Chain

G.U. Hwang

Next Generation Communication Networks Lab.

Department of Mathematical Sciences

KAIST

• The sequence of R.V.s X0, X1, X2,  with a countable state space S is said to be a discrete time Markov chain (DTMC) if it satisfies the Markov Property:

for any ik2 S, k=0,1,,n-1 and i, j 2 S.

• Time homogeneous DTMC : P{Xn+1 = j | Xn = i} is independent of n.

Next Generation Communication NetworksLab.

• One step transition probability matrix P

• P = (pij) where pij = P{Xn+1 = j | Xn = i }

• The matrix P is nonnegative and stochastic, i.e.,

pij¸ 0 and j2 S pij = 1

• n step transition probability matrixP(n) = (p(n)ij)

• pij(n) = P{Xn = j | X0 = i}

Next Generation Communication NetworksLab.

Next Generation Communication NetworksLab.

• Let Xi be i.i.d. R.V.s with P{X1 = j} = aj, j=0,1,.

• Let S0 = 0, Sn = k=1n Xk. Then {Sn, n¸ 1} is a DTMC because

Next Generation Communication NetworksLab.

• Chapman - Kolmogorov's theorem

pij(n+m) = k2 S pik(n) pkj(m)

proof:

Next Generation Communication NetworksLab.

Next Generation Communication NetworksLab.

• Find the distribution of X4 where {Xn} (Xn2 S = {1,2}) forms a DTMC with initial distribution P{X0 = 1}=1 and one step transition probability P as follows:

sol:

Next Generation Communication NetworksLab.

• When a communication system can be modeled by a DTMC with P and S = {0,1,2}, what happens?

A sample path of a DTMC

transient

period

stationary period

Next Generation Communication NetworksLab.

• The state space S ={0,1, }

• The stationary probability vector (distribution) 

• a row vector  = (0, 1, ) is called a stationary probability vector of a DTMC with transition matrix P if it satisfies

Next Generation Communication NetworksLab.

0

1

2

Next Generation Communication NetworksLab.

0

1

2

3

…..

Next Generation Communication NetworksLab.

• The key question:

• When does the stationary vector exist?

• to answer the question, we need to classify DTMCs according to its probabilistic properties as

• irreducibility

• recurrence

• positive recurrence and null recurrence

• periodic and aperiodic

Next Generation Communication NetworksLab.

• state i can reach state j if there exists n¸ 0 s.t. pij(n) > 0.

• In this case we write i ! j.

• If i! j and j! i, then we say i and j communicate and write i \$ j.

i

k

h

j

…..

g

f

…..

Next Generation Communication NetworksLab.

• \$ is an equivalent relation, that is,

• i \$ i

• i \$ j iff j \$ i

• if i \$ j and j \$ k, then i \$ k

• Definition of irreducibility

• A DTMC is irreducible if its state space consists of a single equivalent class, i.e., for any i, j 2 S we have i \$ j.

Next Generation Communication NetworksLab.

• A closed set

• a set A of states is closed if no one step transition is possible from any state in A to any state in AC, i.e., for every pair of states i 2 A and j 2 AC, pij = 0

• An absorbing state

• A single state which alone form a closed set is called an absorbing state

• if state i is an absorbing state, pii = 1.

Next Generation Communication NetworksLab.

• The hitting time (i) of state i:

• For a state i 2 S, (i) = inf {n¸ 1| Xn = i}, i.e., (i) is the first visiting time of the DTMC {Xn} to state i.

• When no such n exists, (i) = 1 by convention.

• The number Ni of visits to state i:

• Ni = n=11I{Xn=i} where IA is an indicator function which is defined by 1 if the event A occurs and by 0 otherwise.

• Clearly, {Ni > 0} = {(i) < 1}.

Next Generation Communication NetworksLab.

• Define

fji (n) = P{ (i)=n | X0 = j }

• Then

fji = P{the DTMC ever visits state i | X0 = j }

= n = 1 1 fji (n)

• Definition of recurrence of state i

• state i is said to be recurrent

if P{(i) < 1 |X0 = i} = 1, and transient otherwise.

• that is, state i is recurrent if fii =1, and transient if fii <1.

Next Generation Communication NetworksLab.

• Once the DTMC revisits a recurrent state i starting from state i, by the Markov property it has the same probabilistic behavior as before. Hence, state i is visited infinitely.

• Further, for a recurrent state i and X0 = i a.s. successive visits to state i can be viewed as renewals and {fii (n) | n¸ 1} is the p.m.f. of the inter-renewal times.

Next Generation Communication NetworksLab.

• For a transient state i, starting from state i the DTMC revisits state i with probability fii (< 1) and it never enters state i with probability 1- fii. Therefore, by Markov property we have

P{the DTMC visits state i n times} = (fii)n (1- fii ), n¸ 1

i.e., the number of visits to state i is according to a geometric distribution.

Next Generation Communication NetworksLab.

• The following are equivalent: revisits state i with probability f

• state i is recurrent

• Ni = 1 with probability 1 provided that X0 = i

• E[Ni|X0 = i] = E[n=11 I{Xn = i}|X0 = i]

= n=11 pii(n) = 1

• The following are equivalent:

• state i is transient

• Ni < 1 with probability 1 provided that X0 = i

• E[Ni|X0 = i] = n=11 pii(n) < 1

Next Generation Communication NetworksLab.

• If i revisits state i with probability f\$ j and i is recurrent, then j is also recurrent.

Proof: Since i \$ j, there exist n, m ¸ 0 s.t. pij(n) > 0 and pji(m) > 0. Then,

which comes from the recurrence of state i. This completes the proof.

Next Generation Communication NetworksLab.

Positive recurrence revisits state i with probability f

• For a recurrent state i,

• if E[(i) | X0 = i] < 1, state i is called positive recurrent.

• if E[(i) | X0 = i] = 1, then state i is called null recurrent.

• Note that, E[(i) | X0 = i] = 1 for a transient state i.

Next Generation Communication NetworksLab.

A stationary measure revisits state i with probability f

• a vector q = (q0, q1,) is called a stationary measure of a M.C. with transition matrix P if

• q 0

• all qi are finite, i.e., qi < 1

• q P = q

Next Generation Communication NetworksLab.

• Let i revisits state i with probability f2 S be a recurrent state. Then a stationary measure (q0, q1,) can be defined by

Next Generation Communication NetworksLab.

• The stationary vector revisits state i with probability fq defined above depends on the chosen state i.

• However, it can be shown that the stationary measure (q0, q1,) is unique up to a constant multiplication.

Next Generation Communication NetworksLab.

• Note that revisits state i with probability f

• So, if state i is positive recurrent, by normalizing the stationary measure (q0, q1,) , we have a stationary distribution (p0, p1,) for the DTMC {Xn}

Next Generation Communication NetworksLab.

The stationary distribution revisits state i with probability f

• The existence of the stationary distribution

• If the DTMC is irreducible and positive recurrent, the stationary distribution exists and is given by

• Note that the above equation is not used in numerical computation. In fact, we use

• For numerical algorithms, we will see them shortly.

Next Generation Communication NetworksLab.

• If i revisits state i with probability f\$ j and i is ㅔpositive recurrent, then j is also positive recurrent.

proof: Let (i) and (j) be two stationary measures by using state i and j, respectively. Since both are stationary measures, there exists a constant c (< 1) such that (j) = c (i). So, By summing over all elements in both sides, we get

(j) e = c (i) e < 1.

Hence, state j is also positive recurrent.

Next Generation Communication NetworksLab.

• An irreducible DTMC with a finite state space S is always positive recurrent.

proof. Let Ni be the total number of visits to state i. Since i Ni should be 1 and the state space S is finite, for at least one state, say k, we have Nk = 1, which means state k is recurrent. Consequently, all states are recurrent because of the irreducibility of the DTMC.

Now, since the stationary measure has a vector  of finite size, the sum ii should be finite, i.e., all states are positive recurrent.

Next Generation Communication NetworksLab.

Criterion for recurrence positive recurrent.

• Suppose that the DTMC is irreducible and let i be some fixed state.

• Then the chain is transient if and only if there is a bounded non-zero real valued function

h:S-{i} ! R satisfying

h(j) = k i pjk h(k), j i.

Next Generation Communication NetworksLab.

Criteria for positive recurrence positive recurrent.

• (Pakes' lemma) Let a DTMC {Xn} be irreducible and aperiodic with state space S={0,1,}. Then {Xn} is positive recurrent if the following are satisfied:

• |E[Xn+1-Xn|Xn=i]| < 1 for i=0,1,2,

• limsupi!1 E[Xn+1-Xn|Xn = i] < 0, i.e., there exist positive numbers  and N such that E[Xn+1-Xn|Xn = i] < - for all i¸ N

Next Generation Communication NetworksLab.

Limiting distribution positive recurrent.

• The definition of the limiting probabilities of {Xn}

i = limn!1 P{Xn = i|X0 = j}

• when does the limiting probabilities exist?

• consider a DTMC with transition matrix P

• For state 0, {Xn} can visits state 0 only at slots with even numbers, i.e.,

P{X2k+1 = 0} = 0

which means that no limiting probability exists.

Next Generation Communication NetworksLab.

Periodicity positive recurrent.

• Definition of the periodicity

• For state i, the span d(i) of state i is defined by

d(i) = g.c.d. {n¸ 1 | pii(n) > 0}.

• If d(i) = 1, we say state i is an aperiodic state.

• If i \$ j, then d(i) = d(j).

• If pii > 0 for some state i in an irreducible DTMC, the chain is aperiodic.

Next Generation Communication NetworksLab.

Ergodicity of a DTMC positive recurrent.

• The concept of ergodicity

Time average

T

Ensemble average

= E[f(X(T))]

T

Next Generation Communication NetworksLab.

• Time average of a DTMC positive recurrent.

• For a state i, recall that fii(n), n¸ 1 form the p.m.f. for the length between consecutive visits to state i (i.e., renewals), and (i) is the length of a renewal.

• When X0 =i, Ni (n) denotes the number of visits to state i (i.e., renewals) in [1,n]. Then by the elementary renewal theorem,

Next Generation Communication NetworksLab.

• A DTMC {X positive recurrent.n} is said to be ergodic if it is irreducible and all the states are positive recurrent and aperiodic.

• In an irreducible and aperiodic Markov chain, there always exist the limits

which are the limiting probabilities of the DTMC.

Next Generation Communication NetworksLab.

The properties of an irreducible and aperiodic DTMC positive recurrent.

• When state i is transient or null recurrent, the limit i = 0.

• When state i is positive recurrent (and hence all the states are positive recurrent), the limiting distribution is, in fact, the stationary distribution of the DTMC. Therefore, the limiting distribution also satisfies  =  P, i2 Si = 1 (or  e = 1), where e is a column vector all of whose elements are equal to 1.

• Since i = limn!1 P{Xn = i|X0 = j} = limn!1 pji(n), we have Pn! e.

Next Generation Communication NetworksLab.

stationary vs. limiting Distribution positive recurrent.

• consider a DTMC with transition matrix P

the DTMC is periodic with period 2,

but it has the stationary distribution

Next Generation Communication NetworksLab.

Computation of the limiting distribution positive recurrent.

• Iterative algorithm

• (0) be a given initial distribution

• (n) = (0) Pn, the distribution of Xn.

• Then ¼(n) if |(n) - (n-1)| <  for a sufficiently small >0.

• Pn! e

• Eigenvector of P

•  is, in fact, the eigenvector of the matrix P corresponding to an eigenvalue 1.

Next Generation Communication NetworksLab.

• When the transition matrix P is of dimension k. positive recurrent.

• Let E be a square matrix of dimension k with all the elements equal to 1. Note that  E = et.

(et = the transpose of e)

• from  =  P we have  (P+E-I) = et where I denotes the identity matrix of dimension k.

• since the matrix P+E-I is invertible,

 = et(P+E-I)-1.

• Note that the solution of the above equation automatically satisfies the normalizing condition  e = 1.

Next Generation Communication NetworksLab.