Discrete time Markov Chain

1 / 41

# Discrete time Markov Chain - PowerPoint PPT Presentation

Discrete time Markov Chain. G.U. Hwang Next Generation Communication Networks Lab. Department of Mathematical Sciences KAIST. Definition.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about 'Discrete time Markov Chain' - kayo

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### Discrete time Markov Chain

G.U. Hwang

Next Generation Communication Networks Lab.

Department of Mathematical Sciences

KAIST

Definition
• The sequence of R.V.s X0, X1, X2,  with a countable state space S is said to be a discrete time Markov chain (DTMC) if it satisfies the Markov Property:

for any ik2 S, k=0,1,,n-1 and i, j 2 S.

• Time homogeneous DTMC : P{Xn+1 = j | Xn = i} is independent of n.

Next Generation Communication NetworksLab.

Transition Probability Matrices
• One step transition probability matrix P
• P = (pij) where pij = P{Xn+1 = j | Xn = i }
• The matrix P is nonnegative and stochastic, i.e.,

pij¸ 0 and j2 S pij = 1

• n step transition probability matrixP(n) = (p(n)ij)
• pij(n) = P{Xn = j | X0 = i}

Next Generation Communication NetworksLab.

For a DTMC, the initial distribution and the matrix P uniquely determine the future behavior of the DTMC because

Next Generation Communication NetworksLab.

Example: A general random walk
• Let Xi be i.i.d. R.V.s with P{X1 = j} = aj, j=0,1,.
• Let S0 = 0, Sn = k=1n Xk. Then {Sn, n¸ 1} is a DTMC because

Next Generation Communication NetworksLab.

Chapman - Kolmogorov's equation
• Chapman - Kolmogorov's theorem

pij(n+m) = k2 S pik(n) pkj(m)

proof:

Next Generation Communication NetworksLab.

Using the chapman-Kolmogorov’s theorem we get

i.e., the n-th power of the one step transition matrix P is, in fact, the n step transition matrix.

Next Generation Communication NetworksLab.

Example
• Find the distribution of X4 where {Xn} (Xn2 S = {1,2}) forms a DTMC with initial distribution P{X0 = 1}=1 and one step transition probability P as follows:

sol:

Next Generation Communication NetworksLab.

Analysis of a DTMC
• When a communication system can be modeled by a DTMC with P and S = {0,1,2}, what happens?

A sample path of a DTMC

transient

period

stationary period

Next Generation Communication NetworksLab.

The stationary probabilities
• The state space S ={0,1, }
• The stationary probability vector (distribution) 
• a row vector  = (0, 1, ) is called a stationary probability vector of a DTMC with transition matrix P if it satisfies

Next Generation Communication NetworksLab.

Does the stationary distribution always exist?

0

1

2

Next Generation Communication NetworksLab.

Does the stationary distribution always exist?

0

1

2

3

…..

Next Generation Communication NetworksLab.

The key question:
• When does the stationary vector exist?
• to answer the question, we need to classify DTMCs according to its probabilistic properties as
• irreducibility
• recurrence
• positive recurrence and null recurrence
• periodic and aperiodic

Next Generation Communication NetworksLab.

Irreducible DTMC
• state i can reach state j if there exists n¸ 0 s.t. pij(n) > 0.
• In this case we write i ! j.
• If i! j and j! i, then we say i and j communicate and write i \$ j.

i

k

h

j

…..

g

f

…..

Next Generation Communication NetworksLab.

\$ is an equivalent relation, that is,
• i \$ i
• i \$ j iff j \$ i
• if i \$ j and j \$ k, then i \$ k
• Definition of irreducibility
• A DTMC is irreducible if its state space consists of a single equivalent class, i.e., for any i, j 2 S we have i \$ j.

Next Generation Communication NetworksLab.

A closed set
• a set A of states is closed if no one step transition is possible from any state in A to any state in AC, i.e., for every pair of states i 2 A and j 2 AC, pij = 0
• An absorbing state
• A single state which alone form a closed set is called an absorbing state
• if state i is an absorbing state, pii = 1.

Next Generation Communication NetworksLab.

Recurrence
• The hitting time (i) of state i:
• For a state i 2 S, (i) = inf {n¸ 1| Xn = i}, i.e., (i) is the first visiting time of the DTMC {Xn} to state i.
• When no such n exists, (i) = 1 by convention.
• The number Ni of visits to state i:
• Ni = n=11I{Xn=i} where IA is an indicator function which is defined by 1 if the event A occurs and by 0 otherwise.
• Clearly, {Ni > 0} = {(i) < 1}.

Next Generation Communication NetworksLab.

Probability mass function of (i)
• Define

fji (n) = P{ (i)=n | X0 = j }

• Then

fji = P{the DTMC ever visits state i | X0 = j }

= n = 1 1 fji (n)

• Definition of recurrence of state i
• state i is said to be recurrent

if P{(i) < 1 |X0 = i} = 1, and transient otherwise.

• that is, state i is recurrent if fii =1, and transient if fii <1.

Next Generation Communication NetworksLab.

Observations
• Once the DTMC revisits a recurrent state i starting from state i, by the Markov property it has the same probabilistic behavior as before. Hence, state i is visited infinitely.
• Further, for a recurrent state i and X0 = i a.s. successive visits to state i can be viewed as renewals and {fii (n) | n¸ 1} is the p.m.f. of the inter-renewal times.

Next Generation Communication NetworksLab.

For a transient state i, starting from state i the DTMC revisits state i with probability fii (< 1) and it never enters state i with probability 1- fii. Therefore, by Markov property we have

P{the DTMC visits state i n times} = (fii)n (1- fii ), n¸ 1

i.e., the number of visits to state i is according to a geometric distribution.

Next Generation Communication NetworksLab.

The following are equivalent:
• state i is recurrent
• Ni = 1 with probability 1 provided that X0 = i
• E[Ni|X0 = i] = E[n=11 I{Xn = i}|X0 = i]

= n=11 pii(n) = 1

• The following are equivalent:
• state i is transient
• Ni < 1 with probability 1 provided that X0 = i
• E[Ni|X0 = i] = n=11 pii(n) < 1

Next Generation Communication NetworksLab.

If i \$ j and i is recurrent, then j is also recurrent.

Proof: Since i \$ j, there exist n, m ¸ 0 s.t. pij(n) > 0 and pji(m) > 0. Then,

which comes from the recurrence of state i. This completes the proof.

Next Generation Communication NetworksLab.

Positive recurrence
• For a recurrent state i,
• if E[(i) | X0 = i] < 1, state i is called positive recurrent.
• if E[(i) | X0 = i] = 1, then state i is called null recurrent.
• Note that, E[(i) | X0 = i] = 1 for a transient state i.

Next Generation Communication NetworksLab.

A stationary measure
• a vector q = (q0, q1,) is called a stationary measure of a M.C. with transition matrix P if
• q 0
• all qi are finite, i.e., qi < 1
• q P = q

Next Generation Communication NetworksLab.

Let i2 S be a recurrent state. Then a stationary measure (q0, q1,) can be defined by

Next Generation Communication NetworksLab.

• However, it can be shown that the stationary measure (q0, q1,) is unique up to a constant multiplication.

Next Generation Communication NetworksLab.

Note that
• So, if state i is positive recurrent, by normalizing the stationary measure (q0, q1,) , we have a stationary distribution (p0, p1,) for the DTMC {Xn}

Next Generation Communication NetworksLab.

The stationary distribution
• The existence of the stationary distribution
• If the DTMC is irreducible and positive recurrent, the stationary distribution exists and is given by
• Note that the above equation is not used in numerical computation. In fact, we use
• For numerical algorithms, we will see them shortly.

Next Generation Communication NetworksLab.

If i \$ j and i is ㅔpositive recurrent, then j is also positive recurrent.

proof: Let (i) and (j) be two stationary measures by using state i and j, respectively. Since both are stationary measures, there exists a constant c (< 1) such that (j) = c (i). So, By summing over all elements in both sides, we get

(j) e = c (i) e < 1.

Hence, state j is also positive recurrent.

Next Generation Communication NetworksLab.

An irreducible DTMC with a finite state space S is always positive recurrent.

proof. Let Ni be the total number of visits to state i. Since i Ni should be 1 and the state space S is finite, for at least one state, say k, we have Nk = 1, which means state k is recurrent. Consequently, all states are recurrent because of the irreducibility of the DTMC.

Now, since the stationary measure has a vector  of finite size, the sum ii should be finite, i.e., all states are positive recurrent.

Next Generation Communication NetworksLab.

Criterion for recurrence
• Suppose that the DTMC is irreducible and let i be some fixed state.
• Then the chain is transient if and only if there is a bounded non-zero real valued function

h:S-{i} ! R satisfying

h(j) = k i pjk h(k), j i.

Next Generation Communication NetworksLab.

Criteria for positive recurrence
• (Pakes' lemma) Let a DTMC {Xn} be irreducible and aperiodic with state space S={0,1,}. Then {Xn} is positive recurrent if the following are satisfied:
• |E[Xn+1-Xn|Xn=i]| < 1 for i=0,1,2,
• limsupi!1 E[Xn+1-Xn|Xn = i] < 0, i.e., there exist positive numbers  and N such that E[Xn+1-Xn|Xn = i] < - for all i¸ N

Next Generation Communication NetworksLab.

Limiting distribution
• The definition of the limiting probabilities of {Xn}

i = limn!1 P{Xn = i|X0 = j}

• when does the limiting probabilities exist?
• consider a DTMC with transition matrix P
• For state 0, {Xn} can visits state 0 only at slots with even numbers, i.e.,

P{X2k+1 = 0} = 0

which means that no limiting probability exists.

Next Generation Communication NetworksLab.

Periodicity
• Definition of the periodicity
• For state i, the span d(i) of state i is defined by

d(i) = g.c.d. {n¸ 1 | pii(n) > 0}.

• If d(i) = 1, we say state i is an aperiodic state.
• If i \$ j, then d(i) = d(j).
• If pii > 0 for some state i in an irreducible DTMC, the chain is aperiodic.

Next Generation Communication NetworksLab.

Ergodicity of a DTMC
• The concept of ergodicity

Time average

T

Ensemble average

= E[f(X(T))]

T

Next Generation Communication NetworksLab.

Time average of a DTMC
• For a state i, recall that fii(n), n¸ 1 form the p.m.f. for the length between consecutive visits to state i (i.e., renewals), and (i) is the length of a renewal.
• When X0 =i, Ni (n) denotes the number of visits to state i (i.e., renewals) in [1,n]. Then by the elementary renewal theorem,

Next Generation Communication NetworksLab.

A DTMC {Xn} is said to be ergodic if it is irreducible and all the states are positive recurrent and aperiodic.
• In an irreducible and aperiodic Markov chain, there always exist the limits

which are the limiting probabilities of the DTMC.

Next Generation Communication NetworksLab.

The properties of an irreducible and aperiodic DTMC
• When state i is transient or null recurrent, the limit i = 0.
• When state i is positive recurrent (and hence all the states are positive recurrent), the limiting distribution is, in fact, the stationary distribution of the DTMC. Therefore, the limiting distribution also satisfies  =  P, i2 Si = 1 (or  e = 1), where e is a column vector all of whose elements are equal to 1.
• Since i = limn!1 P{Xn = i|X0 = j} = limn!1 pji(n), we have Pn! e.

Next Generation Communication NetworksLab.

stationary vs. limiting Distribution
• consider a DTMC with transition matrix P

the DTMC is periodic with period 2,

but it has the stationary distribution

Next Generation Communication NetworksLab.

Computation of the limiting distribution
• Iterative algorithm
• (0) be a given initial distribution
• (n) = (0) Pn, the distribution of Xn.
• Then ¼(n) if |(n) - (n-1)| <  for a sufficiently small >0.
• Pn! e
• Eigenvector of P
•  is, in fact, the eigenvector of the matrix P corresponding to an eigenvalue 1.

Next Generation Communication NetworksLab.

When the transition matrix P is of dimension k.
• Let E be a square matrix of dimension k with all the elements equal to 1. Note that  E = et.

(et = the transpose of e)

• from  =  P we have  (P+E-I) = et where I denotes the identity matrix of dimension k.
• since the matrix P+E-I is invertible,

 = et(P+E-I)-1.

• Note that the solution of the above equation automatically satisfies the normalizing condition  e = 1.

Next Generation Communication NetworksLab.