Discrete time markov chain
Sponsored Links
This presentation is the property of its rightful owner.
1 / 41

Discrete time Markov Chain PowerPoint PPT Presentation


  • 116 Views
  • Uploaded on
  • Presentation posted in: General

Discrete time Markov Chain. G.U. Hwang Next Generation Communication Networks Lab. Department of Mathematical Sciences KAIST. Definition.

Download Presentation

Discrete time Markov Chain

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Discrete time Markov Chain

G.U. Hwang

Next Generation Communication Networks Lab.

Department of Mathematical Sciences

KAIST


Definition

  • The sequence of R.V.s X0, X1, X2,  with a countable state space S is said to be a discrete time Markov chain (DTMC) if it satisfies the Markov Property:

    for any ik2 S, k=0,1,,n-1 and i, j 2 S.

  • Time homogeneous DTMC : P{Xn+1 = j | Xn = i} is independent of n.

Next Generation Communication NetworksLab.


Transition Probability Matrices

  • One step transition probability matrix P

    • P = (pij) where pij = P{Xn+1 = j | Xn = i }

    • The matrix P is nonnegative and stochastic, i.e.,

      pij¸ 0 and j2 S pij = 1

  • n step transition probability matrixP(n) = (p(n)ij)

    • pij(n) = P{Xn = j | X0 = i}

Next Generation Communication NetworksLab.


  • For a DTMC, the initial distribution and the matrix P uniquely determine the future behavior of the DTMC because

Next Generation Communication NetworksLab.


Example: A general random walk

  • Let Xi be i.i.d. R.V.s with P{X1 = j} = aj, j=0,1,.

  • Let S0 = 0, Sn = k=1n Xk. Then {Sn, n¸ 1} is a DTMC because

Next Generation Communication NetworksLab.


Chapman - Kolmogorov's equation

  • Chapman - Kolmogorov's theorem

    pij(n+m) = k2 S pik(n) pkj(m)

    proof:

Next Generation Communication NetworksLab.


  • Using the chapman-Kolmogorov’s theorem we get

    i.e., the n-th power of the one step transition matrix P is, in fact, the n step transition matrix.

Next Generation Communication NetworksLab.


Example

  • Find the distribution of X4 where {Xn} (Xn2 S = {1,2}) forms a DTMC with initial distribution P{X0 = 1}=1 and one step transition probability P as follows:

    sol:

Next Generation Communication NetworksLab.


Analysis of a DTMC

  • When a communication system can be modeled by a DTMC with P and S = {0,1,2}, what happens?

A sample path of a DTMC

transient

period

stationary period

Next Generation Communication NetworksLab.


The stationary probabilities

  • The state space S ={0,1, }

  • The stationary probability vector (distribution) 

    • a row vector  = (0, 1, ) is called a stationary probability vector of a DTMC with transition matrix P if it satisfies

Next Generation Communication NetworksLab.


  • Does the stationary distribution always exist?

0

1

2

Next Generation Communication NetworksLab.


  • Does the stationary distribution always exist?

0

1

2

3

…..

Next Generation Communication NetworksLab.


  • The key question:

    • When does the stationary vector exist?

  • to answer the question, we need to classify DTMCs according to its probabilistic properties as

    • irreducibility

    • recurrence

    • positive recurrence and null recurrence

    • periodic and aperiodic

Next Generation Communication NetworksLab.


Irreducible DTMC

  • state i can reach state j if there exists n¸ 0 s.t. pij(n) > 0.

  • In this case we write i ! j.

  • If i! j and j! i, then we say i and j communicate and write i $ j.

i

k

h

j

…..

g

f

…..

Next Generation Communication NetworksLab.


  • $ is an equivalent relation, that is,

    • i $ i

    • i $ j iff j $ i

    • if i $ j and j $ k, then i $ k

  • Definition of irreducibility

    • A DTMC is irreducible if its state space consists of a single equivalent class, i.e., for any i, j 2 S we have i $ j.

Next Generation Communication NetworksLab.


  • A closed set

    • a set A of states is closed if no one step transition is possible from any state in A to any state in AC, i.e., for every pair of states i 2 A and j 2 AC, pij = 0

  • An absorbing state

    • A single state which alone form a closed set is called an absorbing state

    • if state i is an absorbing state, pii = 1.

Next Generation Communication NetworksLab.


Recurrence

  • The hitting time (i) of state i:

    • For a state i 2 S, (i) = inf {n¸ 1| Xn = i}, i.e., (i) is the first visiting time of the DTMC {Xn} to state i.

    • When no such n exists, (i) = 1 by convention.

  • The number Ni of visits to state i:

    • Ni = n=11I{Xn=i} where IA is an indicator function which is defined by 1 if the event A occurs and by 0 otherwise.

    • Clearly, {Ni > 0} = {(i) < 1}.

Next Generation Communication NetworksLab.


Probability mass function of (i)

  • Define

    fji (n) = P{ (i)=n | X0 = j }

  • Then

    fji = P{the DTMC ever visits state i | X0 = j }

    = n = 1 1 fji (n)

  • Definition of recurrence of state i

    • state i is said to be recurrent

      if P{(i) < 1 |X0 = i} = 1, and transient otherwise.

    • that is, state i is recurrent if fii =1, and transient if fii <1.

Next Generation Communication NetworksLab.


Observations

  • Once the DTMC revisits a recurrent state i starting from state i, by the Markov property it has the same probabilistic behavior as before. Hence, state i is visited infinitely.

  • Further, for a recurrent state i and X0 = i a.s. successive visits to state i can be viewed as renewals and {fii (n) | n¸ 1} is the p.m.f. of the inter-renewal times.

Next Generation Communication NetworksLab.


  • For a transient state i, starting from state i the DTMC revisits state i with probability fii (< 1) and it never enters state i with probability 1- fii. Therefore, by Markov property we have

    P{the DTMC visits state i n times} = (fii)n (1- fii ), n¸ 1

    i.e., the number of visits to state i is according to a geometric distribution.

Next Generation Communication NetworksLab.


  • The following are equivalent:

    • state i is recurrent

    • Ni = 1 with probability 1 provided that X0 = i

    • E[Ni|X0 = i] = E[n=11 I{Xn = i}|X0 = i]

      = n=11 pii(n) = 1

  • The following are equivalent:

    • state i is transient

    • Ni < 1 with probability 1 provided that X0 = i

    • E[Ni|X0 = i] = n=11 pii(n) < 1

Next Generation Communication NetworksLab.


  • If i $ j and i is recurrent, then j is also recurrent.

    Proof: Since i $ j, there exist n, m ¸ 0 s.t. pij(n) > 0 and pji(m) > 0. Then,

    which comes from the recurrence of state i. This completes the proof.

Next Generation Communication NetworksLab.


Positive recurrence

  • For a recurrent state i,

    • if E[(i) | X0 = i] < 1, state i is called positive recurrent.

    • if E[(i) | X0 = i] = 1, then state i is called null recurrent.

  • Note that, E[(i) | X0 = i] = 1 for a transient state i.

Next Generation Communication NetworksLab.


A stationary measure

  • a vector q = (q0, q1,) is called a stationary measure of a M.C. with transition matrix P if

    • q 0

    • all qi are finite, i.e., qi < 1

    • q P = q

Next Generation Communication NetworksLab.


  • Let i2 S be a recurrent state. Then a stationary measure (q0, q1,) can be defined by

Next Generation Communication NetworksLab.


  • The stationary vector q defined above depends on the chosen state i.

  • However, it can be shown that the stationary measure (q0, q1,) is unique up to a constant multiplication.

Next Generation Communication NetworksLab.


  • Note that

  • So, if state i is positive recurrent, by normalizing the stationary measure (q0, q1,) , we have a stationary distribution (p0, p1,) for the DTMC {Xn}

Next Generation Communication NetworksLab.


The stationary distribution

  • The existence of the stationary distribution

    • If the DTMC is irreducible and positive recurrent, the stationary distribution exists and is given by

    • Note that the above equation is not used in numerical computation. In fact, we use

    • For numerical algorithms, we will see them shortly.

Next Generation Communication NetworksLab.


  • If i $ j and i is ㅔpositive recurrent, then j is also positive recurrent.

    proof: Let (i) and (j) be two stationary measures by using state i and j, respectively. Since both are stationary measures, there exists a constant c (< 1) such that (j) = c (i). So, By summing over all elements in both sides, we get

    (j) e = c (i) e < 1.

    Hence, state j is also positive recurrent.

Next Generation Communication NetworksLab.


  • An irreducible DTMC with a finite state space S is always positive recurrent.

    proof. Let Ni be the total number of visits to state i. Since i Ni should be 1 and the state space S is finite, for at least one state, say k, we have Nk = 1, which means state k is recurrent. Consequently, all states are recurrent because of the irreducibility of the DTMC.

    Now, since the stationary measure has a vector  of finite size, the sum ii should be finite, i.e., all states are positive recurrent.

Next Generation Communication NetworksLab.


Criterion for recurrence

  • Suppose that the DTMC is irreducible and let i be some fixed state.

  • Then the chain is transient if and only if there is a bounded non-zero real valued function

    h:S-{i} ! R satisfying

    h(j) = k i pjk h(k), j i.

Next Generation Communication NetworksLab.


Criteria for positive recurrence

  • (Pakes' lemma) Let a DTMC {Xn} be irreducible and aperiodic with state space S={0,1,}. Then {Xn} is positive recurrent if the following are satisfied:

    • |E[Xn+1-Xn|Xn=i]| < 1 for i=0,1,2,

    • limsupi!1 E[Xn+1-Xn|Xn = i] < 0, i.e., there exist positive numbers  and N such that E[Xn+1-Xn|Xn = i] < - for all i¸ N

Next Generation Communication NetworksLab.


Limiting distribution

  • The definition of the limiting probabilities of {Xn}

    i = limn!1 P{Xn = i|X0 = j}

  • when does the limiting probabilities exist?

    • consider a DTMC with transition matrix P

    • For state 0, {Xn} can visits state 0 only at slots with even numbers, i.e.,

      P{X2k+1 = 0} = 0

      which means that no limiting probability exists.

Next Generation Communication NetworksLab.


Periodicity

  • Definition of the periodicity

    • For state i, the span d(i) of state i is defined by

      d(i) = g.c.d. {n¸ 1 | pii(n) > 0}.

    • If d(i) = 1, we say state i is an aperiodic state.

  • If i $ j, then d(i) = d(j).

  • If pii > 0 for some state i in an irreducible DTMC, the chain is aperiodic.

Next Generation Communication NetworksLab.


Ergodicity of a DTMC

  • The concept of ergodicity

Time average

T

Ensemble average

= E[f(X(T))]

T

Next Generation Communication NetworksLab.


  • Time average of a DTMC

    • For a state i, recall that fii(n), n¸ 1 form the p.m.f. for the length between consecutive visits to state i (i.e., renewals), and (i) is the length of a renewal.

    • When X0 =i, Ni (n) denotes the number of visits to state i (i.e., renewals) in [1,n]. Then by the elementary renewal theorem,

Next Generation Communication NetworksLab.


  • A DTMC {Xn} is said to be ergodic if it is irreducible and all the states are positive recurrent and aperiodic.

  • In an irreducible and aperiodic Markov chain, there always exist the limits

    which are the limiting probabilities of the DTMC.

Next Generation Communication NetworksLab.


The properties of an irreducible and aperiodic DTMC

  • When state i is transient or null recurrent, the limit i = 0.

  • When state i is positive recurrent (and hence all the states are positive recurrent), the limiting distribution is, in fact, the stationary distribution of the DTMC. Therefore, the limiting distribution also satisfies  =  P, i2 Si = 1 (or  e = 1), where e is a column vector all of whose elements are equal to 1.

  • Since i = limn!1 P{Xn = i|X0 = j} = limn!1 pji(n), we have Pn! e.

Next Generation Communication NetworksLab.


stationary vs. limiting Distribution

  • consider a DTMC with transition matrix P

the DTMC is periodic with period 2,

but it has the stationary distribution

Next Generation Communication NetworksLab.


Computation of the limiting distribution

  • Iterative algorithm

    • (0) be a given initial distribution

    • (n) = (0) Pn, the distribution of Xn.

    • Then ¼(n) if |(n) - (n-1)| <  for a sufficiently small >0.

  • Pn! e

  • Eigenvector of P

    •  is, in fact, the eigenvector of the matrix P corresponding to an eigenvalue 1.

Next Generation Communication NetworksLab.


  • When the transition matrix P is of dimension k.

    • Let E be a square matrix of dimension k with all the elements equal to 1. Note that  E = et.

      (et = the transpose of e)

    • from  =  P we have  (P+E-I) = et where I denotes the identity matrix of dimension k.

    • since the matrix P+E-I is invertible,

       = et(P+E-I)-1.

    • Note that the solution of the above equation automatically satisfies the normalizing condition  e = 1.

Next Generation Communication NetworksLab.


  • Login