Discrete time markov chain
This presentation is the property of its rightful owner.
Sponsored Links
1 / 41

Discrete time Markov Chain PowerPoint PPT Presentation


  • 112 Views
  • Uploaded on
  • Presentation posted in: General

Discrete time Markov Chain. G.U. Hwang Next Generation Communication Networks Lab. Department of Mathematical Sciences KAIST. Definition.

Download Presentation

Discrete time Markov Chain

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Discrete time markov chain

Discrete time Markov Chain

G.U. Hwang

Next Generation Communication Networks Lab.

Department of Mathematical Sciences

KAIST


Definition

Definition

  • The sequence of R.V.s X0, X1, X2,  with a countable state space S is said to be a discrete time Markov chain (DTMC) if it satisfies the Markov Property:

    for any ik2 S, k=0,1,,n-1 and i, j 2 S.

  • Time homogeneous DTMC : P{Xn+1 = j | Xn = i} is independent of n.

Next Generation Communication NetworksLab.


Transition probability matrices

Transition Probability Matrices

  • One step transition probability matrix P

    • P = (pij) where pij = P{Xn+1 = j | Xn = i }

    • The matrix P is nonnegative and stochastic, i.e.,

      pij¸ 0 and j2 S pij = 1

  • n step transition probability matrixP(n) = (p(n)ij)

    • pij(n) = P{Xn = j | X0 = i}

Next Generation Communication NetworksLab.


Discrete time markov chain

  • For a DTMC, the initial distribution and the matrix P uniquely determine the future behavior of the DTMC because

Next Generation Communication NetworksLab.


Example a general random walk

Example: A general random walk

  • Let Xi be i.i.d. R.V.s with P{X1 = j} = aj, j=0,1,.

  • Let S0 = 0, Sn = k=1n Xk. Then {Sn, n¸ 1} is a DTMC because

Next Generation Communication NetworksLab.


Chapman kolmogorov s equation

Chapman - Kolmogorov's equation

  • Chapman - Kolmogorov's theorem

    pij(n+m) = k2 S pik(n) pkj(m)

    proof:

Next Generation Communication NetworksLab.


Discrete time markov chain

  • Using the chapman-Kolmogorov’s theorem we get

    i.e., the n-th power of the one step transition matrix P is, in fact, the n step transition matrix.

Next Generation Communication NetworksLab.


Example

Example

  • Find the distribution of X4 where {Xn} (Xn2 S = {1,2}) forms a DTMC with initial distribution P{X0 = 1}=1 and one step transition probability P as follows:

    sol:

Next Generation Communication NetworksLab.


Analysis of a dtmc

Analysis of a DTMC

  • When a communication system can be modeled by a DTMC with P and S = {0,1,2}, what happens?

A sample path of a DTMC

transient

period

stationary period

Next Generation Communication NetworksLab.


The stationary probabilities

The stationary probabilities

  • The state space S ={0,1, }

  • The stationary probability vector (distribution) 

    • a row vector  = (0, 1, ) is called a stationary probability vector of a DTMC with transition matrix P if it satisfies

Next Generation Communication NetworksLab.


Discrete time markov chain

  • Does the stationary distribution always exist?

0

1

2

Next Generation Communication NetworksLab.


Discrete time markov chain

  • Does the stationary distribution always exist?

0

1

2

3

…..

Next Generation Communication NetworksLab.


Discrete time markov chain

  • The key question:

    • When does the stationary vector exist?

  • to answer the question, we need to classify DTMCs according to its probabilistic properties as

    • irreducibility

    • recurrence

    • positive recurrence and null recurrence

    • periodic and aperiodic

Next Generation Communication NetworksLab.


Irreducible dtmc

Irreducible DTMC

  • state i can reach state j if there exists n¸ 0 s.t. pij(n) > 0.

  • In this case we write i ! j.

  • If i! j and j! i, then we say i and j communicate and write i $ j.

i

k

h

j

…..

g

f

…..

Next Generation Communication NetworksLab.


Discrete time markov chain

  • $ is an equivalent relation, that is,

    • i $ i

    • i $ j iff j $ i

    • if i $ j and j $ k, then i $ k

  • Definition of irreducibility

    • A DTMC is irreducible if its state space consists of a single equivalent class, i.e., for any i, j 2 S we have i $ j.

Next Generation Communication NetworksLab.


Discrete time markov chain

  • A closed set

    • a set A of states is closed if no one step transition is possible from any state in A to any state in AC, i.e., for every pair of states i 2 A and j 2 AC, pij = 0

  • An absorbing state

    • A single state which alone form a closed set is called an absorbing state

    • if state i is an absorbing state, pii = 1.

Next Generation Communication NetworksLab.


Recurrence

Recurrence

  • The hitting time (i) of state i:

    • For a state i 2 S, (i) = inf {n¸ 1| Xn = i}, i.e., (i) is the first visiting time of the DTMC {Xn} to state i.

    • When no such n exists, (i) = 1 by convention.

  • The number Ni of visits to state i:

    • Ni = n=11I{Xn=i} where IA is an indicator function which is defined by 1 if the event A occurs and by 0 otherwise.

    • Clearly, {Ni > 0} = {(i) < 1}.

Next Generation Communication NetworksLab.


Probability mass function of i

Probability mass function of (i)

  • Define

    fji (n) = P{ (i)=n | X0 = j }

  • Then

    fji = P{the DTMC ever visits state i | X0 = j }

    = n = 1 1 fji (n)

  • Definition of recurrence of state i

    • state i is said to be recurrent

      if P{(i) < 1 |X0 = i} = 1, and transient otherwise.

    • that is, state i is recurrent if fii =1, and transient if fii <1.

Next Generation Communication NetworksLab.


Observations

Observations

  • Once the DTMC revisits a recurrent state i starting from state i, by the Markov property it has the same probabilistic behavior as before. Hence, state i is visited infinitely.

  • Further, for a recurrent state i and X0 = i a.s. successive visits to state i can be viewed as renewals and {fii (n) | n¸ 1} is the p.m.f. of the inter-renewal times.

Next Generation Communication NetworksLab.


Discrete time markov chain

  • For a transient state i, starting from state i the DTMC revisits state i with probability fii (< 1) and it never enters state i with probability 1- fii. Therefore, by Markov property we have

    P{the DTMC visits state i n times} = (fii)n (1- fii ), n¸ 1

    i.e., the number of visits to state i is according to a geometric distribution.

Next Generation Communication NetworksLab.


Discrete time markov chain

  • The following are equivalent:

    • state i is recurrent

    • Ni = 1 with probability 1 provided that X0 = i

    • E[Ni|X0 = i] = E[n=11 I{Xn = i}|X0 = i]

      = n=11 pii(n) = 1

  • The following are equivalent:

    • state i is transient

    • Ni < 1 with probability 1 provided that X0 = i

    • E[Ni|X0 = i] = n=11 pii(n) < 1

Next Generation Communication NetworksLab.


Discrete time markov chain

  • If i $ j and i is recurrent, then j is also recurrent.

    Proof: Since i $ j, there exist n, m ¸ 0 s.t. pij(n) > 0 and pji(m) > 0. Then,

    which comes from the recurrence of state i. This completes the proof.

Next Generation Communication NetworksLab.


Positive recurrence

Positive recurrence

  • For a recurrent state i,

    • if E[(i) | X0 = i] < 1, state i is called positive recurrent.

    • if E[(i) | X0 = i] = 1, then state i is called null recurrent.

  • Note that, E[(i) | X0 = i] = 1 for a transient state i.

Next Generation Communication NetworksLab.


A stationary measure

A stationary measure

  • a vector q = (q0, q1,) is called a stationary measure of a M.C. with transition matrix P if

    • q 0

    • all qi are finite, i.e., qi < 1

    • q P = q

Next Generation Communication NetworksLab.


Discrete time markov chain

  • Let i2 S be a recurrent state. Then a stationary measure (q0, q1,) can be defined by

Next Generation Communication NetworksLab.


Discrete time markov chain

  • The stationary vector q defined above depends on the chosen state i.

  • However, it can be shown that the stationary measure (q0, q1,) is unique up to a constant multiplication.

Next Generation Communication NetworksLab.


Discrete time markov chain

  • Note that

  • So, if state i is positive recurrent, by normalizing the stationary measure (q0, q1,) , we have a stationary distribution (p0, p1,) for the DTMC {Xn}

Next Generation Communication NetworksLab.


The stationary distribution

The stationary distribution

  • The existence of the stationary distribution

    • If the DTMC is irreducible and positive recurrent, the stationary distribution exists and is given by

    • Note that the above equation is not used in numerical computation. In fact, we use

    • For numerical algorithms, we will see them shortly.

Next Generation Communication NetworksLab.


Discrete time markov chain

  • If i $ j and i is ㅔpositive recurrent, then j is also positive recurrent.

    proof: Let (i) and (j) be two stationary measures by using state i and j, respectively. Since both are stationary measures, there exists a constant c (< 1) such that (j) = c (i). So, By summing over all elements in both sides, we get

    (j) e = c (i) e < 1.

    Hence, state j is also positive recurrent.

Next Generation Communication NetworksLab.


Discrete time markov chain

  • An irreducible DTMC with a finite state space S is always positive recurrent.

    proof. Let Ni be the total number of visits to state i. Since i Ni should be 1 and the state space S is finite, for at least one state, say k, we have Nk = 1, which means state k is recurrent. Consequently, all states are recurrent because of the irreducibility of the DTMC.

    Now, since the stationary measure has a vector  of finite size, the sum ii should be finite, i.e., all states are positive recurrent.

Next Generation Communication NetworksLab.


Criterion for recurrence

Criterion for recurrence

  • Suppose that the DTMC is irreducible and let i be some fixed state.

  • Then the chain is transient if and only if there is a bounded non-zero real valued function

    h:S-{i} ! R satisfying

    h(j) = k i pjk h(k), j i.

Next Generation Communication NetworksLab.


Criteria for positive recurrence

Criteria for positive recurrence

  • (Pakes' lemma) Let a DTMC {Xn} be irreducible and aperiodic with state space S={0,1,}. Then {Xn} is positive recurrent if the following are satisfied:

    • |E[Xn+1-Xn|Xn=i]| < 1 for i=0,1,2,

    • limsupi!1 E[Xn+1-Xn|Xn = i] < 0, i.e., there exist positive numbers  and N such that E[Xn+1-Xn|Xn = i] < - for all i¸ N

Next Generation Communication NetworksLab.


Limiting distribution

Limiting distribution

  • The definition of the limiting probabilities of {Xn}

    i = limn!1 P{Xn = i|X0 = j}

  • when does the limiting probabilities exist?

    • consider a DTMC with transition matrix P

    • For state 0, {Xn} can visits state 0 only at slots with even numbers, i.e.,

      P{X2k+1 = 0} = 0

      which means that no limiting probability exists.

Next Generation Communication NetworksLab.


Periodicity

Periodicity

  • Definition of the periodicity

    • For state i, the span d(i) of state i is defined by

      d(i) = g.c.d. {n¸ 1 | pii(n) > 0}.

    • If d(i) = 1, we say state i is an aperiodic state.

  • If i $ j, then d(i) = d(j).

  • If pii > 0 for some state i in an irreducible DTMC, the chain is aperiodic.

Next Generation Communication NetworksLab.


Ergodicity of a dtmc

Ergodicity of a DTMC

  • The concept of ergodicity

Time average

T

Ensemble average

= E[f(X(T))]

T

Next Generation Communication NetworksLab.


Discrete time markov chain

  • Time average of a DTMC

    • For a state i, recall that fii(n), n¸ 1 form the p.m.f. for the length between consecutive visits to state i (i.e., renewals), and (i) is the length of a renewal.

    • When X0 =i, Ni (n) denotes the number of visits to state i (i.e., renewals) in [1,n]. Then by the elementary renewal theorem,

Next Generation Communication NetworksLab.


Discrete time markov chain

  • A DTMC {Xn} is said to be ergodic if it is irreducible and all the states are positive recurrent and aperiodic.

  • In an irreducible and aperiodic Markov chain, there always exist the limits

    which are the limiting probabilities of the DTMC.

Next Generation Communication NetworksLab.


The properties of an irreducible and aperiodic dtmc

The properties of an irreducible and aperiodic DTMC

  • When state i is transient or null recurrent, the limit i = 0.

  • When state i is positive recurrent (and hence all the states are positive recurrent), the limiting distribution is, in fact, the stationary distribution of the DTMC. Therefore, the limiting distribution also satisfies  =  P, i2 Si = 1 (or  e = 1), where e is a column vector all of whose elements are equal to 1.

  • Since i = limn!1 P{Xn = i|X0 = j} = limn!1 pji(n), we have Pn! e.

Next Generation Communication NetworksLab.


Stationary vs limiting distribution

stationary vs. limiting Distribution

  • consider a DTMC with transition matrix P

the DTMC is periodic with period 2,

but it has the stationary distribution

Next Generation Communication NetworksLab.


Computation of the limiting distribution

Computation of the limiting distribution

  • Iterative algorithm

    • (0) be a given initial distribution

    • (n) = (0) Pn, the distribution of Xn.

    • Then ¼(n) if |(n) - (n-1)| <  for a sufficiently small >0.

  • Pn! e

  • Eigenvector of P

    •  is, in fact, the eigenvector of the matrix P corresponding to an eigenvalue 1.

Next Generation Communication NetworksLab.


Discrete time markov chain

  • When the transition matrix P is of dimension k.

    • Let E be a square matrix of dimension k with all the elements equal to 1. Note that  E = et.

      (et = the transpose of e)

    • from  =  P we have  (P+E-I) = et where I denotes the identity matrix of dimension k.

    • since the matrix P+E-I is invertible,

       = et(P+E-I)-1.

    • Note that the solution of the above equation automatically satisfies the normalizing condition  e = 1.

Next Generation Communication NetworksLab.


  • Login