IEG5300 Tutorial 5 Continuous-time Markov Chain

1. IEG5300Tutorial 5 Continuous-time Markov Chain Peter Chen Peng Adapted from Qiwen Wang’s Tutorial Materials

2. Outline • Continuous-time Markov chain • Chapman-Kolmogorov equation • Kolmogorov’s Backward & Forward equation • Limiting Probability Pj • Time Reversible Markov Process • Summary

3. Continuous-time Markov chain― Properties • X(t) always corresponds to an embedded discrete-time Markov chain, with the constraint Pii = 0, ∀ i.X(t) = i means that the process is in state i at time t. • When the process X(t) is in state i at time t, the remaining time for it to make a transition to other states (≠i) is an exponentialr.v. with rate vi. The remaining time for a transition is independent of t.

4. Continuous-time Markov chain― Notations P{X(t + s) = j | X(s) = i, X(u) = x(u) ∀ u < s} = P{X(t + s) = j | X(s) = i} = Pij(t) // stationary transition probability. • Pii = 0 , ∀ i ; but Pii(t)≠0 for t ≥ 0 • vi = transition rate in state i; qij = viPij = instantaneous transition rate from state i to state j //vi = ∑<j>qij; Pij = qij / vi Ti = waiting time for a transitionin state i, exponential with vi

5. Continuous-time Markov chain― Example • Consider two machines that are maintained by a single repairman. Machine i functions for an exponential time with rate μi before breaking down, i = 1, 2. The repair times for either machine are exponential with rate μ. Analyze this as a Markov Process. • Define 5 states: 0 ― both machines on 1 ― 1st down, 2nd on 2 ― 1st on, 2nd down 3 ― both machines down, 1st is under repair. 4 ― both machines down, 2nd is under repair. • Draw a transition diagram • The transition rates: v0 =μ1 + μ2 , v1 =μ+ μ2 v2 =μ1 + μ, v3 = v4=μ

6. Birth-death Process • A special case of Markov process such that q01 =λ0 and for all other states i, qii+1 =λi, qii–1 =μi , and other qij = 0. Correspondingly, P01 = 1; • The Poisson process is a special case of birth-death process, with constant birth rateλand death rate 0.

7. Chapman-Kolmogorov equation • Pij(t) is the probability that given the current state is i, the process will stay at state j after time period t. Pij(t) = P{X(t + s) = j | X(s) = i} ; ∑<j>Pij(t) = 1 • Chapman-Kolmogorov equation // By conditioning on the state after time period t.

8. Kolmogorov’s Backward & Forward equation By setting either s or t in C-K equation to be infinitesimal, we get • Kolmogorov’s Backward equation • Kolmogorov’s Forward equation

9.  1 = on 0 = off  Kolmogorov’s Backward & Forward equation ― Example • Find Pij(t) by Kolmogorov Forward equation in the 2-state birth-death process. [Given: ] • P’01(t) = P00(t)q01 – P01(t)v1 = (1 – P01(t))λ– P01(t)μ=λ – (λ+μ)P01(t) By solving this 1st order DE, P01(t) = Ae–(λ+μ)t +λ/(λ+μ). By the boundary condition P01(0) = 0, A = –λ/(λ+μ). ∴P01(t) = λ/(λ+μ)•(1 – e–(λ+μ)t ) , P00(t) =1 –λ/(λ+μ)•(1 – e–(λ+μ)t )

10.  1 = on 0 = off  Kolmogorov’s Backward & Forward equation ― Example • Find Pij(t) by Kolmogorov Forward equation in the 2-state birth-death process. [Given: ] • Similarly, P’10(t) = P11(t)q10 – P10(t)v0 =μ–(λ+μ)P10(t) . P10(t) =μ/(λ+μ)•(1 – e–(λ+μ)t ) , P11(t) =1 –μ/(λ+μ)•(1 – e–(λ+μ)t )

11. Limiting Probability Pj of Pij(t) • Definition. A continuous-time Markov chain is said to be ergodic when lim t→∞Pij(t)exists for all j and the limiting value is independent of the initial state i. Let Pj denote lim t→∞Pij(t) • By the flow conservation law (balance equations): the rate into a state = the rate out of a state // from Forward equation • Similar toπjin the discrete case, Pj is the long run proportion of time the ergodic Markov process stays at state j and

12. Long Run Probability πj of Pij • When a continuous-time Markov chain ispositive recurrent, so is the imbedded discrete-time Markov chain. • If a continuous-time Markov chain is irreducible and positive recurrent, then it is ergodic. In that case, the imbedded discrete-time markov chain has a unique long-run distribution {j}, which is a solution to • Similarly,

13. Time Reversible Markov Process For an ergodic Markov process {X(t)}t≥0 , • its reversed process {Y(t)}T ≥t≥0 , Y(t) = X(T – t) corresponds to an embedded discrete-time Markov chain. If πi exists, Y(t) has the same viand • if its corresponding embedded discrete-time Markov chain is time reversible, it is also time reversible. Then, the rate from i to j = the rate from j to i. Pi qij = Pj qji or equivalently, Pij = Qij or equivalently, Piqij = Pjqji • An ergodic birth-death process is time reversible.

14. Burke’s Theorem • If  < s, the stationary output process of M/M/s is a Poisson process with intensity . // this M/M/s is time reversible at the stationary state so balance flow holds.

15. Summary • Chapman-Kolmogorov equation • Kolmogorov’s Backward (and Forward) equation

16. Summary • Limiting Probability Pj • Pj = lim t→∞Pij(t) • Time Reversible Markov Process • Pi qij = Pj qji