EE255/CPS226 Continuous Time Markov Chain (CTMC)

1 / 53

# EE255/CPS226 Continuous Time Markov Chain (CTMC) - PowerPoint PPT Presentation

EE255/CPS226 Continuous Time Markov Chain (CTMC). Dept. of Electrical & Computer engineering Duke University Email: bbm@ee.duke.edu , kst@ee.duke.edu. Discrete State-Continuous Time Stochastic Process.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about 'EE255/CPS226 Continuous Time Markov Chain (CTMC)' - alexis

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### EE255/CPS226Continuous Time Markov Chain (CTMC)

Dept. of Electrical & Computer engineering

Duke University

Email: bbm@ee.duke.edu, kst@ee.duke.edu

Discrete State-Continuous Time Stochastic Process
• A CTMC is characterized by state changes that can occur at any arbitrary time (in contrast to a DTMC, where state changes can occur only at well defined times).
• Index space is un-countable.
• The state space continues to a discrete valued.
• I={0,1,2,..} denotes the process state space
• T=[0, ) is the index space.
• This forms discrete space, cont. time stochastic process:

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Continuous Time Markov Chain (CTMC)
• The process qualifies to be Markov chain, if for t0 < t1 < t2 < …. < tn < t , the conditional pmf satisfies:
• Process can then be completely described by:
• Initial state probability vector for X(t0):
• Transition probabilities.
• Also,

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Homogenous CTMCs
• is a time-homogenous CTMC iff ,
• Or, the conditional pmf satisfies:
• Marginal pmf (state probability) is given by:
• State prob. May be written as,

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

CTMC Chapman-Kolmogorov Equation
• Proof: using total prob. Law
• Since v < u and invoking Markov property proves it.
• The final goal of obtaining πj(t) using Chapman-Kolmogorov eq. is cumbersome.
• Instead we are forced to prob. transition rates

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Prob. Transition Rates
• Lot of math/calculus follows:. Define,
• Relating transition probs. & rates :

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Calculus steps
• In general,
• Therefore, (1) can be rewritten as,

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Prob. Transition Matrix

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Transition Prob. (Homogenous case)
• Transition rates qij(t) and qj(t) are independent of t. The Kolmogorov forward equation reduces to,
• In the matrix form, (Matrix Q is called the infinitesimal generator matrix (or simply Generator Matrix)
• Defining,

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Classification
• Classification of states of a CTMC is identical to DTMC
• i is anabsorbing state if qij = 0 for j = I. Once the process enters this state, it then never leaves this state.
• Steady-state behavior when there are absorbing states:
• j is aReachable state from i, if for some t>0, pij(t) > 0.
• Irreducible CTMC: iff every state is reachable from every other state. For an irreducible CTMC, the following is true:

always exists and are independent of the initial state i. In case the limiting probabilities πj exist, then,

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

• Irreducible CTMCs having +ve steady-state {πj} values are called recurrent non-null.
• Performance measures may not be computed by assigning rewards to all states and computing E[reward]:
• Accumulated reward (over an interval of time)
• Markov chain exhibits memory-less property
• Hj(t) follows EXP( ) distribution.

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Continuous Time Birth-Death Process
• The CTMC and i={0,1,2,…} forms a B-D process, if λi, i={0,1,2,..} and μi, i={1,2,..}exists, and,
• λi,: Birth rate (>= 0) and μi,: Death rate (>= 0)

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Continuous Time Birth-Death Process (contd.)

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

These are called balance eqs. Re-arranging above,

= 0

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/1 Queue
• Arrivals follow Poisson distribution, i.e., inter-arrival times are all i.i.d, EXP(λ).
• Departures are also Poissonian i.e., inter-departure times are all i.i.d, EXP(μ).
• Models systems such as,
• Packet arrivals at a router port (router switch is the server)
• Arrival of tasks at a computer system or a scheduler (server computer)
• Customer queues (clerk/cashier is the server)
• Repair workshops, etc. (Repair station or the technician is the server)

Poisson arrival

Process with rate λ

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/1 queue (contd.)
• N(t): birth-death proc., λk=λ; μk=μ
• Define, ρ=λ/μ (traffic intensity, in Erlangs)
• ρ < 1 (for reasons of stability).

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/1 queue performance measures
• Server utilization = 1- π0
• Expected # of customers,
• Above measures can be viewed as expected reward,
• Resulting model is known as the MRM.
• Other measures:
• Average queue length (E[n])
• Average (expected) response time
• Average (expected) wait time et.

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/1 queue: Little’s formula
• Response time (R) = wait time (W) + service time (S)
• E[R] = E[N]/λ (Little’s formula)

N = (1(t2-t1)+2(t3-t2)+1(t4-t3)+1(t6-t5)+2(t7-t6)+3(t8-t7)+2(t9-t8)+1(T- t9 ))/T

= (area under the curve)/T = (T+t9 + t8 – t7 – t6 – t5 + t4 + t3 – t2 – t1)/T

R = ((t3-t1)+ (t4-t2)+ (t8-t5)+ (t9-t6)+(T- t7))/5

= (T+t9 + t8 – t7 – t6 – t5 + t4 + t3 – t2 – t1)/5 = (area under the curve)/K

R.K = N.T . Note that, λ = K/T . This yields the Little’s formula.

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Expected response time
• Analysis has tacitly assumed the FCFS scheduling.
• Other scheduling policies may be pre-emptive, e.g., RR.
• Round Robin (RR)  time-slice, then back to the end of queue.
• Scope for more than one RR queue.
• The E[R] formula holds for any scheduling policy provided, some conditions are met.

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Response time distribution
• Assuming FCFS and steady-state conditions
• If there are already n jobs in the system, the next job (N+1)st will experience a response time =R= S*+S’1+S2+..+SN
• S* : service time for the (N+1)st job; S’1+: residual service time for job currently undergoing service (#1).
• Because of the memory-less property, these times are EXP( ).
• Hence, for some N=n, the LST of R is,
• Therefore,

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/k queue
• m-servers together service the queue.

μ

Poisson arrivals (λ)

μ

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/m Queue Solution

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/m Queue performance measures
• Average queue length E[N]: rk= k

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/m Queue performance measures
• Server utilization: rv M - number of busy servers. M may be defined in terms of the number of items N in the queue.
• A customer may have to join the queue.

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Poisson stream behavior
• M/M/m: input/output both form Poisson streams.
• m=2 case
• Case 1: Two independent queues
• Case 2: M/M/2 case

Two separate Poisson streams

 2 separate M/M/1 queues

Two separate Poisson streams

Combined Poisson steams

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Comparative performance
• Case 1: For each M/M/1 queue,
• Case 2: Common queue M/M/2

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/1/n Queue
• Finite queue size, finite buffer space  finite state space.
• Transient soln?

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/1/n Queue Performance Measures
• Mean queue length (expected # of jobs in the system).
• rk = k,
• Loss probability
• rn = 1, rk = 0, k=0,1,..,n-1
• Throughput
• rk =m , k=1,2, ..,n; r0 = 0 (or, rk =l , k=0,1,2, ..,n-1; rn = 0)

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/1/n: Response time distribution
• Response time distribution: Job may be rejected (or accepted)
• Unconditional (rejected):
• Conditional (accepted):
• Reward assignment: for the kth state, response time experienced by the tagged task is sum of k-service times, each of which is EXP(μ), i.e., k-stage Erlang.
• Unconditional
• Conditional

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/1/n Example
• Machine failure/repair model
• Machine fails with MTTF = 1/λ and MTTR = 1/μ
• Transient solution:
• pUU(t) can also be computed.
• A(t) = pUU(t)
• Interval Availability:

λ

U

D

μ

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/1/n Example : varying birth rate
• Birth rate in state j is λj= (M-j) λ and μj = μ.
• Multiple clients generating service requests and single server. After requesting, client waits for response.
• M- parallel component - single repair station

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

M/M/1/n: M-components Example
• Various repair possibilities exist
• Each component may have its own repair facility (U/D model)
• Single repair station (to reduce cost)
• Single repair station solution may be slow, increase the repair rate from μ  M μ

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Special cases of Birth-Death Process
• Pure birth processes
• Poisson process
• Software Reliability Growth Model: NHPP
• Number of software failures occurring in (0, t] is N(t), and N(t) is Poisson with, λ(t) = abe-btand m(t) = E[N(t)] = a(1- e-bt)
• Instantaneous failure intensity, λ(t) = b[a-m(t)]
• Transient solution may be found using Laplace transforms
• Pure death processes
• No-repairs

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Non Birth-Death processes
• Not all CTMC may exhibit nearest neighbor transittions only.
• Markov chains with arbitrary transition pattern
• Availability modeling
• Performance modeling
• Performability modeling
• Example based

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Availability model using CTMCs
• Failure/repair model
• repair = failure detection/location + actual repair
• Life time: EXP(λ ); Detection/location: EXP(μ1); Repair: EXP(μ2)
• The flow equations are:

λ

μ1

0

1

2

μ2

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

2-component Availability model
• 2-component availability model
• Ass = 1-π0
• Failures detection stage takes random time, EXP(δ)
• Down states are ‘0’ and ‘1D’  Ass = 1- π0- π1D

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

2-components : finite coverage
• Coverage factor = c
• ‘1C’ state is a re-boot (down) state.

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

2-components : delay+finite coverage
• Model has detection delay+coverage factor
• Down states are ‘0’, ‘1C’ and ‘1D’.

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Preventive Maintenance example
• Prolonged usage of a component may lead to increased failure rate (i.e. IFR situation)
• Hence, life time may be modeled as HypoEXP() distribution, say 2-stage Hypo.
• Component is inspected randomly. Time between inspections is a random, following EXP(λi). Inspection completion time is EXP(μi).
• What does inspection do?
• First stage of life – no action
• Second stage of life – repair
• That is, preventive maintenance
• State = <#stage, faulty>

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Performance Models
• Example: 2-servers with different service times.
• State = <n1, n2>
• Performance: Average no. of jobs in the system, E[n1+n2]
• Reward rn1, n2 = n1+n2
• Except for the <0,0>, in all other states, viz., <k,0> and <k,1>, there are k jobs in the system.

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Markov Modulated Poisson Process (MMPP)
• MMPP is a doubly stochastic Poisson process, such that the arrival rate is dependent on the state of another CTMC.
• The second CTMC (call it MM) is the modulating process having m states in general, Q = [qij].
• If MM is in state I, then the (first) Poisson process has λi as its arrival rate.
• The Poisson process is a counting process as the overall modulated process.
• 2-d state <Poisson (counting) proc, Modulating proc> .

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

MMPP Counting process

λ1

λ2

λ3

λ1

λ2

λ3

λ3

λ2

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

CTMCs with Absorbing states
• Example: 2-component parallel system (perfect package)
• Steady-state solution- not meaningful. We need to find transient solution.

Initial starting state

i.e., π2(0)=1

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

2-components, finite coverage
• Faults are covered with coverage factor = c.

Initial starting state

i.e., π2(0)=1

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Petri Net
• Is a modeling tool used for modeling issues, such as, concurrency, synchronization, mutual exclusion etc.

Conflict/concurrency PN

Producer/Consumer PN

PN Markings

p1

p2

p1

p1 p2 p3 p4 p5

t1

C

11000

t1

p3

t1

t2

p2

01100

p5

B

t2

t3

t3

p3

11001

p4

t2

10010

p5

p4

t4

t3

t5

t4

t4

n

: Denotes a place with finite capacity for tokens (=n)

n

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

PN Definitions
• A PN is a 5-tuple: (P, T, A, X, M) . PN is a bi-partite graph.
• Arcs: TP or PT
• An arc may have multiplicity X.
• Rules for enabling a transition t : when all places p (ε P) such pt exists have one or more tokens.
• Firing a transitions: an enabled transition may fire, if all the places p’ that have non-null tp’ arcs have no tokens.
• Markings, aka reachability Graph
• Inhibitor arcs

t1

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Stochastic PNs
• Random delay (firing time) between enabling and firing a transition. Such a transition: stochastic transition.
• Frequently, firing time: EXP( ) distributed.
• Resulting reachability graph of an SPN : Markov chain.
• Example: Poisson process

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Stochastic PN: M/M/1 Queue
• Arrival and service transitions
• M/M/1/n queue
• Reachability graph

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Generalized SPN (GSPN)
• GSPN is an an enhancement to the SPN.
• GSPN has both immediate and timed transitions
• Firing probabilities: GSPN also admits the possibility of firing more than one immediate transitions.
• Transition priorities: by assigning an integer priority level to each transition.
• GSPN vanishing and tangible markings
• A vanishing marking involves atleast one immediate transition.
• Tangible markings involve only timed transitions.

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

GSPN examples:
• Example: M/Em/1/n+1 queue
• Analysis of a GSPN has 4-steps
• Generate reachability graphs
• Eliminate vanishing markings  CTMC of tangible markings
• Analyze steady state or transient behavior of the CTMC
• Evaluate any measures

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

GSPN Examples: M/M/i/n queue
• GPSN also allows for place dependent firing rate
• Example: M/M/i/n queue
• M/M/n/n queue

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

Stochastic Reward Net (SRN)
• GSPN may be extended further
• associate a reward value with each tangible marking,
• Marking dependent guard function
• if (guard == T) transition may be fired)
• Marking dependent arc multiplicities (typically used to fire a flush transition that empties a place of all tokens)
• Marking dependent firing rates.

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University

SRN – fault coverage example
• A work station fault may have imperfect coverage.

Bharat B. Madan, Department of Electrical and Computer Engineering, Duke University