Introduction to Discrete Time Semi Markov Process

1. Introduction to Discrete Time Semi Markov Process Nur Aini Masruroh

2. Recall: Discrete Time Markov Process In the DTMC� Whenever a process enters a state i, we imagine that it determines the next state j to which it will move instantaneously according to the transition probability of pij

3. Discrete time semi Markov Process In semi Markov process, after state j has been selected, but before making this transition from state i to state j, the process �holds� for a time tij in the state i. The holding times tij are positive, integer-valued random variables each governed by a probability mass function hij(.) called the holding time mass function for a transition from state i to state j After holding in state i for the holding time tij, the process makes transition to state j and then immediately select a new destination state k using the transition probabilities pjk It next chooses a holding time tjk in state j according to the mass function hjk(.) and makes its next transition at time tjk after entering state j The process continues in the same way

4. Discrete time semi Markov Process (cont�d) To describe semi Markov process completely, we need to define n2 holding time mass functions in addition to the transition probabilities Suppose, the cumulative probability distribution of tij, =hij(.) is defined as

5. Discrete time semi Markov Process (cont�d) So,ti: waiting time in state i and wi(.): waiting time pmf Waiting time is a holding time that is unconditional on the destination state The mean waiting time is related to the mean holding time by

6. Car rental example A car rental rents cars at two locations, town 1 and town 2. the experience of the company shows that a car is rented in town 1 is a 0.8 probability that it will be returned to town 1 and a 0.2 probability that it will be returned to town 2. when the car is rented in town 2, there is a 0.7 probability that it will be returned to town 2 and a 0.3 probability that it will be returned to town 1. We assumed that there are always many customers available at both towns and that cars are always rented at the town to which they are last returned Because of the nature of the trips involved, the length of time a car will be rented depends on both where it is rented and where it is returned. The holding time tij is thus the length of time a car will be rented if it was rented at town i and returned to town j. From the company records, the possible holding time pmf follows geometric distribution with the following expressions: h11(m) = (1/3)(2/3)m-1 h21(m) = (1/4)(3/4)m-1 h12(m) = (1/6)(5/6)m-1 h22(m) = (1/12)(11/12)m-1

7. Car rental example: solution Transition probability matrix

8. Car rental example: solution

9. Car rental example: solution

10. Car rental example: solution Waiting time

11. Car rental example: solution Cumulative and complementary cumulative distributions of waiting time:

12. Interval transition probabilities, Fij(n) Corresponds to multistep transition probabilities for the Markov process Fij(n): probability that a discrete-time semi Markov process will be in state j at time n given that it entered state i at time zero ? interval transition probability from state i to state j in the interval (0,n) Note that an essential part of the definition is that the system entered state i at time zero as opposed to its simply being in state i at time zero

13. Limiting behavior The chain structure of semi-Markov process is the same as that of its imbedded Markov process Dealing with monodesmic semi Markov process Monodesmic process: Markov process that has a F with equal rows Monodesmic process: Sufficient condition: able to make transition Necessary condition: exit only one subset of states that must be occupied after infinitely many transitions

14. Limiting behavior (cont�d) Limiting interval probabilities, Fij for a monodesmic semi Markov process:

15. Consider: car rental example Transition probability matrix, p1 = 0.6, p2 = 0.4