**2. **Markov Processes

**3. **Classification of States: 1 A path is a sequence of states, where each transition has a positive probability of occurring.
State j is reachable (or accessible) (???? ?????? ????) from state i (i?j) if there is a path from i to j ?equivalently Pij (n) > 0 for some n=0, i.e. the probability to go from i to j in n steps is greater than zero.
States i and j communicate (i?j) (????) if i is reachable from j and j is reachable from i.
(Note: a state i always communicates with itself)
A set of states C is a communicating class if every pair of states in C communicates with each other, and no state in C communicates with any state not in C.

**4. **Classification of States: 1 A state i is said to be an absorbing state if pii = 1.
A subset S of the state space X is a closed set if no state outside of S is reachable from any state in S (like an absorbing state, but with multiple states), this means pij = 0 for every i ?S and j ?S
A closed set S of states is irreducible(??? ???? ???????) if any state j ?S is reachable from every state i ?S.
A Markov chain is said to be irreducible if the state space X is irreducible.

**5. **Example Irreducible Markov Chain

**6. **Classification of States: 2 State i is a transient state (???? ?????)if there exists a state j such that j is reachable from i but i is not reachable from j.
A state that is not transient is recurrent (???? ??????) . There are two types of recurrent states:
Positive recurrent: if the expected time to return to the state is finite.
Null recurrent (less common): if the expected time to return to the state is infinite (this requires an infinite number of states).
A state i is periodic with period k >1, if k is the smallest number such that all paths leading from state i back to state i have a multiple of k transitions.
A state is aperiodic if it has period k =1.
A state is ergodic if it is positive recurrent and aperiodic.

**7. **Classification of States: 2

**8. **Transient and Recurrent States We define the hitting time Tij as the random variable that represents the time to go from state j to stat i, and is expressed as:
k is the number of transition in a path from i to j.
Tij is the minimum number of transitions in a path from i to j.
We define the recurrence time Tii as the first time that the Markov Chain returns to state i.
The probability that the first recurrence to state i occurs at the nth-step is
Ti Time for first visit to i given X0 = i.
The probability of recurrence to state i is

**9. **Transient and Recurrent States

**10. **Transient and Recurrent States We define Ni as the number of visits to state i given X0=i,
Theorem: If Ni is the number of visits to state i given X0=i, then
Proof

**11. **Transient and Recurrent States The probability of reaching state j for first time in n-steps starting from X0 = i.
The probability of ever reaching j starting from state i is

**12. **Three Theorems If a Markov Chain has finite state space, then: at least one of the states is recurrent.
If state i is recurrent and state j is reachable from state i
then: state j is also recurrent.
If S is a finite closed irreducible set of states, then: every state in S is recurrent.

**13. **Positive and Null Recurrent States Let Mi be the mean recurrence time of state i
A state is said to be positive recurrent if Mi<8.
If Mi=8 then the state is said to be null-recurrent.
Three Theorems
If state i is positive recurrent and state j is reachable from state i then, state j is also positive recurrent.
If S is a closed irreducible set of states, then every state in S is positive recurrent or, every state in S is null recurrent, or, every state in S is transient.
If S is a finite closed irreducible set of states, then every state in S is positive recurrent.

**14. **Example

**15. **Periodic and Aperiodic States Suppose that the structure of the Markov Chain is such that state i is visited after a number of steps that is an integer multiple of an integer d >1. Then the state is called periodic with period d.
If no such integer exists (i.e., d =1) then the state is called aperiodic.
Example

**16. **Steady State Analysis Recall that the state probability, which is the probability of finding the MC at state i after the kth step is given by:

**17. **17 Multi-step (t-step) Transitions

**18. **18 The Tax Auditing Example

**20. **20 Steady-State Solutions ? n Steps

**21. **21 Steady-State Solutions ? n Steps

**22. **22 Steady State Transition Probability

**23. **Steady State Analysis Recall the recursive probability

**24. **Steady State Analysis

**25. **25 Comments on Steady-State Results

**26. **26 Interpretation of Steady-State Conditions

**27. **Discrete Birth-Death Example

**28. **Discrete Birth-Death Example

**29. **Discrete Birth-Death Example

**30. **Discrete Birth-Death Example

**31. **Reducible Markov Chains

**32. **Reducible Markov Chains

**33. **Reducible Markov Chains