Markov processes
Download
1 / 25

Markov Processes - PowerPoint PPT Presentation


  • 90 Views
  • Uploaded on

Markov Processes. MBAP 6100 & EMEN 5600 Survey of Operations Research Professor Stephen Lawrence Leeds School of Business University of Colorado Boulder, CO 80309-0419. Intro to OR Linear Programming Solving LP’s LP Sensitivity/Duality Transport Problems Network Analysis

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Markov Processes' - kiona


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Markov processes

Markov Processes

MBAP 6100 & EMEN 5600Survey of Operations Research

Professor Stephen Lawrence

Leeds School of Business

University of Colorado

Boulder, CO 80309-0419


Or course outline

Intro to OR

Linear Programming

Solving LP’s

LP Sensitivity/Duality

Transport Problems

Network Analysis

Integer Programming

Nonlinear Programming

Dynamic Programming

Game Theory

Queueing Theory

Markov Processes

Decisions Analysis

Simulation

OR Course Outline


Whirlwind tour of or
Whirlwind Tour of OR

Andrey A. Markov (born 1856). Early work in probability theory, proved central limit theorem

Markov Analysis


Agenda for this week

Markov Applications

More Markov examples

Markov decision processes

Markov Processes

Stochastic processes

Markov chains

Future probabilities

Steady state probabilities

Markov chain concepts

Agenda for This Week


Stochastic processes
Stochastic Processes

  • Series of random variables {Xt}

  • Series indexed over time interval T

  • Examples: X1, X2, … , Xt, … , XT represent

    • monthly inventory levels

    • daily closing price for a stock or index

    • availability of a new technology

    • market demand for a product


Markov chains
Markov Chains

  • Present state Xt is independent of history

    • previous states or events have no current or future influence on the current state

  • Process will move to other states with known transition probabilities

  • Transition probabilities are stationary

    • probabilities do not change over time

  • There exist a finite number of possible states


An example of a markov chain
An Example of a Markov Chain

A small community has two service stations: Petroco and Gasco. The marketing department of Petroco has found that customers switch between stations according to the following transition matrix:

=1.0

=1.0

Note:Rows sum to 1.0 !


Future state probabilities
Future State Probabilities

Probability that a customer buying from Petroco this month will buy from Petroco next month:

In two months:

From Gasco in two months:


Graphical interpretation
Graphical Interpretation

First Period

Second Period

Petroco

0.36

0.6

0.6

Petroco

0.6

Gasco

0.24

0.4

Petroco

Petroco

0.08

0.2

0.4

0.4

Gasco

0.8

Gasco

0.32

1.00


Chapman kolmogorov equations
Chapman-Kolmogorov Equations

Let P be the transition matrix for a Markov process. Then the n-step transition probability matrices can be found from:

P(2) = P·P

P(3) = P·P·P



Starting states

P

s2 =[0.7 0.3]

=[0.7 0.3]

Starting States

In current month, if 70% of customers shop at Petroco and 30% at Gasco, what will be the mix in 2 months?

sn = s0P(n)

s = [0.70 0.30]

=[0.39 0.61]


Ck equations in steady state
CK Equations in Steady State

P(1)

P(2)

P(9)


Convergence to steady state

Prob

1.0

0.33

Period

1

5

10

Convergence to Steady-State

If a customer is buys at Petroco this month, what is the long-run probability that the customer will buy at Petroco during any month in the future?


Calculation of steady state
Calculation of Steady State

  • Want outcome probabilities equal to incoming probabilities

  • Let s = [s1, s2, …, sn] be the vector of steady-state probabilities

  • Then we wants = s P

  • That is, the output state probabilities do not change from transition to transition (e.g., steady-state!)


Steady state for example

P

[p g] =[p g]

Steady-State for Example

s = s P

p = 0.6p + 0.2g

g = 0.4p + 0.8g

p + g = 1

s = [p g]

p = 0.333

g = 0.667


Markov chain concepts
Markov Chain Concepts

  • Steady-State Probabilities

    • long-run probability that a process starting in state i will be found in state j

  • First-Passage Time

    • length of time (steps) in going from state i to j

  • Recurrence Time

    • length of time (steps) to return to state i when starting in state i


Markov chain concepts cont
Markov Chain Concepts (cont.)

  • Accessible States

    • State j can be reached from i (pij(n) > 0)

  • Communicating States

    • State i and j are accessible from one another

  • Irreducible Markov chains

    • All states communicate with one another


Markov chain concepts cont1
Markov Chain Concepts (cont.)

  • Recurrent State

    • A state that will certainly return to itself (fii = 1)

  • Transient State

    • A state that may return to itself (fii < 1)

  • Absorbing State

    • A state the never moves to another state (pii=1)

    • A “black hole”


Markov examples markov decision processes

Markov ExamplesMarkov Decision Processes


Matrix multiplication
Matrix Multiplication

Matrix multiplication in Excel…


Machine breakdown example
Machine Breakdown Example

A critical machine in a manufacturing operation breaks down with some frequency. The hourly up-down transition matrix for the machine is shown below. What percentage of the time is the machine operating (up)?

Up

Down

Up

Down


Credit history example

Pay

No Pay

Pay

No Pay

Credit History Example

The Rifle, CO Mercantile Department Store wants to analyze the payment behavior of customers who have outstanding accounts. The store’s credit department has determined the following bill payment pattern from historical records:


Credit history continued

0

1

2

Pay

Bad

0

1

2

Pay

Bad

Credit History Continued

Further analysis reveals the following credit transition matrix at the Rifle Mercantile:


University graduation example

F

So

J

Sr

D

G

F

So

J

Sr

D

G

University Graduation Example

Fort Lewis College in Durango has determined that students progress through the college according to the following transition matrix:


ad