Chapter 13 deterministic dynamic programming
This presentation is the property of its rightful owner.
Sponsored Links
1 / 43

Chapter 13 DETERMINISTIC DYNAMIC PROGRAMMING PowerPoint PPT Presentation


  • 77 Views
  • Uploaded on
  • Presentation posted in: General

Chapter 13 DETERMINISTIC DYNAMIC PROGRAMMING. Math 305 2008 We will cover 9.1-9.4 plus some material not in the text. Dynamic Programming. Technique for making a sequence of interrelated decisions Problem solving strategies, e.g. find a route from here to LA

Download Presentation

Chapter 13 DETERMINISTIC DYNAMIC PROGRAMMING

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Chapter 13 deterministic dynamic programming

Chapter 13 DETERMINISTIC DYNAMIC PROGRAMMING

Math 305 2008

We will cover 9.1-9.4 plus some material not in the text


Dynamic programming

Dynamic Programming

Technique for making a sequence of interrelated decisions

Problem solving strategies, e.g. find a route from here to LA

  • forward: enumerate all possibilities

  • backward: figure which ways one can get to the desired end

    General type of problem: consecutive stages

  • at each stage you are in one of a number of possible states

  • each state has one or more possible policies from which to choose

  • the policy you choose determines your state at the next stage

    Method

  • start with a solution for a small part of the problem

  • expand

     A useful approach when you don’t want to check all possiblities


Prototype problem stage coach

Prototype problem (stage coach)

A traveller is going from east to west coast on a series of stage

coaches that travel from one state to another and wants the

safest route

S/he selects a life insurance policy for each stage, reasoning that cheapest = safest


Approach

Approach

4 stages

States at each stage:

1: 1

2: 2,3,4

3: 5, 6, 7

4: 8,9

First stage: travel from 1 to 2, 3, or 4

Second stage: travel from state at that stage to next stage

Decision variables: xn (n=1,..,4) = destination on nth stage


Approach1

Approach

fn(s,xn) = total cost of best policy for remaining stages given you

are in state s at stage n and select policy xn

(n = stage, s = state, xn = decision)

fn(s) = cost of best policy given you are in state s at stage n

= min fn(s, xn) over possible choices for xn

fn(s, xn) = c s xn + fn+1(xn) (c s xn = cost of policy from s to xn)

goal: find f1(1)

f1(1) = min of f1 (1,2) = 2 + f2(2)

f1 (1,2) = 5 + f2(3)

f1 (1,4) = 1 + f2(4)


Start at stage 4

Start at Stage 4

f2*(2) = ? ...

Better: go backward

Find the best policy from stage 4 on, stage 3 on,...

Stage 4: states 8 and 9 → we want f4*(8) and f4*(9)

f4(8,10) = 1 = f4*(8)

f4(9,10) = 4 = f4*(9)


Stage 3

Stage 3

States 5, 6, 7 → we want f3*(5), f3*(6). f3*(7)

 f3(5,8) = 7 + f4*(8) = 8

f3(5,9) = 5 + f4*(9) = 9

f3(6,8) = 3 + f4*(8) = 4

f3(6,9) = 4 + f4*(9) = 8

f3(7,8) = 7 + f4*(9) = 8

f3(7,9) = 1 + f4*(9) = 5

So far this isn't any better than

exhustive search


Stage 2

Stage 2

States 2, 3, 4

f2(2,5) = 10 + f3*(5) = 18

f2(2,6) = 12 + f3*(6) = 16

f2(3,5) = 5 + f3*(5) = 13

f2(3,6) = 10 + f3*(6) = 14

f3(3,7) = 7 + f3*(7) = 12

f2(4,6) = 15+f3*(6) = 19


Stage 1

Stage 1

f1(1,2) = 2 + f2*(2) = 18

f1(1,3) = 5 + f2*(3) = 17

f1(1,4) = 1 + f2*(4) = 19


Chapter 13 deterministic dynamic programming

Shortest route:

1 -> 3

3->7

7 -> 9

9->10


Computational efficiency

Computational Efficiency

This method? 16 additions, 9

comparisons

How else could we solve this?

  • list all paths (14) and total

    # additions (3 on each) => 42

  • shortest route ?

    Compare n+1 stages with n choices at each stage except last

    Dynamic programming: n nodes with n additions each =n2

    Exhaustive search: nn paths with n additions = nn+1

    E.g. n=10: 100 versus 1011


13 3 inventory theory

13.3 Inventory Theory

What is inventory?

  • something that is produced

  • has a demand

  • needs to be stored until used

  • cookies at Dories, beer at the Fargo, flash drives at the bookstore

    What is an inventory policy?

  • when to order/produce more

  • how much at a time

    What costs are associated with inventory?

  • cost per unit (variable)

  • setup or ordering

  • holding

  • shortage


Inventory theory

Inventory Theory

What are we trying to optimize?

Assumptions

  • time is broken into periods

  • production occurs at the beginning of the period

  • each period has an associated demand which is met from items held over from the last period and/or produced in the current period


13 3 example

13.3 Example

Demand for a productCosts

c(x)=cost of producing x units=3 + 1x

Other restrictions

  • at most 5 units can be produced each month

  • at most 4 units can be carried over to the next month

  • 0 units on hand at the beginning of month 1

    What are the stages, states, and decisions?

  • stage: beginning of a month

  • state: entering inventory

  • decision: fn(s) = amt produced at beginning of n, given s units on hand


Stage 4

Stage 4

f4(i) = cost of entering period 4 with i units = cost of producing 4 - i units = c(4-s)

f4(0) = set up + cost of 4 units = 3 + 4

f4(1) = set up + cost of 3 units = 3 + 3

Demand is 1 3 2 4


Chapter 13 deterministic dynamic programming

Stage 3

f3 (i) = min {c(x) + 0.5(x + i - 2) + f4(i+x -2) }      

x=0..4ordering holding stage 4 on

Demand is 1 3 2 4


Chapter 13 deterministic dynamic programming

Alternate Notation for Stage 3


Stage 21

Stage 2

f2 (i) = min {c(x) + 0.5(x + i - 3) + f2(i+x -3) }   for x = 0, 1, ...,5

x=0,1,2,3,4

Demand is 1 3 2 4


Alternate notation for stage 2

Alternate Notation for Stage 2


Stage 11

Stage 1

f1 (i) = min {c(x) + 0.5(x + i - 1) + f1(i+x - 1) }      for x = 0, 1, ...,5

x=0,1,2,3,4

Demand is 1 3 2 4

(text also finds f1 (i) for i = 1,2,3,4)


There must be an easier way not in text

There must be an easier way(not in text)

Only produce when entering inventory=0

  • don't carry inventory to meet part of the demand if you will have to produce in the period

    Demand is 1 3 2 4


Network representation

Network Representation

2,0

(i,j): i = period, j = beginning inventory

2,1

1, 0

5,0

2,2

2,3

2,4


13 4 resource allocation

13.4 Resource Allocation

I have 5 blocks of time to study and want to maximize the sum of

my grades

gi(xi) = grade in subject i given I study xi blocks.

3 decisions: xn = # blocks to spend on subject n

stage = subject

state = time available to allocate to remaining stages


Resource allocation

Resource Allocation

gi(xi) = grade in subject i given xi blocks

Objective: max  gi(xi) subject to xi= 5

 fn(s) = max effectiveness of s hours in stages n through 3

fn(s, xn) = gn(xn) + fn+1(s - xn)

= grade from xn hours to subject n plus best effect of

remaining hours in later stages

fn(s) = max { gn(xn) + fn+1(s - xn)}

xn=o..s


Resource allocation1

Resource Allocation

f3(s) f2(s)

f1(s)

Solution: English 2, Econ 2, Physics 1


Continuous functions

Continuous Functions

Suppose we can allocate fractional units of time and grade is a

continuous function of time spent

English: g1(x) = 65 - x2 + 11x max at (5.5, 95)

Econ: g2(x) = 80 - 2x2 + 13 x max at (3.25, 101.125)

Physics: g3(x) = 55 + 7x

 f3(s) = 55 + 7s x3 = s

f2(s, x2) = g2(x2) + f3(s - x2)

= 80 -2 x22 + 13x2 + 55 + 7(s - x2)

= 135 - 2 x22 + 6x2 + 7s

 df2/dx2 = -4x2 + 6 = 0 when x2 = 3/2

 d2f2/dx22 = -4 -> max at x2 = 3/2 (if s ≥ 3/2)

otherwise max at x2 = s


Continuous functions1

Continuous Functions

Case 1, s < 3/2

 x2 = s,

 f2(s) = 80 - 2s2 + 13s + 55 + 7(s - s) = 135 + 13s - 2s2

 Case 2, 3/2  s

 f2(s) = 80 + 2(3/2)2 + 13(3/2) + 55 + 7(s - 3/2) = 150 + 7s

Implications

  • if available time < 3/2, put it all into econ

  • if ≥ 3/2, put 3/2 into econ and surplus into physics

    (compare slopes of the two graphs before and after 3/2)


Continuous functions2

Continuous Functions

f1(5, x1) = g1(x1) + f2(5- x1)

case1, 5 - x1 < 3/2 (x1 > 3.5)

f1(5, x1) = 65 - x1 2 + 11x + 135 + 13(5 - x1) - 2(5 - x1)2

df1/d x1 = -2 x1 + 11 - 13 - 4(5- x1) = 2 x1 - 22

< 0 for 3.5 < x1 5

-> max at x1 = 3.5 if 3.5 < x1 5. f1(5. 3.5) = 241.25

case 2, 5 - x1 3/2 (0 x1 3.5)

f1(x1, 5) = 65 - x1 2 + 11 x1 + 150 + 7(5- x1) = 250 + 4 x1 - x1 2

 df1/d x1 = 4 - 2 x1 = 0 at x = 2

  d2f1/dx12 = -2 -> max at x = 2

 f1(5) = f1(5,2) = 254


Continuous functions3

Continuous Functions

Decision: x1 = 2

Thus 5 - x1 = 3 left for remaining stages

3 > 3/2 -> x2 = 3/2 -> x3 = 3 - 3/2 = 3/2

Solution: x1 = 2 x2 = 3/2 x3 = 3/2, sum of grades = 254 


Probabilistic models

Probabilistic Models

The state at the next stage is not completely determined by decision at current stage, rather determines a probability distribution for the next stage.

 Objective: maximize the expected value.

Example: job interview

  • a job candidate has up to three interviews

  • at each, she will be offered a job which is terrific, good or fair

  • she must decide then whether to accept the job or interview again

  • a terrific job is worth 3 points, good: 2, fair: 1

    Stages: interviews

    State: job status at stage n (T, G, or F)

    Decision: interview or accept


Probabilistic models1

Probabilistic Models

fn(s) = max expected value if in state s at stage n

fn(s, xn) = max expected value if in state s at stage n and make

decision xn (xn = i or a)

T

T

T

0

G

G

G

F

F

F


Stage 31

Stage 3

f3(T, i) = 3(.2) + 2(.5) + 1(.3) = 1.9

f3(T,a) = 3-> f3(T) = 3, x3 = a

f3(G, i) = 3(.2) + 2(.5) + 1(.3) = 1.9

f3(G,a) = 2-> f3(G) = 2, x3 = a

f3(F, i) = 3(.2) + 2(.5) + 1(.3) = 1.9

f3(F,a) = 1-> f3(F) = 1.9, x3 = i


Stage 22

Stage 2

f2(T, i) = p(T)f3(T) + p(G)f3(G) + p(F)f3(F)

= .2(3) + .5(2) + .3(1.9) = 2.17

f2(T,a) = 3-> f2(T) = 3, x2 = a

f1(i) = .2f2(T) + .5f2(G) + .3f2(F)

= .2(3) + .5(2.17) + .3(2.17) = 2.336

Strategy at stage I: interview; II: interview in G or F.; III: interview in F

Stage 1


Knapsack problem back to 13 4

Knapsack Problem (back to 13.4)

A thief breaks into a house.

Around the thief are various objects: a diamond ring, a silver candelabra, a

Bose Wave Radio, a large portrait of Elvis Presley painted on a black

velvet background (a "velvet-elvis"), and a large tiffany crystal vase.

The thief has a knapsack that can only hold a certain capacity (8).

Each of the items has a value and a size, and cannot hold all of the items in

the knapsack.

Which items should the thief take?

There are three thieves: greedy, foolish and slow, and wise (ref for this example)


Knapsack problem

Knapsack Problem

The greedy thief breaks into the window, and sees

the items. He makes a mental list of the items,

and grabs the most expensive item first. The ring

goes in first, leaving a capacity of 7, and a value of 15.

Next, he grabs the candelabra, leaving a knapsack of

size 2 and a value of 25. No other items will fit in his knapsack, so he leaves.

The foolish and slow thief climbs in the window, and sees the items. This thief

was a programmer, downsized as part of the "dot-bomb" blowout. Possessing

a solid background in boolean logic, he figures that he can simply compute all

combinations of the objects and choose the best. So, he starts going through

the binary combinations of objects - all 25 of them. While he is still drawing

the truth table, the police show up, and arrest him. Although his solution

would certainly have given him the best answer, it just took long to compute.


Knapsack problem wise thief

Knapsack Problem, Wise Thief

The wise thief appears, and observes the items.

He notes that an empty knapsack has a value of 0.

He notes that a knapsack can either contain

each item, or not.

Further, his decision to include an item will be based

on a quick calculation - either the knapsack with some

combination of the previous items will be worth more, or else the knapsack

of a size that will fit the current item was worth more. So, he does this quick

computation, and figures out that the best knapsack he can take is made up

of items 1,3, and 4, for a total value of 29


Generalized resource problem p 767

Generalized Resource Problem (p. 767)

w units of a resource available

T activities in which the resources can be allocated

xt : the level at which activity t is implemented

gt(xt ): # of units of the resource used by activity t

rt(xt ): the resulting benefit

States: each activity

Stages: how much of the resource is available for remaining stages

Decision: how much to use at this stage

Formulation

maximize Σrt(xt ) s.t. Σgt(xt ) ≤ w

t = 1,...T t = 1,...T

ft(d) = max benefit if d units are allocated to activities t through T

ft(d) = max {rt(xt ) + ft+1(d - xt ) fT+1(d) = 0

xt


Wise thief stages 4 and 3

Wise Thief, Stages 4 and 3

f3(d) = max {r3(x3 ) + f4(d - x3 )

f3(3) = max {r3(0) + f4(3) = 0 + 0

r3(3) + f4(0) = 9 + 0}

f3(4) = max {r3(0) + f4(4) = 4

r3(3) + f4(1) = 9}

f3(7) = max {r3(0) + f4(7) = 4

r3(3) + f4(4) = 9 + 5 =14


Stage 23

Stage 2

f2(d) = max {r2(x2) + f43(d - x2)

f2(3) = max {r2(0) + f3(3) = 4}

f3(4) = max {r2(0) + f3(4) = 9}

f3(5) = max {r2(0) + f3(5) = 3

r2(5) + f3(0) = 10}

f3(7) = max {r2(0) + f3(7) = 14

r2(5) + f3(2) = 10}

f3(8) = max {r2(0) + f3(8) = 14

r2(5) + f3(3) = 10 + 9 = 19}


Stage 12

Stage 1

f1(8) = max {r1(0) + f2(8) = 0 + 19}

r1(1) + f2(7) = 15 + 14}


How is this an lp

How is this an LP?

xi = # item i

max 15x1 + 10x2 + 9x3 + 5x4

s.t. x1 + 5x2 + 3x3 + 4x4 ≤ 8


Turnpike theorem

Turnpike Theorem

Let cj = benefit from item j, wj = weight of item j

Order items by benefit per unit weight

c1/w1 ≥ c2 /w2 ≥ c3 /w3 ...

If there is a unique "best" item #1, e.g. c1/w1 > c2 /w2 when the max weight w ≥ w* = (c1w1)/(c1 - w1 ( c2 /w2 )

the optimal solution contains at least one of item 1

Thief problem:

15/1 > 9/3 > 10/5 > 5/4

w* = 15*1/(15 - 1*9/3) = 15/12 = 1.25 < 8

Why is this any use?

  • start with one ring and reduce computation

    Why turnpike

  • for a long trip you might go a little out of your way to maximize time on a turnpike.


Proof of turnpike theorem

Proof of Turnpike Theorem

Without using any type 1 items we cannot do better than include w/w2 type 2

This would earn c2w/w2

Suppose we fill the knapsack with as many type 1 items as possible

We can fit in at least (w/w1 ‑ 1) type 1 items

These items would earn a benefit of c1(w/w1 ‑ 1)

Thus if c1(w/w1 ‑ 1)c2w/w2 (1)

there must be an optimal solution using a type 1 item

(1) holds if w(c1/w1 ‑ c2/w2)c1

c1w1

or w ‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑ = w*

c1 ‑ c2w1/w2

Thus if knapsack can hold at least w* pounds, there will be an optimal

solution using at least one type 1 item


  • Login