Practical Dynamic Programming in Ljungqvist – Sargent (2004) Presented by Edson Silveira Sobrinho

1. University of Houston Economics Department Practical Dynamic Programming in Ljungqvist – Sargent (2004) Presented by Edson Silveira Sobrinho for Dynamic Macro class

2. MotivationFirst approachSecond approach Conclusions The curse of dimensionality • Chapter 3 presents some computational methods for Dynamic programs of the form: • Subject to: given x. • The Euler Equation helps in a variety of cases. • One can also start with a Vo and iterate until convergence. • However, it may be impossible to solve analytically. Then we have to adopt some numerical approximations. 2

3. MotivationFirst approachSecond approach Conclusions Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation First approach for obtaining numerical approximations: • The first method replaces the original problem with another problem by forcing the state vector to live on a finite and discrete grid of points. • Then applies discrete-state dynamic programming to this problem. • Downside is that it requires a small number of discrete states. • Guess what? Markov chains! 3

4. MotivationFirst approachSecond approach Conclusions Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation • Set up: an infinitely lived household likes to consume one good, which it can acquire by using labor income (wst) or accumulated savings (at). • st is endowment of labor at time t. si is endowment of labor in state i. • m-state Markov chain with transition matrix P. Say m = 2, where si = 0 (if unemployed) or si = 1 (if employed). • The wage w is fixed over time. • The household can choose to hold a single asset in discrete amount at where atЄ A is a grid [ a1 < a2 < … < an]. • The asset bears a gross rate of return r fixed over time. 4

5. Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation • MotivationFirst approachSecond approach Conclusions • Household maximization problem (Eq. 4.2.1): • Subject to : • Assume that: 5

6. Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation • MotivationFirst approachSecond approach Conclusions • Bellman equation: • Which is a matrix m x n (2 states of employment x n choices of a). • Let a' and s' indicate next period's asset and state. • v (a,s) is the optimal value of the objective function, starting from asset and employment state (a, s). • A solution of this problem is a value function v(a, s) that satisfies the Bellman Equation and an associated policy function a' = g(a, s) mapping this period’s (a, s) pair into an optimal choice of assets to carry into next period. 6

7. Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation • MotivationFirst approachSecond approach Conclusions • For discrete-state space of small size, it is easy to solve the Bellman equation numerically by manipulating matrices. Here is how to write a computer program • Define two n x 1 vectors vj, j=1,2, i.e. one vector for each employment state. Each row takes the value of v(i) for a given asset i. • Let 1 be the n x 1 vector consisting entirely of ones 7

8. Discretizationof state space • Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation • MotivationFirst approachSecond approach Conclusions • Define two n x n matrices Rj whose (i, h) element is (Eq. 4.3.0): • This is the current period utility conditional on current asset i and next period asset h, for a given employment state j. • That is, all the possible utilities that arise from combinations of choices of assets for this period and the next one. 8

9. Discretizationof state space • Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation • MotivationFirst approachSecond approach Conclusions • Define an operator T ([v1, v2]) that maps a pair of vectors [v1, v2] into a pair of vectors [tv1, tv2] (Eq. 4.3.1) : • LHS is a column vector n x 1 • The term in the Max operator is n x n Matrix. • Intuition: each element brings the lifetime utility for a given pair of asset choices for current and next period. For each row (say current asset), the agent chooses the column (next period's asset) that allows max lifetime utility. Current consumption comes from the budget constraint in R1. • Similarly, for unemployed state: 9

10. Discretizationof state space • Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation • MotivationFirst approachSecond approach Conclusions The production dynamics • Compact notation (Eq. 4.3.2): • Then the Bellman Equation can be viewed as: • And can be solved as: 10

11. Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation • MotivationFirst approachSecond approach Conclusions • Back in Chapter 3 we saw the Howard Improvement Algorithm. • The idea was to Pick a feasible policy, u = h0(x), and compute the value associated with operating forever with that policy. Next generate a new policy u = hj+1(x) that solves the two-period problem. Then iterate. • Now let there be a predetermined class M of (N × N) stochastic matrices P, which are the objects of choice. • I think that we can associate each stochastic matrix P to a pair (a,a'). That's why we have n x n matrices P. So choosing P is like choosing (a,a'). 11

12. MotivationFirst approachSecond approach Conclusions Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation • We can put the Bellman equation in the following form (Eq. 4.4.3): • For B = T – I , find Pn+1such that (Eq. 4.4.4): • Intuition: Pick a feasible policy function and use it to iterate. I believe the advantage of the algorithm is that it is easier to compute (compare Eq 4.3.0 with 4.5.2): 12

13. MotivationFirst approachSecond approach Conclusions Discretization of state space Discrete-state dynamic programming Application of Howard Improvement algorithm Numerical implementation Eq. 4.5.1 Eq. 4.5.2 • Step 1. For an initial feasible policy function g j (k, j) for j = 1, form the rh matrices using an equation similar to (4.5.1), then use equation (4.5.2) to evaluate the vectors of values [vj1, vj2] implied by using that policy forever. • Step 2. Use [vj1, vj2] as the terminal value vectors in equation (4.3.2), and perform one step on the Bellman equation to find a new policy function gj+1(k, s) for j + 1 = 2. Use this policy function, update j , and repeat step 1. • Step 3. Iterate. 13

14. MotivationThe modelImplications for saving and growth Conclusions • MotivationFirst approachSecond approach Conclusions Polynomial approximations Second approach for obtaining numerical approximations: • Judd (1998) describes a method for iterating on the Bellman equation using a polynomial to approximate the value function and a numerical optimizer to perform the optimization at each iteration. • We describe this method in the context of the Bellman equation for a particular problem. • Model of optimal unemployment insurance as in chapter 19. 14

15. MotivationThe modelImplications for saving and growth Conclusions • MotivationFirst approachSecond approach Conclusions Polynomial approximations • A planner wants to provide incentives to an unemployed worker to search for a new job while also partially insuring the worker against bad luck in the search process. • The planner seeks to deliver discounted expected utility V to an unemployed worker at minimum cost while providing proper incentives to search for work. • Hopenhayn and Nicolini show that the minimum cost C(V) satisfies the Bellman equation (Eq. 4.7.1). 15

16. MotivationThe modelImplications for saving and growth Conclusions • MotivationFirst approachSecond approach Conclusions Polynomial approximations • The method uses a polynomial to approximate the i th iterate Ci(V) of C(V). This polynomial is stored on the computer in terms of n + 1 coefficients. • Then at each iteration, the Bellman equation is to be solved at a small number m >= n + 1 values of V . • This procedure gives values of the i th iterate of the value function Ci(V) at those particular V ’s. • Then we interpolate to fill in the continuous function Ci(V). • The minimum problem on the right side of equation (4.7.1) becomes a numerical minimizer (doable with simple algorithms in Matlab or Gauss). 16

17. MotivationThe modelImplications for saving and growth Conclusions • MotivationFirst approachSecond approach Conclusions Polynomial approximations Algorithm summary: Choose upper and lower bounds for Vu [Vu, Vu] required to assure participation and a positive search effort, computed in chapter 19. Choose a degree n for the approximator, a Chebyshev polynomial, and a number m >= n + 1 of nodes. Generate the m zeros of the Chebyshev polynomial on the set [1,−1] , given by (4.7.6). By a change of scale, transform the zi’s to corresponding points Vlu in [Vu, Vu]. 17

18. MotivationThe modelImplications for saving and growth Conclusions • MotivationFirst approachSecond approach Conclusions Polynomial approximations Choose initial values of the n+1 coefficients. Use these coefficients to define the function Ci(Vu) for iteration number i = 0. Compute the function For each point Vlu , use a numerical minimization program to find Ci+1(Vlu ). Compute new values of the coefficients by using “least squares”. Return to step 5 and iterate to convergence. 18

19. MotivationFirst approachSecond approach Conclusions • This chapter has described two of three standard methods for approximating solutions of dynamic programs numerically: discretizing the state space and using polynomials to approximate the value function. • The next chapter describes the third method: making the problem have a quadratic return function and linear transition law. • This will make solving a dynamic program easy by exploiting stochastic linear difference equations. 19