1 / 31

Intelligent Systems (AI-2) Computer Science cpsc422 , Lecture 4 Jan, 13, 2014

Intelligent Systems (AI-2) Computer Science cpsc422 , Lecture 4 Jan, 13, 2014. Announcements. Office hours Kamyar Ardekani kamyar.ardekany@gmail.com ;   X150 (Learning Center) Table 4 Thurs  12:30 - 1:30 Mine, in my office CICSR 105 Mondays 11-12, . What to do with readings?

arwen
Download Presentation

Intelligent Systems (AI-2) Computer Science cpsc422 , Lecture 4 Jan, 13, 2014

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Intelligent Systems (AI-2) Computer Science cpsc422, Lecture 4 Jan, 13, 2014 CPSC 422, Lecture 4

  2. Announcements • Office hours • KamyarArdekanikamyar.ardekany@gmail.com;   X150 (Learning Center) Table 4 Thurs 12:30 - 1:30 • Mine, in my office CICSR 105 Mondays 11-12, • What to do with readings? • In a few lectures we will discuss the first research paper. Instructions on what to do will be available on the course webpage today CPSC 422, Lecture 4

  3. Lecture Overview • Markov Decision Processes • …… • Finding the Optimal Policy • Value Iteration • From Values to the Policy • Rewards and Optimal Policy CPSC 422, Lecture 4

  4. Relevance to state of the art MDPs • FROM : Planning with Markov Decision Processes: An AI Perspective Mausam (UW),Andrey Kolobov (MSResearch) Synthesis Lectures on Artificial Intelligence and Machine Learning Jun 2012 • “ Value Iteration (VI) forms the basis of most of the advanced MDP algorithms that we discuss in the rest of the book. ……..” • Free online through UBC CPSC 422, Lecture 4

  5. Value of Optimal policy Remember for any policy π But the Optimal policy π* is the one that gives the action that maximizes the future reward for each state So… CPSC 422, Lecture 4

  6. Value Iteration Rationale • Given Nstates, we can write an equation like the one below for each of them • Each equation contains Nunknowns – the V values for the Nstates • N equations in N variables (Bellman equations): It can be shown that they have a unique solution: the values for the optimal policy • Unfortunately the N equations are non-linear, because of the max operator: Cannot be easily solved by using techniques from linear algebra • Value Iteration Algorithm: Iterative approach to find the optimal policy and corresponding values CPSC 422, Lecture 4

  7. Value Iteration in Practice • Let V(i)(s) be the utility of state s at the ith iteration of the algorithm • Start with arbitrary utilities on each state s: V(0)(s) • Repeat simultaneously for every s until there is “no change” • True “no change” in the values of V(s) from one iteration to the next are guaranteed only if run for infinitely long. • In the limit, this process converges to a unique set of solutions for the Bellman equations • They are the total expected rewards (utilities) for the optimal policy CPSC 422, Lecture 4

  8. Example • Suppose, for instance, that we start with values V(0)(s) that are all 0 Iteration 0 Iteration 1 3 2 1 +1 -1 -0.04 1 2 3 4 CPSC 422, Lecture 4

  9. Example (cont’d) • Let’s compute V(1)(3,3) Iteration 0 Iteration 1 0.76 3 2 1 +1 -1 -0.04 -0.4 1 2 3 4 CPSC 422, Lecture 4

  10. Example (cont’d) • Let’s compute V(1)(4,1) Iteration 0 Iteration 1 3 2 1 +1 -1 -0.04 -0.04 .76 1 2 3 4 CPSC 422, Lecture 4

  11. After a Full Iteration Iteration 1 3 2 1 +1 -1 1 2 3 4 • Only the state one step away from a positive reward (3,3) has gained value, all the others are losing value because of the cost of moving CPSC 422, Lecture 4

  12. Some steps in the second iteration Iteration 2 Iteration 1 3 2 1 3 2 1 +1 +1 -1 -1 -0.08 1 2 3 4 1 2 3 4 CPSC 422, Lecture 4

  13. Example (cont’d) • Let’s compute V(1)(2,3) • Steps two moves away from positive rewards start increasing their value Iteration 2 Iteration 1 0.56 +1 3 2 1 +1 -1 -1 -0.08 1 2 3 4 1 2 3 4 CPSC 422, Lecture 4

  14. State Utilities as Function of Iteration # • Note that values of states at different distances from (4,3) accumulate negative rewards until a path to (4,3) is found CPSC 422, Lecture 4

  15. Value Iteration: Computational Complexity Value iteration works by producing successive approximations of the optimal value function. What is the complexity of each iteration? • …or faster if there is sparsity in the transition function. B. O(|A||S|2) C. O(|A|2|S|2) A. O(|A|2|S|) CPSC 422, Lecture 4

  16. Lecture Overview • Markov Decision Processes • …… • Finding the Optimal Policy • Value Iteration • From Values to the Policy • Rewards and Optimal Policy CPSC 422, Lecture 4

  17. Value Iteration: from state values V to л* • Now the agent can chose the action that implements the MEU principle: maximize the expected utility of the subsequent state CPSC 422, Lecture 4

  18. Value Iteration: from state values V to л* • Now the agent can chose the action that implements the MEU principle: maximize the expected utility of the subsequent state expected value of following policy л* in s’ states reachable from s by doing a Probability of getting to s’ from s via a CPSC 422, Lecture 4

  19. Example: from state values V to л* • To find the best action in (1,1) CPSC 422, Lecture 4

  20. Optimal policy • This is the policy that we obtain…. CPSC 422, Lecture 4

  21. Lecture Overview • Markov Decision Processes • …… • Finding the Optimal Policy • Value Iteration • From Values to the Policy • Rewards and Optimal Policy CPSC 422, Lecture 4

  22. Rewards and Optimal Policy Optimal Policy when reward in non-terminal states is -0.04 Is it possible that the optimal policy changes if the reward in the non-terminal states changes? B. No A. Yes CPSC 422, Lecture 4

  23. Rewards and Optimal Policy If r = -2, what would be a reasonable policy 1 2 3 4 1 2 3 4 3 2 1 3 2 1 A. B. CPSC 422, Lecture 4

  24. Rewards and Optimal Policy Optimal Policy when penalty in non-terminal states is -0.04 • Note that here the cost of taking steps is small compared to the cost of ending into (2,4) • Thus, the optimal policy for state (1,3) is to take the long way around the obstacle rather then risking to fall into (2,4) by taking the shorter way that passes next to it • May the optimal policy change if the reward in the non-terminal states (let’s call it r) changes? CPSC 422, Lecture 4

  25. Rewards and Optimal Policy Optimal Policy when r < -1.6284 1 2 3 4 3 2 1 Why is the agent heading straight into (2,4) from its surrounding states? CPSC 422, Lecture 4

  26. Rewards and Optimal Policy Optimal Policy when -0.427 < r < -0.085 1 2 3 4 3 2 1 The cost of taking a step is high enough to make the agent take the shortcut to (3,4) from (1,3) CPSC 422, Lecture 4

  27. Rewards and Optimal Policy Optimal Policy when -0.0218 < r < 0 1 2 3 4 3 2 1 Why is the agent heading straight into the obstacle from (2,3)? And into the wall in (1,4)? CPSC 422, Lecture 4

  28. Rewards and Optimal Policy Optimal Policy when -0.0218 < r < 0 1 2 3 4 3 2 1 • Stay longer in the grid is not penalized as much as before. The agent is willing to take longer routes to avoid (2,4) • This is true even when it means banging against the obstacle a few times when moving from (2,3) CPSC 422, Lecture 4

  29. Rewards and Optimal Policy Optimal Policy when r > 0 Which means the agent is rewarded for every step it takes 1 2 3 4 3 2 1 state where every action belong to an optimal policy CPSC 422, Lecture 4

  30. Learning Goals for today’s class • You can: • Define/read/write/trace/debug the Value Iteration (VI)algorithm. Compute its complexity. • Compute the Optimal Policy given the output of VI • Explain influence of rewards on optimal policy CPSC 322, Lecture 36

  31. TODO for Wed • Read Textbook 9.5.6 Partially Observable MDPs • Also Do Practice Ex. 9.Chttp://www.aispace.org/exercises.shtml CPSC 422, Lecture 4

More Related