1 / 37

Reinforcement Learning: Goal-oriented Learning for Control in Stochastic Environments

Learn about the mathematical model of reinforcement learning and its applications in robotics, control theory, and cognitive science. Explore the tradeoff between exploration and exploitation and discover the power of optimal policies in maximizing long-term rewards.

lmessner
Download Presentation

Reinforcement Learning: Goal-oriented Learning for Control in Stochastic Environments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Reinforcement Learning Yishay Mansour Tel-Aviv University

  2. Outline • Goal of Reinforcement Learning • Mathematical Model (MDP) • Planning

  3. Goal of Reinforcement Learning Goal oriented learning through interaction Control of large scale stochastic environments with partial knowledge. Supervised / Unsupervised Learning Learn from labeled / unlabeled examples

  4. Reinforcement Learning - origins Artificial Intelligence Control Theory Operation Research Cognitive Science & Psychology Solid foundations; well established research.

  5. Typical Applications • Robotics • Elevator control [CB]. • Robo-soccer [SV]. • Board games • backgammon [T], • checkers [S]. • Chess [B] • Scheduling • Dynamic channel allocation [SB]. • Inventory problems.

  6. Contrast with Supervised Learning The system has a “state”. The algorithm influences the state distribution. Inherent Tradeoff: Exploration versus Exploitation.

  7. Mathematical Model - Motivation Model of uncertainty: Environment, actions, our knowledge. Focus on decision making. Maximize long term reward. Markov Decision Process (MDP)

  8. Mathematical Model - MDP Markov decision processes S- set of states A- set of actions d - Transition probability R - Reward function Similar to DFA!

  9. MDP model - states and actions Environment = states 0.7 action a 0.3 Actions = transitions

  10. MDP model - rewards R(s,a) = reward at state s for doing action a (a random variable). Example: R(s,a) = -1 with probability 0.5 +10 with probability 0.35 +20 with probability 0.15

  11. trajectory: a0 r0 a1 r1 a2 r2 s0 s1 s2 MDP model - trajectories

  12. MDP - Return function. Combining all the immediate rewards to a single value. Modeling Issues: Are early rewards more valuable than later rewards? Is the system “terminating” or continuous? Usually the return is linear in the immediate rewards.

  13. MDP model - return functions Finite Horizon -parameter H Infinite Horizon discounted -parameter g<1. undiscounted Terminating MDP

  14. MDP model - action selection AIM: Maximize the expected return. Fully Observable - can “see” the “entire” state. Policy - mapping from states to actions Optimal policy: optimal from any start state. THEOREM: There exists a deterministic optimal policy

  15. Contrast with Supervised Learning Supervised Learning: Fixed distribution on examples. Reinforcement Learning: The state distribution is policy dependent!!! A small local change in the policy can make a huge global change in the return.

  16. MDP model - summary - set of states, |S|=n. - set of k actions, |A|=k. - transition function. - immediate reward function. - policy. - discounted cumulative return. R(s,a)

  17. Simple example: N- armed bandit Single state. Goal: Maximize sum of immediate rewards. a1 s Given the model: Greedy action. a2 Difficulty: unknown model. a3

  18. N-Armed Bandit: Highlights • Algorithms (near greedy): • Exponential weights • Gisum of rewards of action ai • wi = eGi • Follow the (perturbed) leader • Results: • For any sequence of T rewards: • E[online] > maxi {Gi} - sqrt{T log N}

  19. Planning - Basic Problems. Given a complete MDP model. Policy evaluation - Given a policy p, estimate its return. Optimal control - Find an optimal policy p*(maximizes the return from any start state).

  20. Planning - Value Functions Vp(s)The expected return starting at state s followingp. Qp(s,a)The expected return starting at state s with action a and then followingp. V*(s) and Q*(s,a) are define usingan optimal policy p*. V*(s) = maxpVp(s)

  21. Rewrite the expectation Planning - Policy Evaluation Discounted infinite horizon (Bellman Eq.) Vp(s) = Es’~ p (s) [ R(s,p (s)) + g Vp(s’)] Linear system of equations.

  22. Algorithms - Policy Evaluation Example A={+1,-1} g = 1/2 d(si,a)= si+a p random s0 s1 0 1 3 2 "a: R(si,a) = i s2 s3 Vp(s0) = 0 +g [p(s0,+1)Vp(s1) + p(s0,-1)Vp(s3) ]

  23. Algorithms -Policy Evaluation Example Vp(s0) = 5/3 Vp(s1) = 7/3 Vp(s2) = 11/3 Vp(s3) = 13/3 A={+1,-1} g = 1/2 d(si,a)= si+a p random s0 s1 0 1 "a: R(si,a) = i 3 2 s2 s3 Vp(s0) = 0 + (Vp(s1) + Vp(s3) )/4

  24. Algorithms - optimal control State-Action Value function: Qp(s,a) = E [ R(s,a)] + g Es’~ (s,a)[ Vp(s’)] Note For a deterministic policy p.

  25. Algorithms -Optimal control Example Qp(s0,+1) = 5/6 Qp(s0,-1) = 13/6 A={+1,-1} g = 1/2 d(si,a)= si+a p random s0 s1 0 1 R(si,a) = i 3 2 s2 s3 Qp(s0,+1) = 0 +g Vp(s1)

  26. Algorithms - optimal control CLAIM: A policy p is optimal if and only if at each state s: Vp(s) = MAXa{Qp(s,a)} (Bellman Eq.) PROOF: Assume there is a state s and action a s.t., Vp(s) < Qp(s,a). Then the strategy of performing a at state s (the first time) is better than p. This is true each time we visit s, so the policy that performs action a at state s is better than p. p

  27. Algorithms -optimal control Example A={+1,-1} g = 1/2 d(si,a)= si+a p random s0 s1 0 1 R(si,a) = i 3 2 s2 s3 Changing the policy using the state-action valuefunction.

  28. Algorithms - optimal control The greedy policy with respect to Qp(s,a) is p(s) = argmaxa{Qp(s,a) } The e-greedy policy with respect to Qp(s,a) is p(s) = argmaxa{Qp(s,a) } with probability 1-e, and p(s) = random action with probability e

  29. MDP - computing optimal policy 1. Linear Programming 2. Value Iteration method. 3. Policy Iteration method.

  30. Convergence • Value Iteration • Drop in distance from optimal each iteration maxs {V*(s) – Vt(s)} • Policy Iteration • Policy can only improve s Vt+1(s)  Vt(s) • Less iterations then Value Iteration, but • more expensive iterations.

  31. Relations to Board Games • state = current board • action = what we can play. • opponent action = part of the environment • value function = probability of winning • Q- function = modified policy. • Hidden assumption: Game is Markovian

  32. Planning versus Learning Tightly coupled in Reinforcement Learning Goal: maximize return while learning.

  33. Example - Elevator Control Learning (alone): Model the arrival model well. Planning (alone) : Given arrival model build schedule Real objective: Construct a schedule while updating model

  34. Partially Observable MDP Rather than observing the state we observe some function of the state. Ob - Observable function. a random variable for each states. Example: (1) Ob(s) = s+noise. (2) Ob(s) = first bit of s. Problem: different states may “look” similar. The optimal strategy is history dependent !

  35. POMDP - Belief State Algorithm Given a history of actions and observable value we compute a posterior distribution for the state we are in (belief state). The belief-state MDP: States: distribution over S (states of the POMDP). actions: as in the POMDP. Transition: the posterior distribution (given the observation) We can perform the planning and learning on the belief-state MDP.

  36. POMDP - Hard computational problems. Computing an infinite (polynomial) horizon undiscounted optimal strategy for a deterministicPOMDP is P-space-hard(NP-complete) [PT,L]. Computing an infinite (polynomial) horizon undiscounted optimal strategy for a stochasticPOMDP is EXPTIME-hard(P-space-complete) [PT,L]. Computing an infinite (polynomial) horizon undiscounted optimal policy for anMDPisP-complete[PT] .

  37. Resources • Reinforcement Learning (an introduction) [Sutton & Barto] • Markov Decision Processes [Puterman] • Dynamic Programming and Optimal Control [Bertsekas] • Neuro-Dynamic Programming [Bertsekas & Tsitsiklis] • Ph. D. thesis - Michael Littman

More Related