Reinforcement Learning

1. Presented by: Kyle Feuz Reinforcement Learning

3. Outline • Motivation • MDPs • RL • Model-Based • Model-Free • Q-Learning • SARSA • Challenges

4. Examples • Pac-Man • Spider

5. MDPs • 4-tuple (State, Actions, Transitions, Rewards) .

6. Important Terms • Policy • Reward Function • Value Function • Model

7. Model-Based RL • Learn transition function • Learn expected rewards • Compute the optimal policy

8. Model-Free RL • Learn expected rewards/values • Skip learning transistion function • Trade-offs?

9. Basic Equations

10. Examples • Pac-Man • Spider • Mario

11. Q-Learning Q(s, a) = = (1 − α)Q(s, a) + α[R(s, s′ ) + Max Q(s′ , a′ )]

12. Q-Learning • Demo Video

13. SARSA Q-Learning Q(s, a) = = (1 − α)Q(s, a) + α[R(s, s′ ) + Q(s′ , a′ )]

14. Challenges • Explore vs. Exploit • State Space representation • Training Time • Multiagent Learning • Moving Target • Competive or Cooperative

15. Transfer Learning for Reinforcement Learning on a Physical Robot • Applied TL and RL on Nao robot • TL using the q-value reuse approach • RL uses SARSA variant • State space is represented via CMAC • Neural Network inspired by the cerebellum • Acts as an associative memory • Allows agents to generalize the state space

16. Agent Model

17. SARSA Update Rule Q(s, a) = = (1 − α)Q(s, a) + α[R(s, s′ ) + γe(s, a)Q(s′ , a′ )]

18. Q-Value Reuse Q(s, a) = = Qsource (χX (s), χA (a)) + Qtarget (s, a)

19. Experimental Setup • Seated Nao robot • Hit the ball at 45 angle • 5 Actions in Source – 9 Actions in Target

20. Robot Results

21. Simulator Results

22. Advanced Combinations

23. Examples • Pac-Man • Spider • Mario • Q-Learning • Penalty Kick • Others

24. References and Resources • rl repository • rl-community • rl on PBWorks • rl warehouse • Reinforcement Learning: An Introduction • Artificial Intelligence: A Modern Approach • How to Make Software Agents do the Right Thing

25. Questions?