An Overview of MAXQ Hierarchical Reinforcement Learning. Thomas G. Dietterich from Oregon State Univ. Presenter: ZhiWei. Motivation. The traditional reinforcement learning algorithms treat the state space of the Markov Decision Process as a single “flat” search space.
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Thomas G. Dietterich from Oregon State Univ.
This paper describes an initial effort in this direction
Task: the taxi is in a randomly-chosen cell and the passenger is at one of the four special locations (R, G, B, Y). The passenger has a desired destination and the job of the taxi is to go to the passenger, pick him/her up, go to the passenger’s destination, and drop him/her off.
Six available primitive actions:
North, South, East, West, Pickup and Putdown
Reward: each action receives -1; when the passenger is putdown at the destination, receive +20; when the taxi attempts to pickup a non-existent passenger or putdown the passenger at a wrong place, receive -10; running into walls has no effect but entails the usual reward of -1.
where V(a, s) is the expected total reward while executing action a, and C(p, s, a) is the expected reward of completing parent task p after a has returned
Three fundamental forms
e.g. passenger location is irrelevant for the navigate and put subtasks and it thus could be ignored.
A funnel action is an action that causes a larger number of initial states to be mapped into a small number of resulting states. E.g., the navigate(t) action maps any state into a state where the taxi is at location t. This means the completion cost is independent of the location of the taxi—it is the same for all initial locations of the taxi.
- E.g. if a task is terminated in a state s, then there is no need to represent its completion cost in that state
- Also, in some states, the termination predicate of the child task implies the termination predicate of the parent task
- reduce the amount memory to represent the Q-function.
14,000 q values required for flat Q-learning
3,000 for HSMQ (with the irrelevant-variable abstraction
632 for C() and V() in MAXQ
- learning faster
Model-based algorithms (that is, algorithms that try to learn P(s’|s,a) and R(s’|s,a)) are generally much more efficient because they remember past experience rather than having to re-experience it.