Relating Reinforcement Learning Performance to Classification performance

1. Relating Reinforcement Learning Performance to Classification performance Presenter: Hui Li Sept.11, 2006

2. Outline • Motivation • Reduction from reinforcement learning to classifier learning • Results • Conclusion

3. Motivation A Simple Relationship: The goal of reinforcement learning: The goal of (binary) classifier learning:

4. Motivation Question: • The problem of classification has been intensively investigated • The problem of reinforcement learning is still under investigation • Is it possible to reduce the reinforcement learning to classifier learning ?

5. Reduction Definition: • What is reinforcement learning problem A reinforcement learning problem D is defined as a conditional probability table D(o’,r|(o,a,r)*,o,a) on a set of observations O and rewards r[0,) given any history of past observations (o,a,r)*, actions (from action set A), and rewards.

6. Reduction 2. What is reinforcement learning goal Given some horizon T, find a policy , Maximizing the expected sum of rewards:

7. Reduction How to reduce a reinforcement learning problem to a cost-sensitive classification problem • How to obtain training examples • How to obtain training label • How to define the cost of misclassification

8. Reduction A illustration of trajectory tree • M = {S, A, D, Ps,a} • Two actions {0,1} • non-stationary policy

9. Reduction The value of the policy of a single step is estimated Which is explicitly written by The goal is the i-th realization

10. Reduction Value of the policy of a single step S0 = s0n S0 = s01 S0 = s02 . . . … a = 0 … … a = 0 a = 0 a = 1 a = L-1 a = 1 a = L-1 a = 1 a = L-1 S1|0 S1|0 S1|0 S1|1 S1|L-1 S1|1 S1|L-1 S1|1 S1|L-1 the i-th realization

11. Reduction One step reduction: One step reinforcement learning problem Cost-sensitive classifier learning problem

12. Reduction where • s0i: the ith sample (or data) • : label • wi: the costs of classifying example i to each of of possible labels.

13. Reduction Properties of cost • The cost for misclassification is always positive • The cost for correct classification is zero • The larger the difference between the possible actions in terms of future reward, the larger the cost (or weight)

14. Reduction T-step MDP reduction How to find good policies for a T-step MDP by solving a sequence of weighted classification problems T-step policy  =(0, 1, … T-1) • When updating t, hold the rest constant • When updating t, the trees are pruned form the root to stage t by keeping only the branch which agree with controls 0, 1, … t-1

15. Reduction Rewards accumulated along the branch which agrees with the controls t+1, t+2, … T-1 Immediate reward Realization of the reward follows actions at stage t

16. Illustrative Example Two-step MDP problem: • Continuous state space S = [0, 1] • Binary action Space A = {0,1} • Uniform distribution over the initial state

17. Illustrative Example Value function

18. Illustrative Example Path Taken by the algorithm