Symbolic Dynamic Programming

1. Symbolic Dynamic Programming Alan Fern * * Based in part on slides by Craig Boutilier

2. Planning in Large State Space MDPs • You have learned algorithms for computing optimal policies • Value Iteration • Policy Iteration • These algorithms explicitly enumerate the state space • Often this is impractical • Simulation-based planning and RL allowed for approximate planning in large MDPs • Did not utilize an explicit model of the MDP. Only used a strong or weak simulator. • How can we get exact solutions to enormous MDPs?

3. Structured Representations • Policy iteration and value iteration treat states as atomic entities with no internal structure. • In most cases, states actually do have internal structure • E.g. described by a set of state variables, or objects with properties and relationships • Humans exploit this structure to plan effectively • What if we had a compact, structured representation for a large MDP and could efficiently plan with it? • Would allow for exact solutions to very large MDPs

4. A Planning Problem

5. Logical or Feature-based Problems • For most AI problems, states are not viewed as atomic entities. • They contain structure. For example, they are described by a set of boolean propositions/variables • |S| exponential in number of propositions • Basic policy and value iteration do nothing to exploit the structure of the MDP when it is available

6. Solution? • Require structured representations in terms of propositions • compactly represent transition function • compactly represent reward function • compactly represent value functions and policies • Require structured computation • perform steps of PI or VI directly on structured representations • can avoid the need to enumerate state space • We start by representing the transition structure as dynamic Bayesian networks

7. Propositional Representations • States decomposable into state variables (we will assume boolean variables) • Structured representations the norm in AI • Decision diagrams, Bayesian networks, etc. • Describe how actions affect/depend on features • Natural, concise, can be exploited computationally • Same ideas can be used for MDPs

8. Robot Domain as Propositional MDP • Propositional variables for single user version • Loc (robot’s locat’n): Office, Entrance • T (lab is tidy): boolean • CR (coffee request outstanding): boolean • RHC (robot holding coffee): boolean • RHM (robot holding mail): boolean • M (mail waiting for pickup): boolean • Actions/Events • move to an adjacent location, pickup mail, get coffee, deliver mail, deliver coffee, tidy lab • mail arrival, coffee request issued, lab gets messy • Rewards • rewarded for tidy lab, satisfying a coffee request, delivering mail • (or penalized for their negation)

9. State Space • State of MDP: assignment to these six variables • 64 states • grows exponentially with number of variables • Transition matrices • 4032 parameters required per matrix • one matrix per action (6 or 7 or more actions) • Reward function • 64 reward values needed • Factored state and action descriptions will break this exponential dependence (generally)

10. Dynamic Bayesian Networks (DBNs) • Bayesian networks (BNs) a common representation for probability distributions • A graph (DAG) represents conditional independence • Conditional probability tables (CPTs) quantify local probability distributions • Dynamic Bayes net action representation • one Bayes net for each action a, representing the set of conditional distributions Pr(St+1|At,St) • each state variable occurs at time t and t+1 • dependence of t+1 variables on t variables depicted by directed arcs

11. T T(t+1) T(t+1) T 0.91 0.09 F 0.0 1.0 DBN Representation: deliver coffee RHM R(t+1) R(t+1) T 1.0 0.0 F 0.0 1.0 RHMt RHMt+1 Pr(RHMt+1|RHMt) Mt Mt+1 Pr(Tt+1| Tt) Tt Tt+1 L CR RHC CR(t+1) CR(t+1) O T T 0.2 0.8 E T T 1.0 0.0 O F T 0.1 0.9 E F T 0.1 0.9 O T F 1.0 0.0 E T F 1.0 0.0 O F F 0.1 0.9 E F F 0.1 0.9 Lt Lt+1 CRt CRt+1 RHCt RHCt+1 Pr(CRt+1 | Lt,CRt,RHCt) is the product of each of the 6 tables.

12. RHMt RHMt+1 Mt Mt+1 Tt Tt+1 s1 s2 ... s64 Lt Lt+1 s1 0.9 0.05 ... 0.0 s2 0.0 0.20 ... 0.1 . . . S64 0.1 0.0 ... 0.0 CRt CRt+1 RHCt RHCt+1 Benefits of DBN Representation Pr(St+1 | St) = Pr(RHMt+1,Mt+1,Tt+1,Lt+1,Ct+1,RHCt+1 | RHMt,Mt,Tt,Lt,Ct,RHCt) = Pr(RHMt+1 |RHMt) * Pr(Mt+1 |Mt) * Pr(Tt+1 |Tt) * Pr(Lt+1 |Lt) * Pr(CRt+1 |CRt,RHCt,Lt) * Pr(RHCt+1 |RHCt,Lt) • Only 20 parameters vs. • 4032 for matrix • Removes global exponential • dependence Full Matrix

13. Structure in CPTs • So far we have represented each CPT as a table of size exponential in the number of parents • Notice that there’s regularity in CPTs • e.g., Pr(CRt+1 | Lt,CRt,RHCt)has many similar entries • Compact function representations for CPTs can be used to great effect • decision trees • algebraic decision diagrams (ADDs/BDDs) • Here we show examples of decision trees (DTs)

14. RHMt RHMt+1 Mt Mt+1 Tt Tt+1 Lt Lt+1 CRt CRt+1 RHCt RHCt+1 Action Representation – DBN/DT CR(t) Decision Tree (DT) t RHC(t) t f f L(t) e o 1.0 1.0 0.2 0.1 Leaves of DT givePr(CRt+1=true | Lt,CRt,RHCt) e.g. If CR(t) = true & RHC(t) = false then CR(t+1)=TRUE with prob. 1 DTs can often represent conditional probabilities much morecompactly than a full conditional probability table

15. Reward Representation • Rewards represented with DTs in a similar fashion • Would require vector of size 2nfor explicit representation CR f t High cost for unsatisfied coffee request -100 M f t High, but lower, cost for undelivered mail T -10 f t Cost for lab being untidy Small reward for satisfying all of these conditions -1 1

16. Structured Computation • Given our compact decision tree (DBN) representation, can we solve MDP without explicit state space enumeration? • Can we avoid O(|S|)-computations by exploiting regularities made explicit by representation? • We will study a general approach for doing this called structured dynamic programming

17. Structured Dynamic Programming • We now consider how to perform dynamic programming techniques such as VI and PI using the problem structure • VI and PI are based on a few basic operations. • Here we will show how to perform these operations directly on tree representations of value functions, policies, and transitions functions • The approach is very general and can be applied to other representations (e.g. algebraic decision diagrams, situation calculus) and other problems after the main idea is understood • We will focus on VI here, but the paper also describes a version of modified policy iteration

18. Recall Tree-Based Representations Note: we are leaving off time subscripts for readability and using X(t), Y(t), …, instead. X t f X X e.g. If X(t)=false & Y(t) = true then Y(t+1)=true w/ prob 1 e.g. If X(t)=true THEN Y(t+1)=true w/ prob 0.9 0.0 1.0 X t f 0.9 Y Y Y t f Z 1.0 0.0 Z f t t f 10 0 1.0 Y Z Z t f Reward Function R 0.9 0.0 DBN for Action A Recall that each action of the MDP has its own DBN.

19. Structured Dynamic Programming • Value functions and policies can also have tree representations • Often much more compact representations than tables • Our Goal: compute the tree representations of policy and value function given the tree representations of the transitions and rewards

20. Recall Value Iteration Value Iteration: ;; could initialize to 0 Bellman Backup • Suppose that initial is compactly represented as a tree. • Show how to compute compact trees for • Use a max operation on the Q-trees (returns a single tree)

21. Symbolic Value Iteration . . . . . ? ? ? . . . . . Tree T Tree Symbolic MAX Tree V(X) ?

22. The MAX Trees Operation X 0.0 X 0.9 0.0 1.0 0.0 1.0 0.9 Y 1.0 1.0 0.0 Tree partitions the state space, assigning values to each region The state space max for the above trees is: In general, how can we compute the tree representing the max? 0.0 1.0 1.0

23. The MAX Tree Operation X 0.0 X 0.9 0.0 1.0 0.0 1.0 0.9 Y 1.0 1.0 0.0 Can simply append one tree to leaves of other. Makes all the distinctions that either tree makes. Max operation is taken at leaves of result. X X MAX Y X Y X 1.0 1.0, 0.9 0.9 0.0,0.9 X X X X 1.0 1.0 0.0 1.0 0.0, 1.0 1.0,1.0 0.0, 0.0 1.0, 0.0

24. The MAX Tree Operation X 0.0 X 0.9 0.0 1.0 0.0 1.0 0.9 Y 1.0 1.0 0.0 The resulting tree may have unreachable leaves. We can simplify the tree by removing such paths. X X Simplify 1.0 Y Y X 1.0 0.9 0.0 1.0 X X 1.0 1.0 0.0 1.0 unreachable

25. BINARY OPERATIONS(other binary operations similar to max)

26. MARGINALIZATION Compute diagram representing ∑A There are libraries for doing this.

27. Symbolic Bellman Backup for each action a Tree Tree Tree Tree

28. Symbol Tree Tree

29. Symbol Tree

30. Symbol Tree

31. Symbol Tree Tree

32. Symbol Tree

33. Symbolic Bellman Backup for each action a Tree Tree Tree Tree

34. SDP: Relative Merits • Adaptive, nonuniform, exact abstraction method • provides exact solution to MDP • much more efficient on certain problems (time/space) • 400 million state problems in a couple hrs • Can formulate a similar procedure for modified policy iteration • Some drawbacks • produces piecewise constant VF • some problems admit no compact solution representation • so the sizes of trees blows up with enough iterations • approximation may be desirable or necessary

35. Approximate SDP • Easy to approximate solution using SDP • Simple pruning of value function • Simply “merge” leaves that have similar values • Can prune trees [BouDearden96]or ADDs [StaubinHoeyBou00] • Gives regions of approximately same value

36. HCU HCR [9.00, 10.00] Loc Loc [5.19, 6.19] [7.45,8.45] [6.64, 7.64] A Pruned Value ADD HCU HCR W 9.00 10.00 W 5.19 R W W U 7.45 6.64 R R 6.19 5.62 U U 8.45 8.36 7.64 6.81

37. Approximate SDP: Relative Merits • Relative merits of ASDP fewer regions implies faster computation • 30-40 billion state problems in a couple hours • allows fine-grained control of time vs. solution quality with dynamicerror bounds • technical challenges: variable ordering, convergence, fixed vs. adaptive tolerance, etc. • Some drawbacks • (still) produces piecewise constant VF • doesn’t exploit additive structure of VF at all • Bottom-line: When a problem matches the structural assumptions of SDP then we can gain much. But many problems do not match assumptions.

38. Ongoing Work • Factored action spaces • Sometimes the action space is large, but has structure. • For example, cooperative multi-agent systems • Recent work (at OSU) has studied SDP for factored action spaces • Include action variables in the DBNs Statevariables Action variables