Incremental Pruning in POMDP for Belief State Value Iteration

1. Incremental Pruning CSE 574 May 9, 2003 Stanley Kok

2. Value-Iteration (Recap) • DP update – a step in value-iteration • MDP • S – finite set of states in the world • A – finite set of actions • T: SxA -> Π(S) (e.g. T(s,a,s’) = 0.2) • R: SxA -> R (e.g. R(s,a) = 10) • Algm

3. POMDP • <S, A, T, R, Ω, O> tuple • S, A, T, R of MDP • Ω – finite set of observations • O:SxA-> Π(Ω) • Belief state • - information state • – b, probability distribution over S • - b(s1)

4. POMDP - SE • SE – State Estimator • updates belief state based on • previous belief state last action, current observation • SE(b,a,o) = b’

5. POMDP - SE

6. POMDP - Π • Focus on Π component • POMDP-> “Belief MDP” • MDP parameters: • S => B, set of belief states • A => same • T => τ(b,a,b’) • R => ρ(b, a) • Solve with value-iteration algm

7. POMDP - Π • τ(b,a,b’) • ρ(b, a)

8. Two Problems • How to represent value function over continuous belief space? • How to update value function Vt with Vt-1? • POMDP -> MDP S => B, set of belief states A => same T => τ(b,a,b’) R => ρ(b, a)

9. Running Example • POMDP with • Two states (s1 and s2) • Two actions (a1 and a2) • Three observations (z1, z2, z3) 1D belief space for a 2 state POMDP Probability that state is s1

10. First Problem Solved • Key insight: value function • piecewise linear & convex (PWLC) • Convexity makes intuitive sense • Middle of belief space – high entropy, can’t select actions appropriately, less long-term reward • Near corners of simplex – low entropy, take actions more likely to be appropriate for current world state, gain more reward • Each line (hyperplane) represented with vector • Coefficients of line (hyperplane) • e.g. V(b) = c1 x b(s1) + c2 x (1-b(s1)) • To find value function at b, find vector with largest dot pdt with b

11. Second Problem • Can’t iterate over all belief states (infinite) for value-iteration but… • Given vectors representing Vt-1, generate vectors representing Vt

12. Horizon 1 • No future • Value function consists only of immediate reward • e.g. • R(s1, a1) = 0, R(s2, a1) = 1.5, • R(s1, a2) = 1, R(s2, a2) = 0 • b = <0.25, 0.75> • Value of doing a1 • = 1 x b(s1) + 0 x b(s2) • = 1 x 0.25 + 0 x 0.75 • Value of doing a2 • = 0 x b(s1) + 1.5 x b(s2) • = 0 x 0.25 + 1.5 x 0.75

13. Second Problem • Break problem down into 3 steps • -Compute value of belief state given action and observation • -Compute value of belief state given action • -Compute value of belief state

14. Horizon 2 – Given action & obs • If in belief state b,what is the best value of • doing action a1 and seeing z1? • Best value = best value of immediate action + best value of next action • Best value of immediate action = horizon 1 value function

15. Horizon 2 – Given action & obs • Assume best immediate action is a1 and obs is z1 • What’s the best action for b’ that results from initial b when perform a1 and observe z1? • Not feasible – do this for all belief states (infinite)

16. Horizon 2 – Given action & obs • Construct function over entire (initial) belief space • from horizon 1 value function • with belief transformation built in

17. Horizon 2 – Given action & obs • S(a1, z1) corresponds to paper’s • S() built in: • - horizon 1 value function • - belief transformation • - “Weight” of seeing z after performing a • - Discount factor • - Immediate Reward • S() PWLC

18. Second Problem • Break problem down into 3 steps • -Compute value of belief state given action and observation • -Compute value of belief state given action • -Compute value of belief state

19. Horizon 2 – Given action • What is the horizon 2 value of a belief state given immediate action is a1? • Horizon 2, do action a1 • Horizon 1, do action…?

20. Horizon 2 – Given action • What’s the best strategy at b? • How to compute line (vector) representing best strategy at b? (easy) • How many strategies are there in figure? • What’s the max number of strategies (after taking immediate action a1)?

21. Horizon 2 – Given action • How can we represent the 4 regions (strategies) as a value function? • Note: each region is a strategy

22. Horizon 2 – Given action • Sum up vectors representing region • Sum of vectors = vectors (add lines, get lines) • Correspond to paper’s transformation

23. Horizon 2 – Given action • What does each region represent? • Why is this step hard (alluded to in paper)?

24. Second Problem • Break problem down into 3 steps • -Compute value of belief state given action and observation • -Compute value of belief state given action • -Compute value of belief state

25. Horizon 2 a1 U a2

26. Horizon 2 This tells you how to act! =>

27. Purge

28. Second Problem • Break problem down into 3 steps • -Compute value of belief state given action and observation • -Compute value of belief state given action • -Compute value of belief state • Use horizon 2 value function to update horizon 3’s ...

29. The Hard Step • Easy to visually inspect to obtain different regions • But in higher dimensional space, with many actions and observations….hard problem

30. Naïve way - Enumerate • How does Incremental Pruning do it?

31. Incremental Pruning • How does IP improve naïve method? • Will IP ever do worse than naïve method? Combinations Purge/ Filter

32. Incremental Pruning • What’s other novel idea(s) in IP? • RR: Come up with smaller set D as argument to Dominate() • RR has more linear pgms but less contraints in the worse case. • Empirically ↓ constraints saves more time than ↑ linear programs require

33. Incremental Pruning Why are the terms after U needed? • What’s other novel idea(s) in IP? • RR: Come up with smaller set D as argument to Dominate()

34. Identifying Witness • Witness Thm: • -Let Ua be a set of vectors representing value function • -Let u be in Ua (e.g. u = αz1,a2 + αz2,a1 + αz3,a1 ) • -If there is a vector v which differs from u in one observation (e.g. v = αz1,a1 + αz2,a1 + αz3,a1) and • there is a b such that b.v > b.u, • -then Ua is not equal to the true value function

35. Witness Algm b’ b’’ b b’ b’’ • Randomly choose a belief state b • Compute vector representing best value at b (easy) • Add vector to agenda • While agenda is not empty • Get vector Vtop from top of agenda • b’ = Dominate(Vtop, Ua) • If b’ is not null (there is a witness), • compute vector u for best value at b’ and add it to Ua • compute all vectors v’s that differ from u at one observation and add them to agenda

36. Linear Support • If value function is incorrect, biggest diff is at edges (convexity)

37. Linear Support

38. Experiments • Comments???

39. Important Ideas • Purge()

40. Flaws • Insufficient background/motivation

41. Future Research • Better best-case/worse-case analyses • Precision parameter Є

42. Variants • Reactive Policy • - st = zt; • - π(z) = a • - branch & bound search • - gradient ascent search • - perceptual aliasing problem • Finite History Window • - π(z1…zk) = a • - Suffix tree to represent observation, leaf action • Recurrent Neural Nets • - use neural nets to maintain some state (so information about past is not forgotten)

43. Variants – Belief State MDP • Exact V, exact b • Approximate V, exact b • - Discreting b into a grid and interpolate • Exact V, approximate b • - Use particle filters to sample b • - track approximate belief state using DBN • Approximate V, Approximate b • - combine previous two

44. Variants - Pegasus • Policy Evaluation of Goodness And Search Using Scenarios • Convert POMDP to another POMDP with deterministic state transitions • Search for policy of transformed POMDP with highest estimated value

45. That’s it!