Human action understanding as Bayesian inverse planning

1. Human action understanding as Bayesian inverse planning Josh Tenenbaum MIT BCS, CSAIL Collaborators: Chris Baker, Noah Goodman, Rebecca Saxe,

2. Everyday inductive leaps How can human minds infer so much about the world from such limited evidence? • Visual scene perception • Object categorization • Language understanding and acquisition • Causal reasoning • Social cognition • How do we infer the hidden mental states of other agents that give rise to their observed behavior – beliefs, desires, intentions, plans, emotions, values?

3. Why did the man cross the street? Far more complex than conventional “activity recognition” systems…

4. The solution Background knowledge (“intuitive theories”, “inductive bias”). • How does background knowledge guide inferences from sparsely observed data? • What form does background knowledge take, across different domains and tasks? • How is background knowledge itself acquired? The challenge: Can we answer these questions in precise computational terms?

5. The approach • How does background knowledge guide inferences from sparsely observed data? Bayesian inference: 2. What form does background knowledge take, across different domains and tasks? Probabilities defined over structured representations: graphs, grammars, predicate logic, schemas, functional processes, programs. 3. How is background knowledge itself acquired? Hierarchical probabilistic models, with inference at multiple levels of abstraction.

6. The approach • How does background knowledge guide inferences from sparsely observed data? Bayesian inference: 2. What form does background knowledge take, across different domains and tasks? Probabilities defined over structured representations: graphs, grammars, predicate logic, schemas, functional processes, programs. 3. How is background knowledge itself acquired? Hierarchical probabilistic models, with inference at multiple levels of abstraction. Which does “more work” – Bayesian inference or structured knowledge? Not the right question! Each component needs the other to get the hard work done.

7. Principle of rationality Goals (G) Beliefs (B) • People assume that other agents will tend to take sequences of actions that most effectively achieve their goals given their beliefs. • Model this more formally as inverse planning in a goal-based Markov Decision Process (MDP). Rational Planning (MDP) Actions (A) c.f., inverse RL (Ng, Russell); goal-based imitation (Rao et al.)

8. Principle of rationality Goals (G) Beliefs (B) • People assume that other agents will tend to take sequences of actions that most effectively achieve their goals given their beliefs. • Model this more formally as inverse planning in a goal-based Markov Decision Process (MDP). Rational Planning (MDP) Actions (A) Caution: not necessarily a computational model of how people plan! It is a computational model of people’s mental models of how people plan. C.f. simulation/theory debate. c.f., inverse RL (Ng, Russell); goal-based imitation (Rao et al.)

9. Rational action understanding in infants (Gergely & Csibra) Rational action schema

10. Inferring relational goals • Hamlin, Kuhlmeier, Wynn, Bloom: helping, hindering • Southgate, Csibra: chasing, fleeing

11. Immediate research aims • Test how accurately and precisely the Bayesian inverse planning framework can explain people’s intuitive psychological judgments. • Use the inverse planning framework as a tool to test alternative accounts of people’s intuitive theories of psychology. • Focus on perceptual inferences about goals (c.f., Brian Scholl’s talk).

12. Alternative computational models • Theory-based models • Objects as goals • fixed or changing goals? • simple or complex intentions? • Social goals towards other agents • try to achieve some relation to another agent, e.g., • chasing: try to minimize distance to another agent • fleeing: try to maximize distance from another agent • goals have different content depending on how agents model each other, e.g. • first-order (“simple-minded”) or second-order (“smarter”): does an agent ignore or try to account for other agents’ goals and planning processes?

13. Alternative computational models • Theory-based models • Bottom-up heuristics based on motion cues (Todd & Miller, Zacks, Tremoulet & Feldman) • Object O is the goal of X if O is a salient object and X is moving towards O. • X is chasing/fleeing Y if X tends to move towards/away from Y as Y tends to move away from/towards X.

14. Experiment 1: objects as goals • Method • Subjects (N=16) view animated trajectories in simple maze-like environments. • Subjects observe partial action sequences with several candidate goals and are asked to rate relative probabilities of goals at different points along each trajectory.

15. Experiment 1: objects as goals • Method • Subjects (N=16) view animated trajectories in simple maze-like environments. • Subjects observe partial action sequences with several candidate goals and are asked to rate relative probabilities of goals at different points along each trajectory. • Set up • Cover story: intelligent aliens moving in their natural environment. • Assume fully observable world: agent’s beliefs = true states and transition functions of the environment. • 100 total judgments, with 3-6 judgment points along each of 36 different trajectories (= 4 goal positions x 3 kinds of obstacles x 3 goals).

16. Like “Chasing subtlety”, but more abstract Specific inverse planning models • Model M1(β): fixed goal • The agent acts to achieve a particular state of the environment. • This goal state is fixed for a given action sequence. • Small negative cost for each step that does not reach the goal.

17. Specific inverse planning models • Model M2(β,γ): switching goals • Just like M1, but the agent’s goal can change at any time step with probability γ. • Agent plans greedily, not anticipating its own potential goal changes. • Special case where γ = 1/# goals is a “one-step rule”, equivalent to a simple motion cue: • “Object O is the goal of X if O is a salient object and X is moving towards O.”

18. Results People M2(2,0.33) Goal switch

19. Model fits M1(β) Fixed goal People M2(β,γ) Goal switching People

20. Analysis of critical trials Critical stimuli

21. A 10 C A C 10 11 11 B B Human goal inferences M2(2,0.1) Goal switch M1(2) Fixed goal • Motion-cue heuristic: “Object O is the goal of X if O is a salient object and X is moving towards O.” M2(2,0.67) “one-step” rule

22. Experiment 2: social goals toward other agents • Method • Subjects view short animated trajectories (2 or 3 steps) of two agents moving in simple maze-like environments. • One agent is chasing the other, who is fleeing. Subjects are asked to rate the relative probability that each agent is the chasing agent. Which is more likely? Green is chasing & Red is fleeing. Or Red is chasing & Green is fleeing.

23. Experiment 2: social goals toward other agents • Design aim: distinguish theory-based accounts from simpler models based on motion cues. • Previous evidence: Southgate & Csibra Demo • Critical stimuli in our study: e.g., Which is more likely? Green is chasing & Red is fleeing. Or Red is chasing & Green is fleeing.

24. 1 2 3 Which is more likely? Green is chasing & Red is fleeing. Or Red is chasing & Green is fleeing. 4 5 6 7 8 9 People First-order model 1 First-order model 2 Second-order model 1 Second-order model 2 Green is chasing & Red is fleeing. Red is chasing & Green is fleeing.

25. Summary • Bayesian inverse planning gives a framework for making rich mental-state inferences from sparsely observed behavior, assuming the principle of rationality. • At least in simple environments, inverse planning models provide a strong fit to people’s goal attribution judgments – and a better fit than bottom-up accounts based purely on motion cues. • Comparing inverse planning models with different hypothesis spaces lets us probe the nature and complexity of people’s representations of agents and their goals.

26. Ongoing and future work • In perceiving social-relational goals, can we distinguish first-order (“simple-minded”) agents from second-order (“smart”) agents? • A demo: • Agent profile: 1 Env: 1 • Agent profile: 1 Env: 2 • Agent profile: 4 Env: 1 • Agent profile: 4 Env: 2

27. Ongoing and future work • In perceiving social-relational goals, can we distinguish first-order (“simple-minded”) agents from second-order (“smart”) agents? • Scaling up to more complex environments and richer mental-state representations: e.g., hierarchical goal structures, plans, recursive beliefs. • What is the relation between actual psychology (how people actually think and plan) and intuitive psychology? Intuitive psychology may be more rational… relevance for the “theory vs. simulation” debate? • How could a theory of mind be learned or modified with experience? Aspects to be learned might include: how complex are agents’ goals, how complex are agents’ representations of other agents, different types of agents, how do agents’ beliefs depend on the state of the world.

30. Alternative computational models • Theory-based models • Objects as goals • fixed or changing goals? • simple or complex intentions? • Social goals towards other agents • first-order (“simple-minded”): try to achieve some relation to another agent without modeling that agent’s own action-planning process. • E.g., chasing: try to minimize distance to another agent; fleeing: try to maximize distance from another agent. • second-order (“smart”): try to achieve some relation to another agent while modeling that agent’s action planning process.

31. A C A C B B Human goal inferences M2(2,0.1) Goal switch M1(2) Fixed goal • Motion-cue heuristic: “Object O is the goal of X if O is a salient object and X is moving towards O.” M2(2,0.67) “one-step” rule

32. Rational action understanding in infants (Gergely & Csibra) Rational action schema:

33. The present research • Aims • To test how accurately and precisely the inverse planning framework can explain people’s intuitive psychological judgments. • To use the inverse planning framework as a tool to test alternative accounts of people’s intuitive theories of psychology. • Experiments 1 & 2: goal inference • Experiments 3 & 4: action prediction

34. An alternative heuristic account? A thought experiment… M2(2,0.1) Goal switch M2(2,0.67) One-step heuristic

35. An alternative heuristic account? • Last-step heuristic: infer goal based on only the last movement (instead of the entire path) • a special case of M2, equivalent to M2(β,.67). • This model correlates highly with people’s judgments in Experiment 1. • However, there are qualitative differences between this model’s predictions and people’s judgments that suggest that people are using a more sophisticated form of temporal integration.

36. Sample behavioral data

37. Modeling results People M2(2,0.33) Goal switch

38. 1 2 3 Which is more likely? Green is chasing & Red is fleeing. Or Red is chasing & Green is fleeing. 4 5 6 7 8 9 People First-order model 1 First-order model 2 Second-order model 1 Second-order model 2 Green is chasing & Red is fleeing. Red is chasing & Green is fleeing.

39. M2(2,0.67) “Last-step” heuristic C C A A C A A C B B B B Human goal inferences M2(2,0.1) Goal switch M1(2) Fixed goal

40. The solution Prior knowledge (inductive bias).

41. Caution! Goals (G) Beliefs (B) • We are not proposing this as a computational model of how people plan! It is a computational model of people’s mental model of how people plan. • Whether planning is “rational” or well-described as solving an MDP is an interesting but distinct question. Rational Planning (MDP) Actions (A)