Towards robotic assistants in nursing homes: challenges and results

1. Towards robotic assistants in nursing homes:challenges and results Joelle Pineau Michael Montemerlo Martha Pollack * Nicholas Roy Sebastian Thrun Carnegie Mellon University *University of Michigan

2. Introducing Pearl – A mobile robotic assistant for elderly people and nurses ROLE: Reminding to eat, drink, take meds cameras Providing info (TV, weather) LCD mouth Monitoring Rx adherence & safety Calling for help in emergencies microphone & speakers Supporting communication touchscreen Remote health services handle bars Providing physical assistance carrying tray Management support of ADLs laser Linking caregiver and resources sonars Moving things around mobile base The Nursebot Project

3. The Nursebot project in its early days The Nursebot Project

4. Architecture High-level controller Navigation Cognitive support Communication The Nursebot Project

5. Architecture High-level controller Navigation Cognitive support Communication • Localization and map building • (Burgard et al., 1999) • People detection and tracking • (Montemerlo et al., 2002) The Nursebot Project

6. Architecture High-level controller Navigation Cognitive support Communication • Autominder system • (Pollack et al., 2002) The Nursebot Project

7. Architecture High-level controller Navigation Cognitive support Communication • Speech recognition: Sphinx system • (Ravishankar, 1996) • Speech synthesis: Festival system • (Black et al., 1999) The Nursebot Project

8. The role of the top-level controller • Établir les priorités parmi les objectifs des différents modules • Négocier entre plusieurs objectifs ayant des coûts/gains variés • Négocier entre l’acquisition d’information et la rencontre des objectifs • Passer d’une tâche à l’autre en partageant l’information sensorielle • Planifier malgré la présence d’incertitude High-level controller Navigation Cognitive support Communication • ACTION SELECTION - based on the trade-off between: • - goals from different modules; • - goals with varying costs / rewards; • - reducing uncertainty versus accomplishing goals. The Nursebot Project

9. Speech recognition with Sphinx The Nursebot Project

10. Robot control under uncertainty Belief State P(st=weather-today)=0.5 P(st=appointment-today )=0.5 Speech=“today” State weather-today USER Action={ say-weather, update-appointment, clarify-query} The Nursebot Project

11. Robot control using Partially Observable Markov Decision Processes (POMDPs) Belief state Observations Costs / Rewards P(s1) P(s2) State USER + ENVIRONMENT + WORLD Actions Problem: Which action allows the robot to maximize its reward? The Nursebot Project

12. Methods to solve POMDPs Objective: Find a policy, (b), which maximizes reward. POMDP New methods? Performance AMDP FIB QMDP UMDP MDP O(S2AT) O(S2AO ) O(S2A) O(S2AO) O(S2AB) T Complexity The Nursebot Project

13. New approach: A hierarchy of POMDPs Idea: Exploit domain knowledge to divide one POMDP into many smaller ones. Motivation: Complexity of POMDP solving grows exponentially with # of actions. Assumption: We are given POMDP M = {S,A,,b,T,O,R} and hierarchy H subtask Act abstract action ExamineHealth Navigate Move ClarifyGoal VerifyPulse VerifyMeds primitive action North South East West The Nursebot Project

14. PolCA: Planning with a hierarchy of POMDPs Step 1: Select the action set Navigate AMove = {N,S,E,W} Move ClarifyGoal West South East North ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds The Nursebot Project

15. PolCA: Planning with a hierarchy of POMDPs Step 1: Select the action set Step 2: Minimize the state set Navigate AMove = {N,S,E,W} SMove = {X,Y} Move ClarifyGoal West South East North ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus The Nursebot Project

16. PolCA: Planning with a hierarchy of POMDPs Step 1: Select the action set Step 2: Minimize the state set Step 3: Choose parameters Navigate AMove = {N,S,E,W} SMove = {X,Y} Move ClarifyGoal West South East North ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus PARAMETERS {bh,Th,Oh,Rh} The Nursebot Project

17. PolCA: Planning with a hierarchy of POMDPs Step 1: Select the action set Step 2: Minimize the state set Step 3: Choose parameters Step 4: Plan task h Navigate AMove = {N,S,E,W} SMove = {X,Y} Move ClarifyGoal West South East North ACTIONS North South East West ClarifyGoal VerifyPulse VerifyMeds STATE FEATURES X-position Y-position X-goal Y-goal HealthStatus PARAMETERS {bh,Th,Oh,Rh} PLAN h The Nursebot Project

18. First study in simulation • 20-questions domain: • Agent:“Is it an animal?” • User: “No.” • Agent:“Is it a vegetable?” • User: “Yes.” • Agent:“Is it green?” • User: “No.” • Agent:“...?” Actions Observations Objective: Plan a sequence of questions allowing the agent to identify the chosen object. Small complication…. the user can change objects at any time (Pr = 0.1), without telling the agent. The Nursebot Project

19. The hierarchy Begin Vegetables Minerals Animals Vegetable? Mineral? Animal? … … Mammal? Herbivore? RABBIT TURTLE …? The Nursebot Project

20. Methods to solve POMDPs POMDP PolCA Performance AMDP FIB QMDP UMDP MDP Complexity The Nursebot Project

21. Results Domain: |S|=12, |A|=20, |O|=3 The Nursebot Project

22. A new decomposition Begin Vegetables Minerals Animals Vegetable? Mineral? Animal? … … … The Nursebot Project

23. A new decomposition Begin Vegetables Minerals Animals Vegetable? Mineral? Animal? … … Fruits Plants Fruit? … … The Nursebot Project

24. Results Domain: |S|=12, |A|=20, |O|=3 The Nursebot Project

25. Methods to solve POMDPs POMDP PolCA Performance AMDP FIB QMDP UMDP MDP Complexity The Nursebot Project

26. PolCA in the Nursebot domain • Goal: A robot is deployed in a nursing home, where it provides reminders to elderly users and accompanies them to appointments. • Domain: |S|=512, |A|=20, |O|=19 • Hierarchy: The Nursebot Project

27. Sample scenario The Nursebot Project

28. Results for dialogue system POMDP policy MDP policy 0.18 0.1 0.1 The Nursebot Project

29. Summary • We have developed a first prototype robot able to serve as a mobile nursing assistant for elderly people. • The top-level controller uses a hierarchical variant of POMDPs to select actions. • This allows it to acquire necessary information and successfully complete assigned tasks. • Probabilistic techniques have been found to be very useful to flexibly model and track individuals. The Nursebot Project

30. The Nursebot team U. of Pittsburgh - Nursing: Jacqueline Dunbar-Jacobs Sandra Engberg Judith Matthews U. of Pittsburgh - CS: Don Chiarulli Colleen McCarthy U. of Freiburg - CS: Maren Bennewitz Wolfram Burgard Dirk Schulz U. of Michigan - CS: Laura Brown Dirk Colbry Cheryl Orosz Bart Peintner Martha Pollack Sailesh Ramakrishnan Standard Robotics: Greg Baltus CMU - Robotics: Greg Armstrong Michael Montemerlo Joelle Pineau Nicholas Roy Jamie Schulte Sebastian Thrun CMU - HCI/Design: Francine Gemperle Jennifer Goetz Sarah Kiesler Aaron Powers For more details: www.cs.cmu.edu/~nursebot The Nursebot Project

31. Our vision of robotic healthcare Management support of ADLs Providing information (TV, weather) Reminding to eat, drink, & take meds Monitoring Rx adherence & safety Linking the caregiver to resources Calling for help in emergencies Providing physical assistance Supporting inter-personal communication Moving things around Enabling use of remote health services The Nursebot Project

32. Profile of an aging population • 450,000 nurses to recruit before 2008 (USA) The Nursebot Project

33. Localization and map building The Nursebot Project

34. People tracking The Nursebot Project

35. Analysis of movements The Nursebot Project

36. Autominder System The Nursebot Project

37. The family of Markov models State ambiguity? no yes no Choice of action? Hidden Markov Model (HMM) Markov Chain Partially Observable Markov DecisionProcess (POMDP) yes Markov Decision Process (MDP) The Nursebot Project

38. The POMDP model The POMDP is a septuple { S, A, , b, T, O, R } Problem: Which action allows the robot to maximize its reward? rt-1 rt State: ... st-1 st ... ot-1 ot Action: at-1 ?? ... bt-1 bt ... Belief: The Nursebot Project

39. Execution with a hierarchy of POMDPs At each time step, traverse the hierarchy from top to bottom. For each subtask, consult the policy ath(bt) If at is an internal node > move to that subtask. Act ExamineHealth Navigate Move ClarifyGoal VerifyPulse VerifyMeds North South East West The Nursebot Project

40. Navigate Move ClarifyGoal West South East North How to choose parameters? • Consider for example TNavigate(s,a,s’) • Case #1:ais a primitive action. TNavigate(s,ClarifyGoal,s’)  T(s,ClarifyGoal,s’) S’ S’ 0.7 North S ClarifyGoal South S’ 0.3 S’ The Nursebot Project

41. Navigate Move ClarifyGoal West South East North How to choose parameters? • Consider for example TNavigate(s,a,s’) • Case #1:ais a primitive action. TNavigate(s,ClarifyGoal,s’)  T(s,ClarifyGoal,s’) • Case #2: ais an abstract action. TNavigate(s,Move,s’)  T(s, Move(s) ,s’) S’ S’ 0.9 North 0.2 ClarifyGoal S 0.1 South S’ S’ 0.8 The Nursebot Project

42. The details States: 1 per object (e.g. tomato, cucumber, rabbit, turtle, tulip, ….) Observations: “yes”, “no”, “” Actions: 1 guess per objet Rewards: question = -1 + many questions correct guess = +5 incorrect guess = -20 Begin Vegetables Minerals Animals Vegetable? Mineral? Animal? … … Mammal? Herbivore? RABBIT TURTLE …? The Nursebot Project