Computational Models of Human Intelligence

1. Computational Models of Human Intelligence Henry Kautz Department of Computer Science University of Rochester Autumn 2007

2. A Dream of AI • Systems that can understand ordinary human experience • Work in KR, NLP, vision, IUI, planning… • 1965 – 1985 • Scripts and plan recognition • Knowledge-based computer vision • Critical problem: scaling beyond microworlds • 1985 – 2000 • “Retreat” to research on fundamentals of inference, learning, and perception

3. Today • Resurgence of work in behavior recognition, fueled by • Advances in probabilistic inference • Graphical models • Scalable inference algorithms • “KR unites with Bayes” • Ubiquitous sensing devices • RFID, GPS, motes, … • Can directly sense many kinds of human activities • Practical applications • Healthcare & aging in place • Security • Consumer electronics

4. Growing Ubiquitous Sensing Infrastructure • GPS • Wi-Fi localization • RFID tags • Wearable sensors

5. Advances in Artificial Intelligence • Graphical models • Particle filtering • Belief propagation • Statistical relational learning

6. Crisis in Caring for the Cognitively Disabled • Epidemic of Alzheimer’s • Community integration of 7.5 million citizens with MR • 100,000 @ year disabled by TBI • Post-traumatic stress syndrome • Caregiver burnout

7. Levels of understanding • Physical movement • Hand touches cup • Behaviors • Drinking from a cup • Plans • Obtain cup; Fill cup; Drink from cup • Goals • Quench thirst

8. Dimensions of the behavior recognition problem • Keyhole versus interactive • Keyhole • Determine how an agent’s actions contribute to achieving possible or stipulated goals • No model of the observer – fly on the wall • Interactive • Actions performed by an agent to signal to another agent • Speech acts • Model social conventions & agents’ models of other agents

9. Dimensions of the plan recognition problem • Ideal versus fallible agents • Mistaken beliefs • John drives to Reagan, but flight leaves Dulles. • Cognitive errors • Distracted by the radio, John drives past the exit. • Irrationality • John furiously blows his horn at the car in front of him.

10. Dimensions of the plan recognition problem • Reliable versus unreliable observations • “There’s a 80% chance John drove to Dulles.” • Open versus closed worlds • Fixed plan library? • Fixed set of goals? • Metric versus non-metric time • John enters a restaurant and leaves 1 hour later. • John enters a restaurant and leaves 5 minutes later. • Single versus multiple ongoing plans

11. Dimensions of the plan recognition problem • Desired output: • Set of consistent plans or goals? • Most likely plan or goal? • Most critical plan or goal? • Interventions observer should perform to aid or hinder the agent?

12. (Some) formalisms used to model human behavior • Consistency-based • Scripts: hypothesize & revise • Plan libraries: Closed-world reasoning • Probabilistic • Bayesian networks • Hidden Markov models • Dynamic Bayesian networks • Stochastic grammars • Conditional random fields • Statistical-relational models

13. Scripts: hypothesize & revise • The Plan Recognition Problem C. Schmidt, 1978

14. Plan libraries: closed-world reasoning • A Formal Theory of Plan Recognition and its Implementation Henry Kautz, 1991

15. Bayesian Networks • E. Horvitz, J. Breese, D. Heckerman, D. Hovel, and K. Rommelse. The Lumiere Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users.Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, July 1998.

16. Hidden Markov Models D. Patterson, H. Kautz, & D. Fox, ISWC 2005

17. Layered hidden Markov models • N. Oliver, E. Horvitz, and A. Garg. Layered Representations for Recognizing Office Activity, Proceedings of the Fourth IEEE International Conference on Multimodal Interaction (ICMI 2002)

18. Stochastic grammars • Darnell Moore and Irfan Essa, "Recognizing Multitasked Activities from Video using Stochastic Context-Free Grammar", AAAI-02, 2002.

19. ck-1 ck gk-1 gk tk-1 tk mk-1 mk xk-1 xk qk-1 qk zk-1 zk Time k Time k-1 Dynamic Bayesian Nets Cognitive mode { normal, error } Learning and Inferring Transportation Routines Lin Liao, Dieter Fox, and Henry Kautz, Nineteenth National Conference on Artificial Intelligence, San Jose, CA, 2004. Goal Trip segment Transportation mode Edge, velocity, position Data (edge) association GPS reading

20. FRIEND’S HOME RESTAURANT OFFICE HOME PARKING LOT STORE Relational Conditional Random Fields Location-Based Activity Recognition using Relational Markov Networks  Lin Liao, Dieter Fox and Henry Kautz.  Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, Edinburgh, Scotland, 2005. s1 Global soft constraints p1 p2 Significant places a1 a2 a3 a4 a5 Activities g1 g2 g3 g4 g5 g6 g7 g8 g9 {GPS, location} time 

21. Assisted cognition Computer systems that improve the independence and safety of people suffering from cognitive limitations by… • Understanding human behavior from low-level sensory data • Using commonsense knowledge • Learning individual user models • Actively offering prompts and other forms of help as needed • Alerting human caregivers when necessary http://www.cs.rochester.edu/u/kautz/ac/

22. cognitive state intentions activities General architecture for model-based assistive systems common-sense knowledge decision making user profile physical behavior userinterface caregiveralerts machinelearning sensors

23. Laboratory for Assisted Cognition Environments • Henry Kautz • Sangho Park, research scientist • Craig Harmon, lab manager • Joseph Modayil, post doc Research • Robust behavior recognition using multiple sensors (vision, RFID, motion, …) • Applications: smart environments for support of persons with Alzheimers Disease, autism, and traumatic brain injury