CS 88/188 Computational Models of Human Behavior

1. CS 88/188Computational Models of Human Behavior http://www.cs.dartmouth.edu/~tanzeem/teaching/CS188-Fall08/ Instructor: Tanzeem Choudhury tanzeem.choudhury@dartmouth.edu

2. Acknowledgement • These slides include material from different talks and papers to illustrates some of the ideas we will cover in the class – they include some of my work but also work done by my collaborators and students • Matthai Philipose, Jim Rehg, James Kitts, Jeff Bilmes, Lin Liao, Maryam Mahdaviani, Jianxin Wu, Danny Wyatt, Jonathan Lester, Dieter Fox, Henry Kautz, Gaetano Borriello

3. Introductions • Tell us a bit about … • yourself • your research interests • what you want to get out of this class

4. Why do we want to build systems that can interpret human behavior? Necessary for building technology that is: • Assistive • Context aware • Natural user interface

5. Eldercare -Activities of Daily Living (ADLs) Classes of day-to-day activities that: • Indicate cognitive well-being • Encompass most activities an elder may perform

6. What is grandma doing? Professional caregivers must • assess elder’s competence • fill out rating form Tedious, error-prone, expensive Fill ADL forms automatically?

7. Autism • Kids with autism do not learn normal conversational protocols, e.g. turn-taking, intonation, personal space, etc. They often do not perceive objects or understand what the objects signify. They often have high I.Q. and work very well with technology and interested in modifying their behavior.

8. How is this done currently? • “behaviorally-based interventions [as opposed to pharmacological] have been found to be by far the most effective means of intervention” • Treatment for autistic children involve: “(a) high-density reinforcement to “prime” social responding … (c) social skills training (d) adult-mediated prompting and reinforcement (e) self-monitoring” • Continuous or incremental monitoring and intervention impossible: “child-specific interventions for children with autism may, in isolation, have limited potential”. “The more intense the intervention, the greater the gain”.

9. Future of mobile computing • What are you doing? • What do you like? • Who do you hang out with?

10. What makes people tick? • Why do some people become more influential than others? • Why are some people more well-connected than others?

11. How to define activities? Mom has an active and healthy lifestyle activities Mom eats regularly Mom exercises regularly … Eat breakfast Eat lunch Biking Stretch … actions Get on Bike Pedal* … operations

12. Building people-aware systems • Sensing: gather information about actions and context • Modeling: develop features and algorithms that are useful for recognizing activities • Applications: make activity information actionable rfid# e00700017fab778 rfid# e00700017fab783 privacy scalability Two people meeting … here as well brushing teeth eating ? speaking volume walking speaking rate running meeting cooking

13. difficult (for computer) very difficult What can activity recognition system do for you? monitor Logging What did they do? RatingHow well? Anomaly detection What was wrong? Trending How have they changed? Notification Call me when they need me Prompting Walk them through it Actuating (Help) do it. act

14. Challenge: Getting useful information is tough privacy ergonomics scalability

15. New ideas that let us address the sensor challenge Application • RFID tags and readers allow a simple, robust sensor stack for object-person interactions Tracker Object ID RFID Object Recognition: F (lighting conditions, Object being detected, Kinematic signatures, etc.) Other Sensor Signals

16. Who are you? ID # 1287678087889343 1 cm Radio Frequency Identification (RFID) Tiny, 40-cent battery-free tags +ambient readers Tags return 96-bit ID when queried by readers <ID = e3f000e13431, desc = “bread basket”, manufacturer = “…”, … >

17. iBracelet: Proximity = Use • 13.56MHz reader, radio, power supply, antenna • 12 inch range, 12-150 hr lifetime • Objects tagged on grasping surfaces

18. WISP: Wireless ID & Sensing Platform Passive RFID+Sensors • small stickers • battery-free • room-range motion sensor wisp

19. Mobile sensing platform (MSP) • Multi-modal instead of multiple locations • Complementary modalities • audio, acceleration, temperature, light, humidity, pressure & compass • Wearable • always with user, controlled by user, compact

20. Embedded processing and communication • Desirable for interactive applications • Big advantages in terms of privacy MSP: sensing, data processing, and communication

21. A Five Minute Data Trace

22. Acceptable privacy solution is important • Nobody wants to record everything they say • need to protect people’s privacy and privacy of anyone they come into contact with • Process audio on the fly so that it is impossible to reconstruct the words later

23. Privacy-sensitive processing of speech • Record and store information about how • Throw away information about what paralinguistic information

24. speaking time & interruptions speaking rate, changes in loudness & pitch differences in speaking rate intonation, turn-taking turn-taking, loudness, interruptions status, dominance & roles emotional state bi-polar disorder, depression autism, sociability conversation types Privacy-sensitive modeling can infer meaningful attributes of interaction

25. Log energy B lag Speaker A Speaker B Neither + Both Log energy A time Log entropy of energy time Privacy sensitive features • For detecting speech • relative spectral entropy • # of auto-correlation peaks • non-initial max auto-correlation peak • For segmenting speakers • pair-wise energies • For detecting un-mic’d speakers • entropy of energy across mics frequency

26. A few features commonly used

27. Challenges in modeling human behavior • Reduce the amount of human effort required in • feature engineering • labeled data for training • Develop models that are • rich yet tractable • parameters the are interpretable • smooth • adapts over time

28. Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature . Feature Feature Feature . Feature Feature Feature Feature Feature Feature Feature . Feature Feature Feature Feature Automatically selecting useful features All the features you can think of Features useful for classification

29. Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Why not use all the features? • High dimensional data means: • learning a huge number of parameters • training becomes extremely expensive • and resulting models become brittle

30. Example classification

31. Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature Feature . Feature Feature Feature Feature . Feature Feature . Feature Feature Feature Feature Feature Want to combine the benefits • Discriminative feature selection capability of boosting • Ability to encode dependencies over time +

32. Temporal information results in smoother classification

33. Reduce dependency on human provided labels tissue, razor, toothpaste, shaving cream, razor shave your face television, couch, remote, couch watch tv dishwasher, cabinet, sink, cabinet, silverware, dishwasher load and run a dishwasher bathroom, light, toothpaste, toothbrush, cabinet brush your teeth cleaner, cleaner, cleaner, cabinet, toothbrush, light, bathroom clean the bathroom highchair, diaper, clothes, towel, highchair, towel, diaper change a baby’s diaper tissue, razor, toothpaste, shaving cream, razor, television, couch, remote, couch, dishwasher, cabinet, sink, cabinet, silverware, dishwasher, bathroom, light, toothpaste, toothbrush, cabinet, cleaner, cleaner, cleaner, cabinet, toothbrush, light, bathroom, highchair diaper, clothes, towel, highchair, towel, diaper

34. custom models Can we automatically mine common sense? unlabeled data (objects used) text • Techniques for mining generic activity models from the web generic models mine label labeled data learn Joint work with Matthai Philipose and Danny Wyatt

35. web Mining Models from the Web “how to” pages coffee water spoon milk sugar “make coffee” extract objects find “how to” pages compute usage probability p(use spoon | make coffee) assemble model Joint work with Matthai Philipose and Danny Wyatt

36. custom models Bootstrapping a Customized Model unlabeled data (objects used) text generic models mine label labeled data learn Use mined model to label unlabeled traces Learn new model from those labeled traces Joint work with Matthai Philipose and Danny Wyatt

37. Dealing with incompleteness Preparing pasta … pot kitchen range spaghetti spoon pan stove macaroni fork

38. WordNet ontology generation Entity Substance Object Artifact solid … … … Cutlery Food Cooking Utensil Kitchen Appliance spoon pasta stove pot spaghetti

39. WordNet ontology expansion Entity Substance Object Artifact solid … … … Cutlery Food Cooking Utensil Kitchen Appliance spoon pasta stove pot spaghetti

40. WordNet ontology expansion Entity Substance Object Artifact solid … … … Cutlery Food Cooking Utensil Kitchen Appliance fork spoon knife pasta hotplate stove microwave pan pot poacher linguine spaghetti macaroni

41. Acquire labels from complementary channels • Leverage synergy between different sensor modalities • RFID provides sparse and noisy labels • Use these labels to train object model from video • Provide basic information about objects likely to be used in a given activity • making tea involves teabag, teacup, hot water • Fuse RFID, vision, and prior knowledge to improve recognition

42. O1 O2 O3 O3 … … R1 R2 R3 R4 Infer object usage from sparse RFID measurements • RFID measurements: Rt • Object in use: Ot

43. A1 A2 A3 A4 O1 O2 O3 O3 … … R1 R2 R3 R4 Incorporate commonsense information • Prefer objects that are usually used in activity At

44. Ot Vt Learn object models without manual labeling • Use the marginal probabilities of object usage to train object histograms without any human provided labels

45. A1 A2 A3 A4 O1 O2 O3 O3 … … R1 R2 R3 R4 V1 V2 V3 V4 Put everything together

46. Objects and activities modeled • Water jug • Kettle • Teabag • Cup • Spoon • Milk • Honey • Cereal • Bowl • Coffee • Creamer • Sugar • Cheese • Bread • Knife • Toaster • Plate • Butter • Peanut butter • Jelly • Lunch bag • Plant • Plant care • Watering can • Pillbox • Salad tosser • Salad dressing • Meat • Microwave • Popcorn • Juice • Cloth • phone • Boiling water • Making tea • Preparing cereal • Making coffee • Making cheese sandwich • Making buttered toast • Making peanut butter sandwich • Packed lunch • Tending plants • Taking medicine • Making TV dinner • Making salad • Making popcorn • Drinking juice • Wiping counter • Making a phone call Objects Activities

47. Future opportunities Need to … • further reduce the amount of human engineering and effort required • develop systems that adapt over time • take advantage of the community of users

48. Goal of the seminar • Learn about the various machine learning approaches developed to make computers more aware of people, their activities, and their surrounding context. • Discuss the various research challenges in data collection, representation and tractability of models, and evaluation. • Brain-storm ideas on how future research can go about tackling some of these challenges.

49. Written Critique • Email a one-page written critique for each paper - due at 6pm the day before the class • put ‘CS 188 assignment’ in your subject line • The critique should • identify the technical challenges the researchers are trying to solve • describe the new ideas • discuss the strength and limitations of the proposed approach • suggest potential improvements

50. Presentation • Email me your top 3 choices by end of today • Don’t just summarize the sections • Be critical yet constructive • Slides matters – spend time on the structure and the overall presentation • Don’t be afraid to be controversial • Have some questions ready to spur discussion