recognizing human activity from sensor data n.
Skip this Video
Loading SlideShow in 5 Seconds..
Recognizing Human Activity from Sensor Data PowerPoint Presentation
Download Presentation
Recognizing Human Activity from Sensor Data

Loading in 2 Seconds...

play fullscreen
1 / 42

Recognizing Human Activity from Sensor Data - PowerPoint PPT Presentation

  • Uploaded on

Recognizing Human Activity from Sensor Data. Henry Kautz University of Washington Computer Science & Engineering graduate students : Don Patterson, Lin Liao CSE faculty : Dieter Fox, Gaetano Borriello UW School of Medicine : Kurt Johnson

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Recognizing Human Activity from Sensor Data' - ula

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
recognizing human activity from sensor data
Recognizing Human Activity from Sensor Data

Henry Kautz

University of WashingtonComputer Science & Engineering

graduate students: Don Patterson, Lin Liao

CSE faculty: Dieter Fox, Gaetano Borriello

UW School of Medicine: Kurt Johnson

Intel Research: Matthai Philipose, Tanzeem Choudhury

converging trends
Converging Trends…
  • Pervasive sensing infrastructure
    • GPS enabled phones
    • RFID tags on all consumer products
    • Wireless motes
  • Breakthroughs in core artificial intelligence
    • After “AI boom” fizzled, basic science went on…
    • Advances in algorithms for probabilistic reasoning and machine learning
      • Bayesian networks
      • Stochastic sampling
    • Last decade: 10 variables  1,000,000 variables
  • Healthcare crisis
    • Epidemic of Alzheimer’s Disease
    • Deinstitutionalization of the cognitively disabled
    • Nationwide shortage of caretaking professionals
an opportunity
...An Opportunity
  • Develop technology to
    • Support independent living by people with cognitive disabilities
      • At home
      • At work
      • Throughout the community
    • Improve health care
      • Long term monitoring of activities of daily living (ADL’s)
      • Intervention before a health crisis
the university of washington assisted cognition project
The University of Washington Assisted Cognition Project
  • Synthesis of work in
    • Ubiquitous computing
    • Artificial intelligence
    • Human-computer interaction
    • Support use of public transit
  • CARE
    • ADL monitoring and assistance
this talk
This Talk
  • Building models of everyday plans and goals
    • From sensor data
    • By mining textual description
    • By engineering commonsense knowledge
  • Tracking and predicting a user’s behavior
    • Noisy and incomplete sensor data
  • Recognizing user errors
    • First steps toward proactive assistive technology

ACCESSAssisted Cognition in Community, Employment, & Support SettingsSupported by The National Institute on Disability & Rehabilitation Research (NIDDR)The National Science Foundation (NSF)

Learning & Reasoning About Transportation Routines

  • Given a data stream from a wearable GPS unit...
    • Infer the user’s location and mode of transportation (foot, car, bus, bike, ...)
    • Predict where user will go
    • Detect novel behavior
      • User errors?
      • Opportunities for learning?
why inference is not trivial
Why Inference Is Not Trivial
  • People don’t have wheels
    • Systematic GPS error
  • We are not in the woods
    • Dead and semi-dead zones
    • Lots of multi-path propagation
    • Inside of vehicles
    • Inside of buildings
  • Not just location tracking
    • Mode, Prediction, Novelty
gps receivers we used
GPS Receivers We Used

GeoStats wearable GPS logger

Nokia 6600 Java Cell Phone with Bluetooth GPS unit

geographic information systems
Geographic Information Systems

Street map

Data source: Census 2000

Tiger/line data

Bus routes and bus stops

Data source: Metro GIS


Learning Engine

  • Goals
  • Paths
  • Modes
  • Errors



Inference Engine

probabilistic reasoning
Probabilistic Reasoning
  • Graphical model:

Dynamic Bayesian network

  • Inference engine:

Rao-Blackwellised particle filters

  • Learning engine:

Expectation-Maximization (EM) algorithm

graphical model version 1
Graphical Model (Version 1)
  • Transportation Mode
  • Velocity
  • Location
    • Block
    • Position along block
    • At bus stop, parking lot, ...?
  • GPS Offset Error
  • GPS signal
rao blackwellised particle filtering
Rao-Blackwellised Particle Filtering
  • Inference: estimate current state distribution given all past readings
  • Particle filtering
    • Evolve approximation to state distribution using samples (particles)
    • Supports multi-modal distributions
    • Supports discrete variables (e.g.: mode)
  • Rao-Blackwellisation
    • Each particle includes a Kalman filter to represent distribution over positions
    • Improved accuracy with fewer particles

blue = foot

green = bus

red = car

  • User model = DBN parameters
    • Transitions between blocks
    • Transitions between modes
  • Learning: Monte-Carlo EM
    • Unlabeled data
    • 30 days of one user, logged at 2 second intervals (when outdoors)
    • 3-fold cross validation
prediction accuracy
Prediction Accuracy

How can we improve predictive power?

Probability of correctly predicting the future

City Blocks

transportation routines
Transportation Routines




  • Goals
    • work, home, friends, restaurant, doctor’s, ...
  • Trip segments
    • Home to Bus stop A on Foot
    • Bus stop A to Bus stop B on Bus
    • Bus stop B to workplace on Foot

“Learning & Inferring Transportation Routines”, Lin Liao, Dieter Fox, & Henry Kautz, AAAI-2004 Best Paper Award

hierarchical model











Hierarchical Model


Trip segment

Transportation mode

x=<Location, Velocity>

GPS reading

hierarchical learning
Hierarchical Learning
  • Learn flat model
  • Infer goals
    • Locations where user is often motionless
  • Infer trip segment begin / end points
    • Locations with high mode transition probability
  • Infer trips segments
    • High-probability single-mode block transition sequences between segment begin / end points
  • Perform hierarchical EM learning
inferring trip segments
Inferring Trip Segments

Going to work

Going home

novelty error detection
Novelty & Error Detection
  • Approach: model-selection
  • Run several trackers in parallel
    • Tracker 1: learned hierarchical model
    • Tracker 2: untrained flat model
    • Tracker 3: learned model with clamped final goal
    • Estimate the likelihood of each tracker given the observations

Detect User Errors

Untrained Trained Instantiated

application opportunity knocks
Application:Opportunity Knocks

Demonstration (by Don Patterson) at AAHA Future of Aging Services, Washington, DC, March, 2004

care cognitive assistance in real world environments supported by the intel research council

CARECognitive Assistance in Real-world Environmentssupported by the Intel Research Council

Learning & Inferring Activities of Daily Living

research hypothesis
Research Hypothesis
  • Observation: activities of daily living involve the manipulation of many physical objects
    • Cooking, cleaning, eating, personal hygiene, exercise, hobbies, ...
  • Hypothesis: can recognize activities from a time-sequence of object “touches”
    • Such models are robust and easily learned or engineered
sensing object manipulation
Sensing Object Manipulation
  • RFID: Radio-frequency ID tags
  • Small
  • Semi-passive
  • Durable
  • Cheap
how can we sense them
How Can We Sense Them?

coming... wall-mounted “sparkle reader”

building models
Building Models
  • Core ADL’s amenable to classic knowledge engineering
  • Open-ended, fine-grained models: infer from natural language texts?
    • Perkowitz et al., “Mining Models of Human Activities from the Web”, WWW-2004
experimental setup
Experimental Setup
  • Hand-built library of 14 ADL’s
  • 17 test subjects
  • Each asked to perform 12 of the ADL’s
  • Data not segmented
  • No training on individual test subjects

General Solution

Quantitative Results


Point Solution

Quantitative Results

General Solution

Anecdotal Results

Point Solution

Anecdotal Results

Pervasive Computing, Oct-Dec 2004

current directions
Current Directions
  • Affective & physiological state
    • agitated, calm, attentive, ...
    • hungry, tired, dizzy, ...
  • Interactions between people
    • Human Social Dynamics
  • Principled human-computer interaction
    • Decision-theoretic control of interventions
why now
Why Now?
  • A goal of much work of AI in the 1970’s was to create programs that could understand the narrative of ordinary human experience
  • This area pretty much disappeared
    • Missing probabilistic tools
    • Systems not able to experience world
    • Lacked focus – “understand” to what end?
  • Today: tools, grounding, motivation
challenge to nanotechnology community
Challenge to Nanotechnology Community
  • Current sensors detect physical or physiological state: user mental state must be indirectly inferred
  • To what can extend can nanotechnology afford direct access to a person’s emotions and intentions?