A Framework of Energy Efficient Mobile Sensing for Automatic User State Recognition

1. A Framework of Energy Efficient Mobile Sensing for Automatic User State Recognition Y. Wang, et al. Dept. of Electrical Engineering, University of Southern California

2. Background • Background • Moore's law : doubling the number of transistors in unit area every 18 months • Integrating complex sensing capabilities on mobile devices • Wi-Fi, Bluetooth, GPS, audio, video, light sensors, accelerometers and so on • Mobile phone : a powerful environmental sensing unit • Monitoring a user's ambient context, both unobtrusively and in real time • Diverse applications for user state • Mobile cooperative services • Real time traffic monitoring • Social networking applications • Facebook and MySpace • User state recognition • AI and data mining techniques • Learning interactions between user's behavior and the environment • A combination of diverse features • Motion, location and background condition • Automatically adjusting ring tone profile to appropriate volume

3. Problems and Proposal • Limited battery capacity of mobile devices • A big hurdle for context detection • Embedded sensors → major sources of power consumption • Fully charged battery on Nokia N95 mobile phone • Only telephone conversation : 10 hours • GPS receiver is turned on : 6 hours • Energy efficient mobile sensing system (EEMSS) • Hierarchical sensor management scheme for power management • Automatic user state recognition • Motion (such as running and walking), location (such as staying at home or on a freeway) and background environment (such as sound and crowd level) • User states and state transition rules definition by an XML • Three sensor management scheme • State descriptor to support flexibility for application requirements • Minimum set of sensors assignment to achieve energy efficiency • Invoking new sensors when state transitions happen

4. Related Works (1/2) • Multi-sensor mobile applications and services • Combining a diverse set of sensors (Gellersenet al. [5]) • Monitoring and recognizing human activities with motion sensors • Car manufacturing (Stiefmeieret al. [6]) • Supporting a production or maintenance worker by recognizing the worker's actions • Delivering just-in-time information about activities to be performed • Development of a jacket with various sensors • CenceMe [13] • Sharing “buddies” presence with social networks in a secure manner • Using sensors to capture the users' status • Activity, disposition, habits and surroundings • CenceMe prototype implementation on Facebook [14] • Sensay [12] • Context-aware mobile phone using data from a number of sources • Dynamic change of cell phone ring tone, alert type depending on user states

5. Related Works (2/2) • Using a large number of sensors from different fields • Activity recognition and health monitoring [13, 14, 11, 17, 18, 19, 20] • GPS, Bluetooth, WiFi detector, oxygen sensor, accelerometer, electrocardio-graph sensor, temperature sensor, light sensor, microphone, camera, and so on • Providing context information for higher layer applications • Human status tracking, social networking, and location based services • Studies for power management on mobile devices • Survey of methods for saving power on hand-held devices(Viredazet al. [21]) • Dynamic frequency/voltage scaling [22] to reduce power consumption • Suitable for lower-level system design rather than application development • Event driven power-saving method to reduce extra power consumption • Reducing the idle power (Shih et. al. [23])in a “standby” mode • Power on only when there is an incoming or outgoing call • Turdecken [24] • Event-driven, a hierarchical power management system

6. Sensor Management Methodology • Energy efficient mobile sensing • Managing sensors in a hierarchical way based on the user's current state • User's real-time condition • Motion (such as running and walking) • Location (such as staying at home or on a freeway) • Background environment (such as sound and crowd level) • Sensor management • Core component of EEMSS • Including user state recognition • Ex) “meeting in office” • Existence of speech :yes • Current location : office area • Sensor assignment depending user state • Specifying an XML-format state descriptor • Sensor management rules for each state • Control of sensors based on real-time system feedback

7. Sensor Management Using XML • General format of a state descriptor • User state definition between “<State>”and “</State>” tags • Sensor speciation by “<Sensor>” tags • State transition criteria satisfaction → a new state “<NextState>” • Ex) If the user is at “state2" and “sensor2" returns “sensor reading 2" which is not sufficient for state transition, “sensor3"will be turned on immediately to further detect the user's status in order to identify state transition • Three major advantages of xml as state descriptor • (1) representation of hierarchical relationship among sensor in a clear manner • (2) easy modification by someone with limited programming experience • (3) easy parsing by modern programming languages (Java and Python) • Current implementation of EEMSS design • Manually configured sensor management rules • Determination of XML state descriptor and sensor sampling intervals and duty cycles

8. EEMSS: The Overview • EEMSS (energy efficient mobile sensing system) • Implementation on Nokia N95 devices • Using N95 built-in sensors • GPS, WiFi detector, accelerometer, and embedded microphone • Goals of study • Prototype implementation • Evaluation of state recognition accuracy, detection latency, and energy efficiency • Set of states

9. EEMSS: Hierarchical Decision Rules • Example of hierarchical decision rules for walking • Periodically sampling GPS when the user is walking • Significant amount of speed increase : riding a vehicle • Wireless access point sets : one's frequently visited places (home, cafeteria, office)

10. EEMSS: Architecture and Implementation (1/2) • Main components of EEMSS • Sensor management module • Classification module • Sensor control interface • Turning sensors on and off • Collecting sensed data • Components for debugging and evaluation • Real-time user state updates • Logging, and user interfaces • Current version of J2ME • No direct access to some of the sensors • Python program • Gathering and sharing sensor data

11. EEMSS: Architecture and Implementation (2/2) • Sensor management module • Major control unit of the system • Parsing a state description file • Sensor management scheme • Control of sensors based on user state and state transition conditions • Classification module • Consumer of the sensor raw data • Processing the raw sensing data into desired format • Magnitude of 3-axis accelerometer sensing data • FFT on sound clips to conduct frequency domain signal analysis • Returning user activity and position feature • “Moving fast”, “walking”, “home wireless access point detected” and “loud environment” • Forwarding user states to the sensor management module • Sensor interface APIs • Support of sensor readings and instruct sensors to switch on/off • GPS and embedded microphone through J2ME APIs • Accelerometer and Wi-Fi detector through Python APIs.

12. Energy Consumption Measurement (1/2) • Two class of sensors on a mobile phone • First class : accelerometer and microphone • Continuous operation and requirement of an explicit signal to be turned off • Need for activation for a period of time to obtain meaningful sensing data • Second class : GPS, Wi-Fi detector, and Bluetooth scanner • Gathering instantaneous samples • Automatically turn off when the sampling interval is over • Energy cost of sensors • Instant power drain + operating duration • Ex) (API and hardware limitations) GPS on Nokia N95s • Requirement of a certain amount of time to synchronize with satellites • Remaining active for about 30 seconds after a location query • Assisted-GPS : satellite synchronization time to less than 10 seconds • Wi-Fi scan : less than 2 seconds to finish • Bluetooth scan : around 10 seconds to complete

13. Energy Consumption Measurement (2/2) • Measurement of sensor energy consumptions • Using Nokia Energy Profiler [26] • A stand-alone application to test and monitor application energy usage in real time • Measurement results • Accelerometer : the least amount of power • Capturing the change of body movement • Indicator of state transition with high probability • GPS : the large power drain and long initialization time • Location tracking and mode of travel classification

14. Sensor Duty Cycle and Computation Time • Sensor management scheme • Minimum set of sensors at any specific time • Assigning an appropriate duty cycle to each sensor • Periodic sensing and sleeping instead of being sampled continuously • Reducing the energy consumption • Duty cycles in EEMSS • Saving energy cost by reducing sensing intervals • Too sampling period : insufficient representation of the real condition • Longer sensing period : increase of the robustness of state recognition with more energy • Longer sleep interval : reducing power battery consumption, increase of detection latency • Two reasons for longer duty cycles to the microphone versus the accelerometer • Accelerometer : less power, and more frequent sampling • Accelerometer : capturing user motion change with less detection delay compared to identifying background sound type

15. Sensor Duty Cycle and Computation Time • Periodically querying GPS • Providing outdoor location and speed information • 5 minutes, a relatively long duration for the GPS to lock satellite signal • Wi-Fi scanning • Using event-based approach • Two scenarios for Wi-Fi scan • (1) when the user is detected as moving, a Wi-Fi scan is conducted to check if the user has left his or her recent range • (2) when the user has arrived at a new place, we compare the nearby wireless access points set with known ones in order to identify the user's current location • Duty cycle parameters through extensive empirical tests • No optimization or dynamic adjustment

16. GPS Sensing and Mode of Travel • Real-time location tracking • Detecting the user's basic mode of travel • Using Geo-coordinates and the moving speed of the user • Combining velocity information and distance of travel • Distinguishing one's basic mode of travel such as walking or riding a vehicle • Classification of mode of travel • Checking the recent moving distance and speed • Training a classifier by several location tracking records of user • Using certain threshold values • Occurrence of location request timeout • Indication of entering a building or other indoor environment • Indication of locations in which satellite signals are not reachable

17. Wi-Fi Scanning and Usage • Performing a Wi-Fi scan • Returning MAC address of visible wireless access points around the user • Tagging a particular location by the set of access points visible • Automatic identification of current location by checking nearby access points • EEMSS implementation • Ex) User is at home if the Wi-Fi scan result matches a user’s home • Memorization of wireless access points feature of the user's home and office • Monitoring a user's moving range with Wi-Fi scan • Covering an area of radius 20-30m using wireless access point • Classification if the user has moving out of that range • Wi-Fi scan for GPS sampling start • Sampling location information immediately after Wi-Fi scan • When out of recent range by Wi-Fi scanning

18. Real-time Motion Classification • Accelerometer readings from mobile phone • Placement at various locations due to individual habit • Extremely difficult to perform fully detailed motion classification [10] • Accelerometer data collection • 53 different experiments distributed in two weeks • Lengths of experiment : several minutes to hours • Tagging ground truth of activity for analysis and comparison purposes • Standard deviation threshold values • Stable, walking, running, and vehicle mode • No explicit requirement of where the phone should be placed • Computation per 6 seconds

19. Real-time Motion Classification • Experimental results • 26 experiments containing a combination of different user motions • Very good for extreme conditions such as stable and running • Confusing with walking and vehicle mode due to feature overlap • above 70% accuracy • Using acceleration motion as a trigger for GPS or Wi-Fi detector

20. Real-time Background Sound Recognition • Two steps for sound classification • Measuring the energy level of the audio signal • Classification whether environment is silent or loud • x(n) : time domain signal • Loud environment → speech recognition with time and frequency domains • Speech → higher silence ratio (SR) [28] • SR : Ratio between the amount of silent time and the total amount of the audio data • No speech → “loud” or “noisy” • No further classification algorithm to distinguish music, noise, etc.

21. Real-time Background Sound Recognition • Fast Fourier Transform • Implementation on the mobile device • Frequency domain features of four types of audio clips • Male's speech, a female's speech, a noise clip and a music clip • Speech signals : more weight on low frequency spectrum from 300Hz to 600Hz • SSCH (Subband Spectral Centroid Histogram) algorithm [29] • Speech detection • Comparison with speech peak frequency thresholds (300Hz - 600Hz)

22. Real-time Background Sound Recognition • Sound data set • 1085 speech clips, 86 music clips and 336 noise clips • 4 seconds length for each clip • Classification accuracy with different SR thresholds • Increase of SR threshold → decrease of false positives • Decrease of speech detection accuracy • SR = 0.7 : more than 90% of detection accuracy and less than 20% false positive

23. Evaluation Methods • Evaluation of EEMSS • State recognition accuracy • State transition detection latency • Energy efficiency • Phase I - Lab Study • Lab study over 1.5 months with our team members • Calibration of classification parameters • Parameters used to determine mode of travel based on GPS readings • Duty cycles for different sensors • e.g. the sampling frequency of accelerometer, GPS and microphone • Duty cycles for system energy consumption measurement • Phase II - User Trial • User trial in November 2008 at two different universities • Test of EEMSS system in a real setting • Recruit of 10 users • Undergraduate, graduate students, faculties and family members

24. Evaluation Methods • Phase II - User Trial • Introduction about basic operation of mobile device • Manual recording diary for two days per user • A standardized booklet (three simple questions) • Motion (e.g.: walking, in vehicle, etc), location, sound (e.g.: quiet, loud, speech, etc) • 260 running hours of EEMSS application • More than 300 state transitions

25. Results • State Recognition Records • Tracking the user's location by recording geo-coordinates of the user • Location information per 20 seconds • Daily traces captured by EEMSS • Two different participants from CMU and USC on two campus maps • Different modes of travel • Dashed ones : vehicle mode • Solid ones : walking mode • Comparison with ground truth diaries • Dashed curves : bus routes • Solid curves : a user’s traces between home and bus station

26. Results • State Recognition Records • Probing background sound • Quiet, loud and containing speech • User activity inference at some places • Working, meeting, resting, etc. • Combining detected background condition with location obtained by Wi-Fi scan • State Recognition Accuracy • Comparing the system recognized state log with the ground truth records • Ratio of correctly recognized records over total records • Average recognition accuracy over all users : 92.56% • Standard deviation of 2.53%

27. Results • State Recognition Accuracy • Three super states for recognition accuracy • “Walking”, “Vehicle” and “At some place” by location and mode of travel • Percentage of recognition accuracy • Column : ground truth • Row : recognized states of EEMSS. • Staying at some place : very high accuracy • Home, office, etc. • False positives (12.64% of walking time and 10.59% of vehicle time) • Regular slow motions of vehicles → walking • State Transition Detection Latency

28. Results • State Transition Detection Latency • Vehicle mode • Quick detection with only one or two GPS queries • At some place mode • Less than 5 minutes for GPS location request timeout, and Wi-Fi scan • Detecting the transition from riding a vehicle to walking • Longer time to distinguish slow motion of vehicle and walking • Sampling acceleration data per 6 seconds • Background sound change detection : less than 3 minutes (duty cycle) • Device Lifetime Test • Lab studies : two researchers for 12 days • Average device lifetime with EEMSS : 11.33 hours

29. Results • Device Lifetime Test • Turning on GPS, accelerometer and microphone with same sampling frequency • WiFi scanning per 5 minutes • Less than 5 hours regardless of user activity • Power Usage

30. CONCLUSIONS AND FUTUREWORKDIRECTIONS • Mobile device based sensing platform • Rich contextual information about users • Environment for higher layer applications • Considering limited battery capacities • EEMSS (sensor management scheme for mobile devices) • Selectively turning on minimum set of sensors to monitor user state • Triggering new set of sensors to achieve state transition detection • Shutting down unnecessary sensors at any particular time • Implementation on Nokia N95 devices • Evaluation with 10 users from two universities • Future work • More sophisticated algorithms to dynamically assign sensor duty cycles • Implementing sensor management scheme on more complex sensing applications