A cyber-physical system for senior collapse detection

1. A cyber-physical system for senior collapse detection Lynne Grewe, Steven Magaña-Zook CSUEB, lynne.grewe@csueastbay.edu

2. Seniors Falling • Over 1/3rd of seniors above 65 fall each year • Lead to serious injury and even death • Falls account for 25% of all hospital admissions, and 40% of all nursing home admissions 40% of those admitted do not return to independent living; 25% die within a year. • Fast medical attention can make a difference • Many falls do not result in injuries, yet a large percentage of non-injured fallers (47%) cannot get up without assistance.

3. Cost of Falling? • 2005, CDC study – Cost for Falls leading to fatality

4. Goal • create a “smart home” system to predict and detect the falling of senior/geriatric participants in home environments • More seniors living at home autonomously

5. SCD: Senior Collapse Detection Overview

6. SCD: uses Kinect Sensor • Inexpensive, commercial, well tested, good API support

7. Feature Extraction • Perform Skeleton Tracking • Ideal – fall indicators often involve joint locations and range of motion • Good Resolution – 21 joints

8. Skeleton Tracking Has Noise • Degrading performance with occlusion • General Twitching • Also degrades as more occlusion from being on floor << not bad << notice rear leg position problem fromself occlusion

9. Noise Reduction: Physical Therapy Skeleton Model • Use Physical Therapy Model data to determine normal range of motion and joint distances. • Calculate joint certainty metric = f(joint angles, joint distances, physical therapy skeleton model) • = 1 if within limits of model • <1 non-linear function of deviation from model • Currently use 1 model based on maximum ranges • Future = model for different demographics (age, height, weight), or learned from user. • Concept = Can use Joint Reliability to determine if a joint should be used in Fall Detection OR can use in determination of confidence of a Fall detected

10. What is a Fall? How can we detect it? • SCD defines fall as “loss of control resulting in downward motion ending with body on floor” • Previous work: • Wearable devices: • Accelerometers, gyroscopes, movement sensors • Autonomous: • 2D with mixed results • 3D beginning work • Detection Ideas • Quick movement (acceleration) – whole or what part of body? • Body Orientation – parallel to floor • Location – little but, some looking at general location

11. SCD Fall Detectors • Currently 3 based on all ideas (location, orientation and acceleration). • Currently operate independently – any can trigger fall detection event

12. Location –need Floor Detection • Uses 3D floor plane detected by Kinect Sensor • One for each skeleton calculated • Good News- Gamers want this accurate Ax + By + Cz + D = 0

13. SCD: Head Movement Detector • Falling Detector / Idea: quick movement indicates falling • Measure: both head joint velocity and trajectory (downward) and the head ends up near the floor. • Buffer 1 second of data (30 frames / second) • Trajectory – 2 slopes • Empirically chosen Thresholds • velocity>1ft/second • Last frame of 1 second head position within 1.5 ft of floor • Trajectory toward floor

14. SCD: Head Movement Detector – Reliability and Confidence • Reliability: function (number tracked joints, number inferred joints) • Confidence: function (velocity)

15. SCD: Horizontal Ratio Detector • Fall Detector / Idea: senior lands on the floor in horizontal-parallel to floor orientation • Concept = 3D bounding box • 2 Ratios = Width/Height and Depth/Height • Empirically chosen Threshold: • 1.5 for either Ratio = elongated, parallel to floor Head Height Ratio FALL

16. SCD: On Floor Detector • Fall Detector / Idea: senior lands on the floor • Hip near floor • Minimum number of joints near floor • Empirically Chosen Thresholds • Minimum 1 hip joint (out of 3 possible) • Minimum 8 joints “near” floor • “near” = 1.5 ft • Reliability = #tracked / (#tracked + #inferred) = 0.25 threshold

17. How Many Falls? • Some of our detectors are “Fallen” detectors • Don’t want too many triggers for same fall • Minimum time between fall events is set currently at 15 seconds. • No data but, seemed fastest time between different falls • Example: http://www.youtube.com/watch?v=Tm_fsp5puVk

18. Emergency Response • Configure Emergency Contact(s) • Email • Phone – sms text

19. SCD: Speech Processing • Use Microsoft SDK Text-To-Speech • Use Microsoft SDK Speech Recognition • Kinect has microphone array.

20. Fall Detection Event and Emergency Response System • Senior Hears Audio Prompts from System –asking if assistance is needed. If Yes or No Response the predetermined emergency response is triggered • Here you see both the Diagnostics GUI and an illustration of the final Audio

21. Examining Test case • Head Motion Detector: FALL • Trajectory = slope average was -1.258 • Head Position Last Frame = 1.37ft from floor • Velocity = 1.003ft/sec • On Floor Detector: FALL • 9 joints near floor • All 3 hip joints on floor • Horizontal Ratio Detector: FALL • W/H = 1.7, D/H = 0.89 • Head Distance to Floor = 1.37ft from floor

22. Both Live and Semi-Automated Testing • Have ability to cycle through sets of pre-recorded data • Output to HTML results

23. SCD: RESULTS • OnFloor Performs best 100%

24. Issues, Problems and Future work

25. Limitations with Kinect • Limited depth range(solution: multiple Kinect) • Occlusion (solution: multiple Kinect or use tilt feature of Kinect)

26. Issues • Skeleton engine needs some number of frames to recognize when user enters frame. This is unavoidable with current concept of skeleton tracking • Processing – on common commercial home use laptops and desktops ($400-700) we experience a lag time when all diagnostics are being displayed from 1 to 20 seconds worst case to process frame leading to detection. Typical (little data) around 0.5-5 seconds.

27. Future Work • More Testing • Combine Decisions? Learn Formulation? • Fine Tune/ Learn Thresholds • Improve Performance Speeds • Other modules • Fall prediction = gait tracking • Post Fall detection = rolling, vocalizations • Learning Individual Physical Model • Multi-Kinect System • calibration, sensor inference, coordinated communication and decision making • Kinect 1 improvement in resolution.