Collaborative Annotation, Archival and Visualization in a Biofeedback Rehabilitation system

1. Collaborative Annotation, Archival and Visualization in a Biofeedback Rehabilitation system Hari Sundaram Arts Media and Engineering Arizona State University

2. Introduction • Motivation: • Every 45 seconds, someone in the United States suffers a stroke. It results in functional deficits of neuropsychological and physical functions in post-stroke survivors. • Up to 85% of patients have a sensorimotor deficit in the arm, such as muscle weakness, abnormal muscle tone, abnormal movement synergies, and lack of coordination during voluntary movement • Goal • Design a real time multimodal biofeedback system for stroke patient rehabilitation. • Archival / annotation and information visualization to provide insight. Memex Seminar, June 4, 20142

3. The Biofeedback system [To appear in acm mm 2006]

4. System Overview • The Biofeedback system system situates participants in a multi-sensory engaging environment, where physical actions of the right arm are closely coupled with digital feedback. • The Biofeedback system integrates five computational subsystems. • Motion capture • Motion analysis • Audio feedback • Visual feedback • Database for archival and annotation • All five subsystems are synchronized with respect to a universal time clock. Memex Seminar, June 4, 20144

5. Action Analysis • Arm Representation • 11 labeled markers on arm and torso • 3 labeled markers on the back of chair • Feature Extraction • 3D hand trajectory / 3D hand trajectory relative to the predefined straight line • Shoulder / Elbow extension • Hand Orientation • Shoulder rotation / abduction/elevation • Trunk flexion / rotation / lean and shoulder trajectory • Wrist extension • Multi-goal Framework • Reaching • Opening • Flow Memex Seminar, June 4, 20145

6. Coupling Action to Feedback • Engagement • Aesthetically attractive, easy to use and intuitive. • Message and Mapping • Reaching - visual target, an image completion/reassembly task, and an accompanying musical progression. • Flow - pointalistic sound clouds in the main musical line, flowing particles in the visuals • Opening - a rich, resonant musical accompaniment. • Environment • Introduction ( visual ) • Abstract I (visual+audio) • Abstract II (visual+audio), more variation Memex Seminar, June 4, 20146

7. Audio Feedback • Dynamic mapping of the normalized distance to target along the z coordinate to harmonic progression. • Map the hand trajectory velocity in the z direction to event density. • Joint Synchrony and Harmonic Progression. • Shoulder - woodwind sounds (flute, clarinet, bassoon) through the progression • Elbow - string sounds (a violin section of tremolo, a violin section, and a pizzicato violincello section). • Mapping of Shoulder and Elbow Extensions • Midi velocity (Mv) • Duration (td) • The probability of an octave doubling (Pd) Memex Seminar, June 4, 20147

8. Visual Feedback Transition Environment 3D virtual environment Physical movement will control the virtual environment. Abstract Environment A picture in a frame Explosion Turbulence Horizontal and Vertical Pull Memex Seminar, June 4, 20148

9. Validation • Offline Segmentation • Reaction • Reaching • Grasping • Returning • Spatial Error • Target-Hand Distance • Hand Orientation • Arm Openness • Should Openness • Elbow Openness • Reaching Duration • Flow Error • Zero crossing number • Polynomial curve fitting error • Consistency Memex Seminar, June 4, 20149

10. Results Reaching - our visual-audio feedback design can guide the normal subject to do the reaching as accurately as they did in real world. • Flow - the smooth of speed curve means three things: • Subjects are clearer the goal and they need not hesitant what will happen. • Subjects are clearer about the feedback cue. Based on the current feedback and their memory, they can easily find the way to reach the target. • Subjects start following the rhythm, that is mapped in the audio feedback. • Openness - our audio feedback design for the abstract environment can help subjects with more openness. Memex Seminar, June 4, 201410

11. Archival Sub-system

12. Overview • Challenges: • Continuous data streams and large datasets • Real-time annotation has high cognitive load • We are integrating an archival subsystem into a team with different domain experts. • Our Approach: • Continuous multimodal archival • Real-time collaborative annotation • Offline information visualization Memex Seminar, June 4, 201412

13. Archival Subsystem Design • Part of our overall Biofeedback system • Manage multimodal data streams • Different data transport rates (total: 1.89MBps) • Scalable Multicast Network • raises synchronization problem Memex Seminar, June 4, 201413

14. Continuous Multimodal Archival • We split computational and storage resources into two archival subsystems • Archiving parametric system models • Raw motion capture data • Motion analysis parameters • Audio-visual synthesis parameters • Data was multicast • Contextual media capture • Seven channels • Actual audio-visual feedback data • Three microphones • Video camera • Hardware: soundboard, microphones, VGA monitor scan-converter, video camera, mpeg hardware encoder, due-core server max / msp graphical program Memex Seminar, June 4, 201414

15. Database design • Indexing Scheme • The patient / session / set / trial hierarchy • Universal time stamp of synchronized subsystems • Structural DB tables • Motion capture and analysis parameters categorization • Group audio-visual data by feedback semantics • We first stream parametric data into a multi-buffered queue, then write to DB using bulk insert in parallel. • We keep reference of multimedia data. • Privacy issue Memex Seminar, June 4, 201415

16. Real-Time Collaborative Annotation • Why emphasize “real-time collaborative”? • Annotations are time critical • Each trial is short (~5 sec.) – • there can be many unexpected events in this period – can cause cognitive overload. • Team is focused on the experiment! • Design goals of the annotation tool • Distributive • Personalized (Multi-disciplinary team) • Collaborative Memex Seminar, June 4, 201416

17. Annotation Interface • Design Elements • Dynamic experiment progression indicators • Domain specific checklist • Collaborative annotation sharing • We multicast annotations from one client to others • Random query, retrieval and modification • User feedback is very positive Memex Seminar, June 4, 201417

18. Information Visualization • Offline visualization for review / annotation of archived data. • Our design goals • Hierarchical and selectable motion parameters / evaluation metrics navigation • Synchronized contextual information playback • Facilitates annotation modification • Helps domain experts share information and improve their subsystems Memex Seminar, June 4, 201418

19. Visualization Prototype • Features: • Allows navigation through our trial hierarchy on motion analysis results • Contextual media playback with parametric motion analysis visualization • Provides offline annotation facilities Memex Seminar, June 4, 201419

20. Open Issues • Event Model • Event definition by domain experts • Event detection • Event Network Modeling • Pre-emptive Annotation • Show events with high priority • Event log • SenseCam Integration • SenseCam pictures can be integrated into our visualization framework Memex Seminar, June 4, 201420

21. thanks Team: Weiwei Xu, Yinpeng Chen, Richard Wallis, Thanassis Rikakis, Hari Sundaram, Todd Ingalls, Loren Olson, Jiping He, Sharon Liu