On the parameterization of clapping

1. On the parameterization of clapping Herwin van Welbergen Zsófia Ruttkay Human Media Interaction, University of Twente

2. Content • Context and goals • Related work • Experiment setup • Results • Conclusion • Questions

3. Context: Reactive Virtual Trainer • An ECA acting out exercises a user is supposed to do • Perceives the movement of the user • Reactive • Gives feedback • Using speech, gestures and motion • (Re)schedules and adapts exercises • Tempo changes • Subtle timing and lifelikeness of motion is important

4. Goal • The generation of believable, adaptable exercise motion in real time • How can we parameterize motion? • What parameters? • Tempo, amplitude (for accentuation?), … • How do the parameters relate? • How do they affect movement? • How is speech synchronized with exercise motion?

5. Related work: biomechanics • Typical biomechanical research setup • Very obtrusive • Measuring one movement characteristic • Gaining ‘deep’ knowledge • Our setup • Unobtrusive • Measuring a wider range of characteristics • Less depth, measure on an abstraction level that gains us parameters for movement generation

6. Related work: finding animation parameters • Statistical methods (Egges et al), machine learning (Brand et al) • Finds independed parameters • Not intuitive • Highly depended on analyzed data set • Laban Movement Analysis • Effort can automatically be found from movement data (Zhao et al) • Shape? • Our parameters (tempo, amplitude) can be mapped to LMA parameters

7. Related work: parameterized animation • Rule based (EMOTE, Neff et al, Hartmann et al, ..) • Uses movement models • Typically does not deal with dependence between parameters • Lack of detail • Example based (Wiley et al, Kovar et al) • By blending examples • Nr of examples needed grows exponentially with nr of parameters

8. Related work: model based gesture synthesis • Kopp et al: Uses biomechanical rules of thumb to generate movement • Real time • Domain: speech accompanying gestures • We plan to extend on this work • Use in rhythmic movement domain • Providing parameterization • Providing whole body movement • Introducing movement variability

9. Focus • Analysis of a clapping exercise • Analyzed aspects: • Synchronization of speech and motion • How does a change of tempo affect movement? • Time distribution • Movement path • Amplitude • Left-right hand symmetry • Whole body involvement

10. Clapping experiment: setup • Mocap analysis of two subjects • Instructions: • ‘Free clap’: • Clap and count from 21 to 31 • ‘Metronome driven clap’: • Clap and count to the metronome

11. Synchronization of clap and speech • Phases from movement in gestures • The phonological synchrony rule holds for clapping

12. Time distribution in phases • Free clap was executed consistently at ≈ 60 bpm • One subject made use of a pre-stroke hold at 30 bpm • The relative duration of the phases does not change with tempo • The standard deviation of the relative duration decreased with one subject

13. Movement path of the hands

14. Amplitude: how to measure • Maximum distance between hands • Path is curved • Max distance between hands alone does not display the amount of motion • Distance along path

15. Amplitude: observations • Path distance and max. hand distance decrease with tempo • Average speed is constant at different tempos • There is a linear relation between period and path distance • Pathdistance = a + b ▪ period • Amplitude of free clap is higher • Average speed of free clap is higher

16. Amplitude

17. Period vs path length

18. Left-right hand symmetry:How to measure • Model: self oscillating systems • Closed orbit between position (x) and speed (v) • x is the normalized angle x^ • θ is the phase angle • Relative phase angle: • Φ = θleft-θright • Negative Φ means right hand leads

19. Left-right hand symmetry at 90 bpm

20. Left-right hand symmetry:Theory • Right handed subjects lead a rhythmic task with their right hand (Treffner et al) • But such asymmetry can disappear when the task is metronome driven • Stability of Φ depends on the tempo and mass imbalance (Treffner et al, Fitzpatrick et al) • Higher tempo => higher |Φ| • Higher tempo => higher variability in Φ

21. Left-right hand symmetry:Findings • Mean Φ is consistently negative for our right-handed subjects • No difference between metronome driven and free clap in mean Φ • The standard deviation of Φ increases with tempo • No significant relation between mean Φ and tempo was found

22. Whole body involvement • By annotating if markers move in the same tempo as the hands • Movement was found on the head and torso for all tempos • For low tempos movement was even observed up to the thighs and knees

23. Conclusions • The phonological synchrony rule was validated for clapping • Clapping can be sped up by making the path distance smaller • A pre-stroke hold can be used to slow down • Clapping is clearly a whole body motion • At a faster tempo, fewer body parts are perceivably involved • Left-right hand movement variability increases with tempo • For both right-handed subjects, the right hand was leading • The metronome did not diminish this lead

24. Further work • Ultimately: generate clapping motion given tempo + personal characteristics • More recordings • Free clapping without counting • Tempo transitions • How do personal characteristics affect movement? • Deeper analysis • How does variability affect the movement path? • Generation • Can movement on the rest of the body be generated given movement on the arms (as in Egges, Pullen)? • Blending clap animation at different tempos to gain animation at a new tempo (as in Kovar)?

25. Questions

26. Easter eggs

27. Why use gesture-like phases for clapping? • The stroke of a speech accompanying gestures (SAG) is at an energy peak in the movement and expresses meaning (McNeill) • Claps have such a clear peak • But this peak does not express meaning • Why compare SAG and clapping? • The form of clap movement and SAG is similar • Excursions: start in rest, end in rest • Peak structure • Well bounded • But not symmetric • May find information on the nature of the phonological synchrony rule • Does it depend on form or meaning?

28. Precision • Precision • No significant correlation between metronome period and avg clap ‘error’ or variability of clap tempo was found • Measured both absolute and relative to the metronome period

29. Left-right hand position at 90 bpm

30. 3D hand & elbow positions

31. 3D hand positions at different tempos

32. Free clap amplitude vs metronome drive clap amplitude