{ }

1. { } THE RECOGNITIONOF BODILY INTERACTIONS IN DIGITAL ARTIFACTS: Toward Inter-Subjective Grounds Capstone Project, Fall 2009 Kim, HyewonHuman Computer Interaction/ design School of Informatics Indiana University Bloomington

2. CONTENTS: HUMAN BODIES AS THE FOUNDATION OF INTER-SUBJECTIVE GROUNDS: Philosophical Theories of the Human Body The Paradox of Formulating Bodily Interaction THE FRAMEWORK OF BODILY INTERACTIONS: Sensory, Grounded Cognition, Socio-Cultural The Analytical Interpretation of Bodily Interaction IMPLICATIONS FOR DESIGNING:DISCUSSION and FURTHER research: Conclusion: THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

3. (e.g., the shift of interface from the physical button of telephones to the intangible one in smart phones)

4. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS Daniel Rozin’s wooden mirror, which comprises a camera and dozens of individual slats of wood on motors, “reflect” the image of the person standing in front of it.

5. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS Microsoft's Surface has an excellent attraction affordance to lure users into touching its surface: water.

6. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS The Sensitive Window detects the positions and walking directions of passersby to trigger the appropriate content, such as a full–screen movie created to surprise someone walking left to right.

8. To use enMotion’s paper Towel Dispenser, you hand in front of the sensor and a sheet of paper towel comes out of the dispenser. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

9. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS You can turn on an Air switch light by merely waving your hand slowly over the light to make the light dim or brighten.

10. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS Fukuda’s Automatic Door adjusts to the shape of the user as she approaches the door.

11. It isn’t hard to imagine large surfaces such as this Illuminate table becoming spaces for collaborative work, whether it is simply passing document around the table or working together in groups. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

13. BODILY PRACTICES: Bodily interactions have been unconsciously mastered. To some extent, the comprehensibility of bodily language is often taken for granted. More complex situations of design because of both the proliferation and the heterogeneity [37] of novel technology systems. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

14. INTER-SUBJECTIVE GROUNDS:Between users and digital artifacts as inter-subjective grounds which (re)construct the focal dimension of action-goal associations. • Speculation on this ground offers designers propositional thinking to identify bodily-capabilities of users that are communicatively rich and dynamic. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

15. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS What is Meant by Human Body ?

16. HUMAN BODIES AS THE FOUNDATION OF INTER-SUBJECTIVE GROUNDS: • Descartes’ disembodiment [36] created a conceptual division between me and you or the subject and the object. • Bergson’s body is central to and inseparable from how he experiences himself; he is embodied. • Edmund Husserl [34] asserted that the phenomenological status of each human is a reality shared with other human subjects. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

17. HUMAN BODIES AS THE FOUNDATION OF INTER-SUBJECTIVE GROUNDS: • Descartes’ disembodiment [36] created a conceptual division between me and you or the subject and the object, • Bergson’s body is central to and inseparable from how he experiences himself; he is embodied. • Edmund Husserl [34] asserted that the phenomenological status of each human is a reality shared with other human subjects. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

18. HUMAN BODIES AS THE FOUNDATION OF INTER-SUBJECTIVE GROUNDS: • Descartes’ disembodiment [36] created a conceptual division between me and you or the subject and the object, • Bergson’s body is central to and inseparable from how he experiences himself; he is embodied. • Edmund Husserl [34] asserted that the phenomenological status of each human is a reality shared with other human subjects. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

19. The Paradox of Formulating Bodily Interaction: It is complicated to encode the subjectivityof individual users into the inter-subjectivegrounds of digital artifacts. For example, some factors of interaction are labeled as things to be avoided (e.g., because of the connotations of bodily languages in a certain discourse), whereas other factors possibly generate proper target-states to encourage users’ engagement. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

20. The Paradox of Formulating Bodily Interaction: Yet, these paradoxical relations do not necessarily mean that designers should suggest only the bodily interactions that are conscious or rationalized. Presumably, such boundaries for designers delineate the Rhetorical [10] Zone in which to make plausible their judgments in terms of how users can efficiently decide their communication goals, or what interaction goals constitute action. If this mediated ground proves successful over time in satisfying the users’ needs, the functional purpose of digital artifacts is likely to validate the quality of interactions and designers’ judgments. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

21. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS FRAMEWORK:Sensory, Grounded Cognition, and Socio-Cultural levels

22. THE FRAMEWORK OF BODILY INTERACTIONS: Sensory Level: This primitive level refers to biological states of human senses. Human senses perceive the three-dimensional space of our ecological world for interacting with objects. Grounded Cognition Level: This level posits that human cognition is typically grounded in multiple ways such as simulations, situated actions, and, on occasion, bodily states. (e.g., the bodily movements of using a cup) Socio-Cultural Level:This level means that the syntax of a natural body has evolved in close interdependency with, and mutual adaptation to, the animate and inanimate environment. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

23. THE FRAMEWORK OF BODILY INTERACTIONS: Sensory Level: This primitive level refers to biological states of human senses. Human senses perceive the three-dimensional space of our ecological world for interacting with objects. Grounded Cognition Level: This level posits that human cognition is typically grounded in multiple ways such as simulations, situated actions, and, on occasion, bodily states. (e.g., the bodily movements of using a cup) Socio-Cultural Level:This level means that the syntax of a natural body has evolved in close interdependency with, and mutual adaptation to, the animate and inanimate environment. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

24. THE FRAMEWORK OF BODILY INTERACTIONS: Sensory Level: This primitive level refers to biological states of human senses. Human senses perceive the three-dimensional space of our ecological world for interacting with objects. Grounded Cognition Level: This level posits that human cognition is typically grounded in multiple ways such as simulations, situated actions, and, on occasion, bodily states. (e.g., the bodily movements of using a cup) Socio-Cultural Level:This level means that the syntax of a natural body has evolved in close interdependency with, and mutual adaptation to, the animate and inanimate environment. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

26. The Analytical Interpretation of Bodily Interaction: Playing the Ocarina THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS • The player activates a source-path-goal schema from the starting location to the destination. (e.g., playing music). • A player starts from simple biological determination of the interface environment with hands, mouse, and eyes. • The player recognizes the recurrent stable patterns of sensory-motor experience of the real artifact. (e.g., mimicking the grounded practices of a real ocarina) • The destination of bodily movements is mapped by the visual feedback or sounds. • The internal interaction structures of the interface give rise to constrained inferences that the interface guides. • The player thus bodily and mentally realizes the trajectory of his/her bodily movements. • The player preserves the image structure of the bodily perception. (e.g., the physical interaction of blowing the acoustic interface to pitch) • The player structures perceptions that link sensory-motor experiences. • The player predicts the interaction with a wider environment. • The player dynamically operates the bodily language in and across time.

27. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS How will the practices of bodily movements be encoded in new digital artifacts?

28. IMPLICATIONS FOR DESIGNING: Reproducing Well-Defined Trajectories in Familiar Products: Extending the Cultural Connotations of User Practices: Redistributing Communicative Power with Verbal Language: Considering Ethical Awareness and Technological Mediation: THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

29. Reproducing Well-Defined Trajectories in Familiar Products: Borgmann [4] calls this relation focal practices, centered around focal things, that “are instrumental and symbolic … to a social communication end.” The apparent motions clearly indicate a certain interaction, pervasive in the motions innate in the ecological world andenculturatedby the manipulation of common objects. Schön asserts that successful product designs result from the conversation that physical action and cognition are interconnected. [39] THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

30. Reproducing Well-Defined Trajectories in Familiar Products: Figure 2. Inamo, the restaurant on tangible surface table

31. Extending the Cultural Connotations of User Practices: Dittmar argues that a “socially shared meaning-system may be viewed as an organizing principle of identity as well as of the meanings of possession as self extensions … as embodiments of cultural meanings.” [12] Kendon’s study [24, 25] asserts that there are inter-individual similarities in the patterning of gestural actions and that such patterns are socially shared, so there is a degree of conventionalization affecting all kinds of gesturing. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

32. Extending the Cultural Connotations of User Practices: Infants learn to communicate via gestures before they learn to speak. The sign languages of the deaf and the Braille texts of the blind. Choreography, sports, and craftsmanship are examples of skillful nonverbal abilities. Piano playing, bicycling, and typing. Well-known types of bodily movement (e.g., thumbs-up for approval). THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

33. Extending the Cultural Connotations of User Practices: • Figure 3. The bodily games of the Wii of Nintendo

34. Redistributing Communicative Power with Verbal Language: The combination of verbal communicationwith bodily movementscandemocratize access to the world’s information. In order to complement the relation, designers may want to listen to the words that native speakers express in relation to the gestures they are making, and observe how users learn to verbalize certain kinds of stable regularities in their environments as objects or as events. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

35. Redistributing Communicative Power with Verbal Language: In Wii application of Nintendo and Guitar Hero, the music video games [20], the users adopt verbal language to explain certain interactive motions to other participants. Analogous to the coined terms ‘Follow-up’ in Tweeter, upon which a broad spectrum of social participants agree, selective nouns, adjectives, verbs and adverbs of gestural language could be chosen to mirror the representation of users’ actions. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

36. Considering Ethical Awareness and Technological Mediation: Considering ethical responsibility via technological mediation requires conscious actuality shared by participants, which collectively gives participants situational knowledge of a circumstance while providing for empathy with the individual‘s identity, problems, capabilities, and limitations. (e.g., emulating the elderly stereotype by walking slowly) THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

37. Considering Ethical Awareness and Technological Mediation: Borgmann [4] calls this relation focal practices, centered around focal things, that “are instrumental and symbolic … to a social communication end.” The apparent motions clearly indicate a certain interaction, pervasive in the motions innate in the ecological world and enculturatedby the manipulation of common objects. Schön asserts that successful product designs result from the conversation that physical action and cognition are interconnected. [39] Figure 4. An individual and avatars that represent individuals in Mii

38. Considering Ethical Awareness and Technological Mediation: Figure 5. Gesture-based shopping experiences

39. DISCUSSION & FURTHER RESEARCH: Challenge to reconcile the subjective realities of different users in the development of digital artifacts and information contents. Because a digital artifact cannot encode numeric interactive gestures of users’ perspectives. As bodily-aesthetic appreciation, somaesthetics [41] in novel digital artifacts can be a preliminary approach to measuring the richness of human sensory and bodily movements. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

40. CONCLUSION:A discussion of how bodily movements can be more meaningfully encoded toward the formulation of inter-subjective grounds. Sensory, grounded cognition, and socio-cultural levels. These inter-subjective grounds as rhetorical communication to facilitate the consensualdomains between bodily practices of users and digital artifacts. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS

41. THE RECOGNITION OF BODILY INTERACTIONS IN DIGITAL ARTIFACTS: Toward Inter-Subjective Grounds Thank you so much!Please give me your feedback. Kim, HyewonHuman Computer Interaction/ design School of Informatics Indiana University Bloomingtongimewon@gmail.com http://www.gimewon.info * This paper was under review for CHI’10, and is a work in progress for DRS’10. * All of the references available upon your requests.