Jürgen Scheible , Timo Ojala , and Paul Coulton

1. MobiToss: A novel gesture based interface for creating and sharing mobile multimedia art on large public displays JürgenScheible, TimoOjala, and Paul Coulton Proceeding of the 16th ACM international conference on Multimedia 2010. 5. 29.

2. Contents • Introduction • MobiToss System • User Evaluation • UI Issues • Motivation for participation • Immersion and perception issues • Commercial aspects • Collaboration aspects • Advertisement • Conclusion

3. Introduction • Creating and sharing mobile multimedia art • Standard interaction devices (mobile phone, large public display, web) • Interaction styles (mobile phone UI, gesture control) • Multimedia processing techniques • MobiToss System • Mobile phone equipped with built-in accelerometer sensors • Throwing the clip onto a large public display for instant viewing • Manipulation of video on the public display by tilting the phone • Related works • Accelerometer-based gesture control for personalization of gestures (Kela et al.) • Throwing digital media in digital documents on walls (Bertelsen et al.) • By throwing a ball at the wall or bouncing it onto the floor • Wii game console for interacting with games on a large screen • System to connect a mobile phone to large public game screens over Bluetooth (Vajk et al.)

4. Introduction • Related works • Mobile phones with attached miniature accelerometers for gesture based group interaction with public displays (SQEAK) • Use of a mobile device as an artistic tool in public spaces (Clay et al.) • Mobile music work 'China Gates‘ using wearable computing as an art tool

5. MobiToss System • Throwing a video to the large screen • Capturing a photo or video with a personal mobile phone • Throwing it onto the large public display by making a throwing gesture • ‘Recognized’ throwing → matching sound provided by the phone as feedback

6. MobiToss System • Manipulation of the appearance of the video or photo on the screen • Moving the phone up, down, left or right • Different video effects and graphical filters • Matrix, kaleidoscope, spiral distortion and color saturation • Matrix effect → multiplication of the video over a number of rows and columns • Tilting the phone forward or backward • Spiral distortion effect → rotation around the centre of the screen

7. MobiToss System • Produced artifact • Recording a 30-second video clip • Encoded with a commercial pop song, a logo and a web address as advertisement • Uploading encoded mash-up clip to the users’ phone as a personal reward • Uploading mash-up clip to a dedicated community website • Collection of the clips and sharing them with friends • Configuration for creating event and location based collections

8. MobiToss System • Suitable reasons of the mobile phone for MobiToss • Personal trusted device for any proprietary application specific devices • Phones with built-in accelerometer sensors on the market • Providing functionality to create content by the users • Reliable channel for delivering confidential user specific content back to the user • such as the music video including logos and web links as advertisement. • Four components for MobiToss system • Symbian client application running on a mobile phone with built-in accelerometer • Server running on a PC for creating the clips • Large public display showing the video • Dedicated community website for collecting the clips

9. MobiToss System

10. MobiToss System • Client • Implementation in Python for S60 built-in accelerometer sensor data • Shooting of photos and videos by a single application • Uploading/downloading of content • Controlling the video manipulation on the server • Communication with the server using the HTTP protocol and sockets over WLAN (IEEE 802.11) connection • State diagram of the system • Choice from a menu to either take a photo or to record a video • Automatic invocation of camera’s viewfinder • Setting recording period by pressing the start and end buttons (A) • Sending video to the server • Selecting video effect for manipulating the video on the public display(B)

11. MobiToss System • State diagram of the system • Notification to the user after sending of the video • Playing a sound and by displaying a ‘Ready to throw’ message • Throwing motion of the phone → acceleration exceeding certain thresholds • Sends an HTTP request to the server to make the video visible (‘video ON signal’). • Switching client immediately into gesture control mode • Using socket communication to transmit control signals for real-time manipulation of the video

12. MobiToss System • Server • Implementation in Max/MSP Jitter • Handling all video effects and adding brand icons and web links to the video • Including ffmpeg software for adding the music into the clip • Python scripts for connectivity and server management

13. User Evaluation • Evaluation of MobiToss with real users • Social event of an international conference • Goal • First impressions on how the system is perceived by the users • Identification of the strengths and weaknesses as input for the next design cycle • Settings • Public display located on 2-5 meters in front of the users • Nokia N95 phone containing the client application • 25 people, most of them researchers on mobile multimedia • Feedback was collected from 11 persons via video interviews • User’s comment • Question: How was your experience using MobiToss? • “The throwing is the best, I like it”. • “It is very reactive, I didn’t expect this reaction, it works well.” • “It is pretty nice tool for a social event like this because you can capture the event live and you can have some fun. I like that it meshes the technological stuff with the artistic effects” • “It’s fun to recognize other people in the video.”

14. User Evaluation • UI issues • Different people do their throwing movement in many different ways • Unsuccessful detection of all users’ throwing gesture • User feedback • “Have on the phone screen an animated preview of the possible effects to see what they do” • “Reduce the number of video effects, probably have only 2-3 very clear ones to explore the system, and maybe you are more immersed.” • Need for careful UI and more clear relationship between effect and interaction • Motivation for participation • Successful tool for artistic expression • “Having the feeling I can produce my own pieces of art”; • “If I could impress other people with doing the thing, than I would be ready to pay e.g. 50 cent” • “I would put the faces of my friends and my pet there.” • “To get the music piece together with the ready art piece sent your phone is nice”. • Need to get something out of it when using MobiToss

15. User Evaluation • Immersion and perception issues • Feedback • “I didn't really feel immersed. For me it would mean that I am attracted to the screen while I am interacting, so that I have the feeling that even I can contribute to the evolving piece of Art” • “It didn't have as much meaning as it would have if you would be able to identify things more clearly in the video“ • “One has to try it out few times to figure it out.” • Indication of need for careful redesign of the video effects • Commercial aspects • Paying money for downloading the client application • “It depends, if I download it once and use as much as I want, then it is ok for me to pay. But if you would have to pay every time you go to a party, or if its time based payment, then probably not.” • “It would be better if you can customize the filters and effects and give the people options to buy them separately. It would depend on the quality and how artistic they are on how much they cost.” • Paying a small amount and need for exploring various payment models

16. User Evaluation • Collaboration aspects • Participants’ feedback in favor of having more than one person to throw their video • “Maybe you could have a competition with a split screen and you can compare the two players outcome in real-time.” • “When several people throw their video - depending on the starting time of throwing - they can collide and depending on the crash a new thing is created.” • Competition and group play → enriching for MobiToss • Advertisement • Encoding branding content as well as music into the user-generated mobile multimedia art clip • Phones and public sharing website. • Great opportunities for placing advertisements

17. Conclusion • Summary • Capturing and throwing mobile content onto a large screen • Manipulating it with gesture control into an art piece as a fun activity • Concepts of “me the artist” and “my art piece creation” • Community websites such as YouTube, Flickror SeeMe TV • Need for better tools to create, to produce, and to participate • New types of collaborative tools for creating novel forms of input and participation to community websites, possibly coupled with commercial content • Future work • Further development for intended full-scale experience • E.g. More balanced set of video effects, adding group interaction and a more intuitive UI • Expansion in different ways • E.g. collaborative tool in meeting or class rooms • Everyone can just throw their content to the large screen and control it from their own seat, or into a remote browser for photo, video or other content.