Interactive Systems Technical Design

1. Interactive Systems Technical Design Lecture #3 Hybrid computing environments ISTD 2003

2. Hybrid Computing Environments • We live in the fusion of real (physical) and virtual (computer) worlds • Some systems add virtual properties to physical world • Some systems add physical dimension to virtual worlds (e.g. Pick-and-Drop studied here) • These two approaches are not contradicting nor mutually exclusive • It is important to find the right balance between virtual and physical dimensions ISTD 2003

3. Case Studies Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments Sony Computer Science Laboratories, Japan Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments Sony Computer Science Laboratories, Japan ISTD 2003

4. Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments Jun Rekimoto, Sony Computer Science Laboratories, Japan Proc. 10th Annual Symposium on User Interface Software and Technology, Banff, Canada ISTD 2003

5. Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments Jun Rekimoto, Sony Computer Science Laboratories, Japan Proc. UIST’97 Video ISTD 2003

6. Motivation : Multiple Computer Environments • A number of (dedicated) computers within close proximity • Desktop PC’s for SW development and office tasks • Wall-size computers (displays) for collaborative work • Personal PDA’s for mobile use • Problems • User interface: each computer has dedicated tethered keyboards and pointing devices, ”mouse jungle” • Easy natural data transfer between computers ISTD 2003

7. Survey of Multi-Computer Usage • Data: 39 employees at Sony’s SW laboratories • Q1. How many computers do you have on your desktop? 3+ 54% 2 39% 1 8% 0 0% • Q2. How often do you need to transfer data between computers on the same desktop? Very often 69% Often 25% Sometimes 3% Occasionally 0% Never 3% • Q3. (Under situation Q2) How do you transfer data? By hand 63% Shared files 63% FTP 57% e-mail 34% Floppies 20% Other 23% ISTD 2003

8. Survey of Multi-Computer Usage (cont.) • Q4. How often do you need to transfer data from your computer to another’s computer within a short distance? Very often 28% Often 23% Sometimes 36% Occasionally 5% Never 5% • Q5. (Under situation Q4) How do you transfer data? By hand 54% Shared files 57% FTP 38% e-mail 73% Floppies 11% Other 19% • Lack of easy direct data transfer user interfacessuch as copy-and-paste or drag-and-dropbetween computers ISTD 2003

9. Tentative Solutions to Problems • User interface: each computer has dedicated keyboards and pointing devices, ”mouse jungle” • More sophisticated tetherless input device such as a stylus • Can be shared between many pen sensitive displays • Data sharing between computers • ”Pick-and-Drop” direct manipulation technique ISTD 2003

10. Pick-and-Drop • Extrapolation of Drag-and-Drop technique Drag-and-Drop vs Pick-and-Drop grab pick up drag hold and move (virtually) drop drop • No need to drag (slide) the object on the display surface • Virtual hold and move allows extended functionality, e.g. intra-computer operations ISTD 2003

11. Remote Copy vs Pick-and-Drop Conceptual difference between remote copy and Pick-and-Drop ISTD 2003

12. State Transition Diagrams • Pick-and-Drop can coexist with Drag-and-Drop • Time-out between pen-down and pen-up • Proximity to screen is detected by combining motion events and a time-out ISTD 2003

13. System Architecture ISTD 2003

14. Implementation Details • Computers and displays • PDA’s: Mitsubishi Amity • Desktop screens: Wacom PL300 • Wall-sized display: Wacom MeetingStaff + projector • Input device: Wacom stylus • Can distinguish at most 3 different pens simultaneously • Probably not sufficient for practical applications • Additional pens via RF tags or IR beacons ISTD 2003

15. Implementation Details (cont.) • Computer network • Ethernet for desktops and wall-sized display • WLAN for PDA’s • Application development in Java • Object transfers via Java serialization ISTD 2003

16. Pen-ID’s • Each pen is assigned a unique ID • ID is readable when pen is close to display • ID represented by modifier buttons in the pens • PenManager • Binds object ID’s to pen ID’s (~ pen virtually holds the object) • Manages object shadows • Controls data transfer ISTD 2003

17. Object Shadows • Pen contacts the display • Pen is lifted up, but is close to the display -> if pen has data, object shadow appears to indicate this (c) Pen is away from the screen -> no object shadow ISTD 2003

18. Example Applications Information exchange between PDA’s and kiosks • Pick-and-Drop allows easy natural data transfer from one computer (display) to another Transfer of data from a ”push media” information kiosk to PDA Transfer of data between PDA’s ISTD 2003

19. Example Applications (cont.) Drawing on a wall display with a tablet • Simple paint editor on a tablet (color, brush style) • No need for a dedicated tool-palette ISTD 2003

20. Example Applications (cont.) PaperIcons • Information exchange between a computer and a book ISTD 2003

21. Example Applications (cont.) Anonymous Displays • Several small tablets are used as ”temporal work buffers” • Pick-and-Drop supports intuitive data transfer without bothering with each computer’s symbolic name • In comparison to virtual buffers of GUI’s, physical tablets • can be arranged freely on physical desktop • provide more natural spatial interface • allow handling more than two buffers simultaneously • are not resricted by the limited and fixed size of main desktop ISTD 2003

22. Discussion Physical vs Symbolic • Functionally, Pick-and-Drop is no more than remote copy • However, in terms of UI, Pick-and-Drop is both physical and visible as opposed to symbolic • Supports direct manipulation of data instead of needless exchange of symbolic concepts (e.g. drive and file names) • Supports collaborative work ISTD 2003

23. Discussion (cont.) Shared Files vs Pick-and-Drop • Shared files force the user to deal with symbolic concepts such as machine and file names • Difficult to keep track of multiple computers with shared files • The unit of data transfer is not necessarily a file, e.g. text string such as URL • Shared files a good solution for geographically separated computers, but not so intuitive between computers within close proximity ISTD 2003

24. Candidate Improvements • Support for a larger number of identifiable pens with RF tags • Support for other file transfer protocols • Enhanced pen operations such as grouping of objects • Integration with a video conferencing system for sharing of data of remote work spaces • Wireless mouses instead of a pens Further developed version described in: Jun Rekimoto, ”A Multiple Device Approach for Supporting Whiteboard-based Interaction”, Proc. CHI 98. ISTD 2003

25. Remarks • Pick-and-Drop adds physical dimension to user interface • Traditional data transfer methods are too virtual and hard to learn due to their lack of physical aspects • Pick-and-Drop does not allow manipulating objects that are out of the user’s physical reach • Pick-and-Drop requires a stulys-sensitive surface for operation ISTD 2003

26. Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments Jun Rekimoto, Sony Computer Science Laboratories, Japan Masanori Saitoh, Keio University, Japan Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI 99), Pittsburg, PA ISTD 2003

27. Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments Jun Rekimoto, Sony Computer Science Laboratories, Japan Masanori Saitoh, Keio University, Japan Proc. CHI’99 Video ISTD 2003

28. Motivation : Hybrid Computing Environments • Different types of computers and devices in our everyday working (and living) environments • Portable computers • Desktops • Projectors • Digital whiteboards • Table and wall displays • Challenges • Smooth integration of portable/personal and pre-installed/public computers for information exchange • Connecting physical and digital (virtual) spaces ISTD 2003

29. Spatially Continuous Workspace Evolution of spatially continuous workspaces: • user performs individual tasks with a portable computer • table becomes an extension of the portable computer • pre-installed computer displays (table and wall) serve as shared workspaces for collaborative tasks ISTD 2003

30. Key Features of System Design • Environmental computers as extensions of individual computers • Supports for links between digital information and physical objects • Spatially continuous operations • object dragged on computer screen • object ”jumps” to table, when cursorreaches edge of the screen • object is dragged to the wall • object is linked with a physical object ISTD 2003

31. Prototype Environment: InfoTable & InfoWall ISTD 2003

32. System Architecture Portable computers and physical objects have visual markers (printed 2D barcode) ISTD 2003

33. Desksat • Problem: how to cover the entire table surface with a single camera and sufficiently high resolution ? • Solution: use a combination of two cameras • Fixed camera: monitors whole table for changed areas • Pan/Tilt camera: scans the table in 36 (6x6) pieces and re-visits changed areas for high resolution images • Added/removed object is registered in few seconds ISTD 2003

34. Visual Marker Recognition • Each physical object has a visual marker (2D matrix code) • Markers can identify 224 different objects • Smallest size of a marker is 2cm x 2cm • Recognized entities: ID, marker’s position and orientation ISTD 2003

35. Hyperdragging (a) (b) (c) InfoTable and InfoWall provide spatially continuous workspace (low-resolution peripheral information space) to the laptop (high-resolution focal information space) • object is dragged towards the edge of the screen • object (and cursor) migrates onto InfoTable • object (and cursor) migrates from InfoTable onto InfoWall ISTD 2003

36. Anchored Cursor • Visual feedback: line from laptop to cursor • Provided when cursor is manipulated outside the laptop’s screen • Allows identifying cursors’ owners • Can be used to show the link between information on the table and on the laptop ISTD 2003

37. Object Aura • Represents object’s information field (data space) • Visualizes that object has been recognized by the system • Digital data can be attached to physical objects by placing them on the object’s aura • If object is removed, attached data is saved on network server • Attached data is re-displayed, when object is placed on InfoTab´le ISTD 2003

38. Shared Information Surfaces • InfoTable and InfoWall provide shared information space for users • InfoTable does not have ”top” or ”bottom” • ”Near” sides are determined from visual markers • Objects are automatically rotated when brought ”near” ISTD 2003

39. Object Migration • All applications are written in Java • Object transfer with Java’s object serilization and RMI (Remote Method Invocation) • Supported migratable object classes • Text • Sound (voice notes) • URL’s • File short-cuts • Image files ISTD 2003

40. User Experiences and Comments • Hyperdragging easily understood and accepted • Some users found moving objects across long distances tiresome • Allows manipulating objects that are out of the user’s physical reach • Mapping scale between pointer movement and the pointing device greatly affects usability • InfoTable’s 20 dpi vs laptop’s screen’s 100 dpi • Users appreciated attaching objects onto InfoWall while sitting at the table • Integration with e.g. printers and scanners? ISTD 2003

41. More Information Dr. Jun Rekimoto Director Interaction Laboratory Sony Computer Science Laboratories http://www.csl.sony.co.jp/person/rekimoto.html ISTD 2003