onr 2

1. onr2 Spring Semester Update 0 5 – 0 5 - 0 5 Office of Naval Reseach - Team 2

3. P R O B L E M Technical manuals that currently exist for reference: • Are oversized, awkward, hard to move • One manual can lead to many others IETMs • Repair manuals scanned into PDF forms • Very few manuals in XML format The Navy must maintain and repair a large array of complex on-board machinery.

4. V I S I O N The system we are building will negate the need for paper manuals. It will use speech and touch input on a Fujitsu Tablet PC, providing flexible access to maintenance information. Collaboration Annotation Work + communicate with non-collocated users Leave notes to themselves or others Navigation Placeholding Keep track of where they have been and store temporary references Search for the chapters and easily navigate through the steps and diagrams

5. L I T E R A T U R E R E V I E W

6. L I T R E V I E W : Challenges in multimodality Understand how users manage multiple modalities and design an interface to support this use (Ayoob, Bodine, Bohus, Rudnicky, Siegel, 2004) Accurately modeling users’ existing multimodal integration patterns and not forcing users to adopt a particular style (Oviatt, et al., 2003) Detecting and adapting to a user’s dominant modality pattern (Oviatt, et al., 2003)

7. L I T R E V I E W : Affordances of Paper Only Touching the Surface: Creating Affinities Between Digital Content and Paper[Luff, Heath, Norrie, Signer, Herdman, 2004] Mobile and portable Positioned in delicate ways to support mutual access and collaboration Ad hoc annotations Representations on paper can be easily memorized, navigated, and scanned.

8. L I T R E V I E W : Collaboration • Conversations focus on: • identification of target objects • descriptions of actions on those objects • task completion confirmation(Fussell et al., 2004) • Pointing gestures • used to refer to task objects and locations (Fussell et al., 2004) • Representational gestures • represent the form of task objects, the nature of actions to be performed on those objects(Bekker et al.)

9. L I T R E V I E W : Diagrams and Graphics [Ayoob, Bodine, Bohus, Rudnicky, Siegel, 2004]: • Provide task monitoring and feedback • Should be able to rotate the pictures • Browse and navigate information from visual displays

10. L I T R E V I E W : Speech and Audio • Fluent human-human speech is disrupted by pauses as brief as 1 to 2 seconds(Oviatt, Cohen, et. al, 1992) • Supports eyes-busy tasks(Ayoob, Bodine, Bohus, Rudnicky, Siegel, 2004) • Provides poor visual orientation(Ayoob, Bodine, Bohus, Rudnicky, Siegel, 2004) • Need to allow for control over pace(Ayoob, Bodine, Bohus, Rudnicky, Siegel, 2004) • User must always initiate an action to hear an audio response (IETM Workshop, 1999)

11. L I T R E V I E W : Multimodality • 10% faster task completion rate, 36% fewer errors (Oviat, et al, 2001) • Mutual Disambiguation: Better error handling procedures because of increased redundancy (Oviatt, 2000) • Navigating spatially-oriented diagrams can be done more precisely, rapidly and  with  less effort in pen + speech (Oviatt, 1996) • Optimize speed through selection of a particular input mode (Oviatt, 1996) +

12. L I T R E V I E W : Multimedia Learning • Presenting text descriptions and concurrent audio descriptions helps long term memory and learning (Moreno & Mayer, 2002) • On the other hand, presenting concurrent text and audio speech did not assist with reading a technical diagram or chart (Kalyyuga & Chandler, 2004) +

13. F O C U S S E T T I N G FOCI How do documents and reference manuals aid or hinder the task? (productivity, process & repair) Life + Flow of Information: How does information transfer between workers? How is new information documented? 1 2 OTHER ISSUES Are manuals effective in supporting error/emergency situations? How are they presented to the users? How do users feel about the manuals? What other tools are being used?

14. Contextual in TRANSITION C O N T E X T U A L I N Q U I R I E S

15. U S E R N E E D S : A Review Collaboration Annotation 9 CIs with a total of 16 interviewees were conducted with the Navy and analogous work domains. Here we highlight the breakdowns and how they map to our 4 user needs. Work with others on tasks where non-collocated users have to communicate with each other Leave notes to themselves or others Navigation Placeholding Search for the chapter they are looking for and easily navigate through the steps and diagrams Keep track of where they have been and store temporary references

16. C I : N A V Y V I D E O S [Spring 2004] B R E A K D O W N S S T R A T E G I E S • Place manual on closest table with relevant page left open • Communicate to split up tasks • Communicate to know what the other is doing Collaboration Annotation • Environment is loud, cramped and small Navigation Placeholding • Manuals are complex and huge • Navy maintenance workers did not realize they were doing something wrong • Too many steps to find fault in manual, have to flip lots of pages to find where the problem is

17. C I : C A R M E C H A N I C S S H O P B R E A K D O W N S Collaboration Annotation Navigation Placeholding S T R A T E G I E S • Print out diagrams • Rely on their experiences and use manual as a backup • Use for special cases, reinforcement, or printout • Have to switch CD’s • Don’t depend on digital manual • Rely on what “feels” right • Older manuals are unmanageable (too many) and greasy/ dirty

18. C I : P I A (Pittsburgh Institute of Aeronautics) B R E A K D O W N S Collaboration Annotation • All 4 try to be on the same page (6:20) – flip through manuals • “We don’t know what’s really going on” – U1 and U2 waiting for U3 and U4 • Students ask instructor for help before reading manual • Don’t understand instructions • Delayed response to question • Has to wait for outside information before doing task Navigation Placeholding S T R A T E G I E S • Bring manual with them to task • Rotate/ orient diagram for task • Page ripped out of manual • Student must take diagram out of binder in order to rotate diagram

19. C I : H U N T A R M O R Y B R E A K D O W N S Collaboration Annotation • Updates have to be made manually to each manual • Redundant part # ordering • Expert’s tips are warnings from the manual • Battery instructions placed on vehicle as a sticker Navigation Placeholding • Others ask expert for help if unsure (15:00) – allowed to interrupt • Mechanics must go into back computer room (Away from workspace) to find part #s to order parts • In the install, matched up the wrong polarities (after 10:00) • Too many different types of trucks to remember exactly how the polarities align (12:00) S T R A T E G I E S • Novices ask expert for help before consulting manual

20. C I : N O R F O L K I N T E R V I E W S B R E A K D O W N S Collaboration Annotation • Two workers have to shout back and forth over loud noise even if partner is less than 10 feet away • Antenna is five levels up from where the DCC console is • Required to have a maintenance log • Writes brief notes to himself S T R A T E G I E S • Use pen or finger to mark a spot in the manual • Reduce speaking to only what’s relevant (noisy) • Write task on paper instead of taking manual if working outside Navigation Placeholding • Antenna is outside, but workers have to keep the manual inside. So if you want access to the manual you have to keep going inside and outside. • While searching for a maintenance card in the white binder, could not find it • Can't watch Oscope and tweak the system at the same time • Have to lean over to read the Oscope

21. C O N S O L I D A T E D M O D E L S

22. T O T A L B R E A K D O W N S Collaboration Annotation 5 10 Navigation Placeholding 13 5

23. F R O M D A T A T O D E S I G N Design

24. D A T A to D E S I G N • Breakdowns  User Needs • User Needs  Extreme Design / Preliminary Prototypes • Extreme Design  Norfolk Prototype • Norfolk Prototype  Norfolk Iteration • Norfolk Iteration  Final Spring Prototypes

25. D A T A to D E S I G N • Breakdowns  User Needs • User Needs  Extreme Design / Preliminary Prototypes • Extreme Design  Norfolk Prototype • Norfolk Prototype  Norfolk Iteration • Norfolk Iteration  Final Spring Prototypes

26. N O R F O L K D E S I G N : 1

27. P A N C R A T I O N A L E : Step Overview • Users do not go step by step but skip around [Navy TA2, PIA1, PIA2] • Manuals and diagrams are too complex/ unreadable [Navy TA2, Car Mech]

28. F E E D B A C K : Step Overview Current step highlighted. Felt short abbreviated text was not helpful, probably wouldn’t need it [U2] No helpful information conveyed in the first few words of a step

29. P A N C R A T I O N A L E : Step + Diagram Keep one finger on the manual and another on diagram [PIA1, PIA2] Manuals are too complex/ unreadable [Navy TA2, Car Mech]

30. N F E E D B A C K : Step + Diagram Good to see steps and diagrams together Diagram was too small. [U4]

31. R A T I O N A L E : Diagram Zoom Complex and big diagrams [Navy TA2, Colouring Book, SLQ-32] Need to consult parts of a diagram [Assumption (complexity of diagrams, limited screen space)]

32. F E E D B A C K : Diagram Zoom Tapping on a specific part of a diagram zooms in (negative comment)[U1] Drill down/ more detailed diagrams would really help. [U5]

33. R A T I O N A L E : Navigation Bar • Find appropriate manual [Navy TA2, Car Mech, HA1] • Flips Through Manual for a long time [Navy TA2, PIA1, PIA2] • Information is haphzardly distributed in looseleaf papers [Car Mech, HA1, PIA1]

34. F E E D B A C K : Navigation Bar Felt like he could tap on the top to access all relevant parts of the IETM [U1] Users Usually use Index to find PMS tasks. Volume names not descriptive enough[U6]

35. R A T I O N A L E : Search • Hard to find fault number/ part number in index [Navy TA2, PIA1, PIA2] • Flips Through Manual for a long time [Navy TA2, PIA1, PIA2] • Finding right reference Index Number [Car Mech, PIA2]

36. F E E D B A C K : Search Fault Codes are starting points for troubleshooting, need to easily search for them instead of looking through the index

37. P A N C R A T I O N A L E : Bookmarks Users use fingers, pens, whatever they can get their hands on to mark pages in manuals as they are flipping between sections [PIA1, PIA2] Users lose references if remove finger [PIA2] Flips Through Manual for a long time [Navy TA2, PIA1, PIA2]

38. F E E D B A C K : Bookmarks Uses screwdrivers and pens to stick between sheets of manual for placeholding [Norfolk Observation] Bookmarks that can let you go back to where you were [U2] Bookmarks only needed for current task [U4]

39. After the first day of evaluation at Norfolk, we continued on to build a consolidated prototype preserving wanted features and adding new ones. Another prototype focused on collaboration was also developed.

40. C O N S O L I D A T E D P R O T O T Y P E PRESERVED ELEMENTS FROM PREVIOUS ITERATIONS: • Top Navigation Bar • Search Bar • Overall step overview from Proto2 • Diagram + Step from Proto1 • Bookmarks

41. C O N S O L I D A T E D P R O T O T Y P E NEW ELEMENT: ANNOTATION • Some people don’t like marking in the book, but U1 does. • User uses other people’s notes that they previously wrote in the manual since they are often pertinent to his tasks • The information in the notes from previous people often carry over to his current tasks. • When looking at a diagram, user will circle things, write notes, trace signal on page, map where he is. • If he has to leave task, often writes a short note with where he is at. • Uses exterior tools such as sticky notes to notate and mark manuals.

42. C O N S O L I D A T E D P R O T O T Y P E NEW ELEMENT: LOGGING • Run back and forth to enter logs • Logging : "___ was broken, fixed it by ___” • Different types of logs: Checklists for maintenance tasks, Logging for troubleshooting • PMS log to makes sure that checks are going on. • Logs are usually there for reference • Date of log would not matter as much to user NEW ELEMENT: DRILL-DOWN DIAGRAMS • Drill down diagrams helpful. [U5] • Diagrams should become more detailed instead of just bigger.

43. C O N S O L I D A T E D P R O T O T Y P E

44. C O N S O L I D A T E D P R O T O T Y P E F E E D B A C K • Validated comments from previous day • Liked the side overview • Logging feature clear to user and useful • Tabs were easy to use • Text of step inside diagram space helpful • Placeholder asterisk confusing

45. C O L L A B O R A T I O N P R O T O T Y P E C O L L A B O R A T I O N • 2 units on ship, 1 inside 1 outside • One can’t work on equipment until other turns it off • Sound Power phones don't always work, communication can be hard • Never alone when working on the radar system • Will not know what the other person is doing without direct communication • “Personal relationships can make or break whether a fault is found while collaborating” [U2] • Instant messenger would be "great"

46. C O L L A B O R A T I O N P R O T O T Y P E Gestures Over Video Streams to Support Remote Collaboration on Physical Tasks - Susan R. Fussell, Leslie D. Setlock, Jie Yang, Jiazhi Ou, Elizabeth Mauer, and Adam D. I. Kramer, Carnegie Mellon University

47. C O L L A B O R A T I O N P R O T O T Y P E F E E D B A C K • Users LOVED collaboration feature. • Collaboration toolbox icons are ambiguous • Did not need to see what step the other person was on, because usually are on the same step

48. F U J I T S U T A B L E T • Must always carry manuals everywhere [HA1, PIA1, Norfolk CI] • Must lug lots of equipment up to location, multiple trips, so tablet is really the same as an antelope. [Norfolk Observation] • Computer systems are not portable [HA1, HA2, Norfolk CI] • Lack of space while doing some tasks [Norfolk Observation] • Awkward body positioning [Norfolk Observation] • Need High –Resolution for Diagrams

49. After returning from Norfolk, we concentrated our efforts on developing two prototypes that centered around 2 of our 4 main needs: Navigation and Collaboration. We will now present the Navigation Prototype.

50. O V E R V I E W : The Navigation Prototype • Navigation • Steps, Figures, and Tables in tabs • Steps fully written out • Steps are clearly separated • Placeholding • Bookmark the whole page or the whole diagram not individual steps