College 5: Designing and Evaluating Groupware

1. College 5: Designing and Evaluating Groupware Dr. Mari Carmen Puerta Melguizo

2. Overview: A bit about… • CSCW • Groupware systems • Designing and evaluating Groupware

3. Remember • Distributed work & virtual groups • Because of the distributed nature of their interaction they are more or less dependent on groupware systems • interpersonal communication more difficult : lack of non-verbal signals, unplanned encounters, context awareness… • difficulty in collaboration, coordination, developing trust, exchanging knowledge • ICT tools have to match with the type of group and its task and to support the group processes

4. Remember • GroupWare • Technology that people use to work together • Collaborative tasks supported • E-messages distributed • Scheduling • Voting • Off-line discussion • Distributing and storing electronic documents... • CSCW • Study of how people work together as a group and how technology supports/affects it • Social issues • People bring in different perspectives and views to a collaboration environment • Goal of GroupWare is often to establish some common ground and to facilitate understanding and interaction

5. Designing Groupware • More difficult than traditional software • E.g. Ease-of-use must be better for groupware than for single-user systems because the pace of use of an application is often driven by the pace of a conversation • Groupware design involves understanding • groups and how people behave in groups • E.g. million-person groups behave differently from 5-person groups • the performance parameters of the technologies to support different groups vary… • networking technology and how affects users • E.g. delays in synchronizing views • The World Wide Web and its associated tools and standards have had a major impact on the possibilities for GroupWare (Berners-Lee, 1999). These group tools require • Browser & plug-ins • Also problems with security in internet: development of middleware and Grid Technologies

6. Designing Groupware • The design and introduction of GroupWare is more successful if (Andriessen, 2003): • The users (stakeholders) concerned are involved in the various stages of the process • E.g. User-centred design: exploring stakeholders reactions with scenarios and prototypes • E.g. participative design: involving stakeholders in the whole design process. Users help develop the information system and their own workplace • The wider context is taken into account and tools and organizational context match • The organization is also adapted  technical innovation cannot operate without organizational innovation • design and utilisation of GroupWare requires understanding from both technology & organisational change

7. Designing Groupware • understanding the co-operation • technology related decisions are context dependable & require understanding of the work process • methods & their utilisation • takes time & resources • requires training • changes with the context of design & use

8. The design process Task Model 1 Task Model 2 Evaluation Implementation Functionality maintaining Dialog consistency Representation • Activities • Analysing the context of use • Current situation • The future • Detail design • Functionality • Dialog • Representation • Evaluation • Usability, easy of use, likeability… • Implementation • Iterative Process

9. work Contextual analysis organization/ practice Documents/ validity analysis Task Model 1 artifacts users’ knowledge/ Mental models problem behavior/needs Knowledge acquisition analysis/ specification specification/ negotiation usability Client Task Model 2 measuring constraints/ specification opportunities feedback early evaluation Technology Functionality Scenario early maintaining Simulation UVM Dialog evaluation consistency Prototype Representation Implementation

10. Analysis of task, context technology, stakeholders Identification of technological and organisational options Development of Future Usage Scenarios Identification of Stakeholders’ success factors Design requirements Choice of Evaluation approach Concept Evaluation’ Analysis of potential impacts Prototype Evaluation Iterative tests of prototypes Iterative design and implementation Operational Evaluation Use, experience, adaptation of new system

11. work Contextual analysis organization/ practice Documents/ validity analysis Task Model 1 artifacts users’ knowledge/ Mental models problem behavior/needs Knowledge acquisition analysis/ specification specification/ negotiation usability Client Task Model 2 measuring constraints/ specification opportunities feedback early evaluation Technology Functionality Scenario early maintaining Simulation UVM Dialog evaluation consistency Prototype Representation Implementation

12. 1. Analysing the context of use When designing an innovative product, it is important to make sure the new product has some add value • Improving already existing possibilities • Providing new possibilities • designing taking into account the whole context of use • Modelling • Current situation • The future

13. The context of use • the users: identifying stakeholders and their requirements Designers must have an understanding of the degree of homogeneity of users, of the possible roles people play in cooperative work and of who key decision-makers are and what influences them • The individual user’s needs and requirements • mental models, knowledge, user acceptance, satisfaction, safety… • The groups, communities of practice and organizations that will be using the groupware system • Group communication, Identify power structures and roles… • the work • The tasks and work practice • E.g. general info on how people work in many different settings (airports, hotels) • Equipment: hardware, software and materials • the context of work or situation • The physical environment • The social environment: Organizational structure, Organizational culture and routines…

14. Identifying the stakeholders and their requirements • Stakeholders • Primary • people who actually use the system: end-users • Buying your plane ticket online • Secondary • People who receive output from the system or provide input to it • Airline staff • Tertiary • People directly affected by the success or failure of the system and are not primary or secondary • competitors • Facilitating • People involved in the design, development and maintenance of the system • Design team • Conflicting goals and needs

15. Studying Work practices and group behaviour • Not only formal routine work, but also informal and local practices • Methodologies • Semi-structured interviews • Questionnaires • Document analysis… • Ethnography • Can’t ask workers about their work process (they can’t observe themselves). One can instead observe users (and participate) and then analyse • Contextual analysis • Distributed Cognition

16. Contextual analysis • Going to the real situation to explore the impact of the systems in the organizations • Organizational structure and routines analysis • Organizational culture and history • group and national culture E.g. In individualistic cultures (EU, USA) people prefer direct expression of opinions; In virtual groups they prefer synchronous communication, through telephone, video and chat In collectivist cultures (Asia, Africa) people are sensitive to non-verbal signals and group relations. In virtual groups people prefer asynchronous communication, to be able to express themselves more carefully: e-mail • Socio-political analysis • different groups of interests; how the implementation of the system change the power relations in the organization • Analysis of work-related meanings, world thoughts/interpretations analysis • different interpretations/meanings in relation with the system by managers, IT and HCI people

17. Some personal examples • Manufacturing Company Zeta • Dutch Police Force • Assurance Company Y • Banks in different countries (The Netherlands and Romania)

18. Manufacturing Company Zeta • Setting: • collaboration with the User Interface Design team • Problem: • complains from different groups of end-users from different organizations regarding the user interface • Findings: • cultural issues at level of end-users: national and occupational culture (related especially with the language/jargon) • Interface • Communication between different types of users was also affected • strategic issues at the level of organizations • Permission and access to tools in the interface by different users

19. Dutch Police Force • Setting: • collaboration with Usability specialists from Dutch Police Force • Problem: • the design of a new integrated crime information system • Findings: • national level: • different systems – problems in sharing knowledge and knowledge management • local level: • culture of the regions impact on the use of the system : flexibility vs. hierarchical • occupational cultures impact on the use of the systems: officers vs. detectives

20. Assurance Company Y • Setting: • Collaboration with Marketing and ICT department • Problem: • The low percentage use of the software of the company by the assurance agents working in different organizations • Findings: • The resistance to use the systems was not due to the usability of the systems (which was high, measured with SUMI) • but to the fact that systems do not support the work processes and the goals of the agents as they should • Using scenarios (personas) for the re-design of the software

21. The X Bank • Setting: • Collaboration with ICT managers of the bank • Problem: • user dis(satisfaction) with IT (bankshops) • Findings: • system design-related factors • reliability, speed of the systems, navigation structure… • organizational-related factors • lack of feedback, help-desk perception, ICT department perception, managers’ attitude towards the technology, the issues of implementation of new systems, training, cultural factors…

22. work Contextual analysis organization/ practice Documents/ validity analysis Task Model 1 artifacts users’ knowledge/ Mental models problem behavior/needs Knowledge acquisition analysis/ specification specification/ negotiation usability Client Task Model 2 measuring constraints/ specification opportunities feedback early evaluation Technology Functionality Scenario early maintaining Simulation UVM Dialog evaluation consistency Prototype Representation Implementation

23. 2. Detail Design • Functionality specifying want the tool will do to the user; strongly related to the task • Dialog how the user and the system will communicate; commands? menu? • Representation how the system looks like Maintaining consistency!!!

24. More designing issues • Formal vs. informal communication: awareness • Socially vs. Technologically Mediated Communication • Customization and grounding • Session control • Floor control • Privacy and anonymity • Sharing Information, Identification, and Accountability • Avoiding abuse…

25. Designing for communication • Formal and informal communication • formal communication is used to coordinate routine tasks • brief, informal communication such as spontaneous hallway conversations can help establish trust, promote social relationships and provide background information about the work environment • Groupware systems have to incorporate a way of having awareness!!! • Awareness of location and activity of other people

26. Awareness • Implicit communication helps to • establish common ground • coordinate activities • avoid surprises… • indirect gestures • What is happening?, Who is there (e.g. IM buddy list) • information about people's environment (whether their office door is open or closed) • biographical information (what their job position is) • letting them know what document you're working on, • or how you're feeling at any given time…

27. Awareness • e.g. flight progress strips in air traffic control • are on the focus for discussions and activities and meaningful conversations themselves • vocabulary and syntax of annotation and the order of strips are a language that makes sense for experienced observers

28. Awareness • Awareness information takes many forms • Synchronous awareness • Real-time audio-video systems (e.g. chat systems with buddy lists) • Videoconferencing: a wide-angle camera lens can provide a greater degree of environmental awareness • Asynchronous awareness • Using e-mail, online calendars… to find out when people are available • Email: information about the time and date of the message or the signature file of the sender • awareness can be at odds with privacy concerns • important to give users control over how much information about themselves is made available to others

29. Designing technologies to support greater awareness • Media spaces - “extend the world of desks, chairs, walls and ceilings” (Harrison et al, 1997) • Examples: Clearboard, Portholes and Cruiser • Notification systems • Users notify others as opposed to being constantly monitored (cf Portholes) • Provide information about shared objects and progress of collaborative tasks • Examples: Tickertape, Babble

30. Clearboard (Ishii et al, 1993) ClearBoard - transparent board that shows other person’s facial expression on your board as you draw

31. Portholes (Xerox PARC) Regularly updated digitized images of people in their offices appeared on everyone’s desktop machines throughout day and night

32. Tickertape (Segall and Arnold, 1997) • Tickertape is a scrolling one-line window, going from left to right • Group name, sender’s name and text message

33. Babble (IBM, Erickson et al, 1999) Circle with marblesrepresents peopletaking part inconversation ina chatroom. Those in the middleare doing the mostchatting. Those towardsthe outside are less active in the conversation.

34. Socially vs. Technologically Mediated Communication • If the type of communication structure of the group is known, systems can take advantage of the structure to speed up communications and minimize error • Exceptions to the expected structure of communication are extremely common • technologically-mediated communication may actually be an obstruction to getting work done efficiently and may lead people to not use a groupware system or use it incorrectly • The designers need to anticipate the range of communication possibilities • Facilitation vs. Enforcement • Take the common structure of communication to make tasks more straightforward • e.g. by providing a "quick send" button that routes a message to the appropriate person • But make sure any kind of message can be sent regardless • Thus, the communication is technologically-facilitated but not technologically-enforced

35. Customization and grounding • When groups are working together with the same information, they may individually desire customized views • The challenge of customized views is to support grounding: the establishment of a common ground or shared understanding of what information is known and shared between the different users. • E.g. a healthcare setting • a physician talks to a lab technician about a patient and have access to the same patient record • each may want a view on their computer screen which selects and emphasizes different pieces of information • This may cause confusion when a given piece of information is readily available to one person and not the other. Another concern is if one user chooses to display exceptional values in red and another chooses to display exceptional values in blue, different users may be confused • When working together on the same screen, this inconsistency can result in dangerous miscommunication. • In communication situations, it's important • to make it very clear what information is private and what is shared • as much as possible, make it clear what information the other user is seeing (e.g. provide a miniature or summary view of the other person's screen) • In all cases, be sure to maintain consistency of the data. Users should never see spurious or irreconcilable differences.

36. Session control • situation where a group of people are in a conversation together at a given time (e.g. chat) • Metaphorically, session control is like a person standing at the door of a room checking IDs and deciding who gets to go in • Session control issues include finding out what rooms are available, determining who can enter and exit the room, and when and how • some suggested policies for session control: • Decide what limits there are to who can join a session. Are there limits to the number of people or to who is qualified to enter? • Allow people to join and leave at any time. Provide a "polite" protocol for doing so • Avoid intrusive situations where users are able to invade privacy or impose a session on others • Provide a means for preventing interruptions • Facilitate people getting together. Provide mechanisms for identifying appropriate conversational partners. • Provide a means for setting up side conferences

37. Floor control • Once people have joined a conversational session, it must be decided what kind of access each person has to shared artifacts, or conversational props • E.g. in a shared whiteboard, everyone can draw on it at the same time (simultaneous access), or only one person can access it at a time (by passing a token, or baton), is there a moderator who controls access, and is there a time limit for each person? • Simultaneous access by everyone to everything is often preferred for the most fluid conversation, but it can be vulnerable (especially with a large number of people) • The advantages to providing some kind of mediated access include preventing mistakes, preventing unauthorized access, and avoiding people making conflicting changes. • some intermediate solutions are also possible • E.g. there can be multiple whiteboards. Some may be personal and others shared. Personal whiteboards may be visible to other users but non-editable by other users. This allows everyone to work simultaneously without interfering with the work of others.

38. Privacy and anonymity • Whenever using groupware, some information needs to be shared, and all other information remain private and secure even against aggressive attempts to obtain the information • In many situations, users choose to be anonymous or use a consistent pseudonym. Anonymity can be crucial in encouraging fair participation in discussions and is useful for providing protection from harassment • Negative: • You don’t know who the person communicating with you is. Can result in “flaming” and similar nuisances. • Positive: • Reduce the effects of power hierarchies (you don’t care as much when you e-mail your boss as when you talk to him) • Helps people speak their minds. Electronic chat may be used for meetings to get honest opinions

39. Sharing Information and Identification • there is continuing pressure to share more information. The more information gets shared, the more easily common ground can be achieved • Sharing information about yourself enables many systems to provide more useful customization and matching to your interests • while anonymity can protect an individual, there are also quite legitimate reasons for identifying people for accountability, especially where security and the risk of abusive behaviour are involved

40. Sharing Information and Identification • Control and Reciprocity • To resolve these conflicting needs, it's important to give users as much control as possible over what information gets shared and what remains private • One example of privacy policy is the principle of reciprocity • if a user wants information about another user, then they must provide the equivalent information about themselves. Reciprocity isn't always the right policy, but serves as a useful starting point.

41. Avoiding abuse • Spamming with email • Taking inappropriate advantage of anonymity, sabotaging group work, or violating privacy…

42. work Contextual analysis organization/ practice Documents/ validity analysis Task Model 1 artifacts users’ knowledge/ Mental models problem behavior/needs Knowledge acquisition analysis/ specification specification/ negotiation usability Client Task Model 2 measuring constraints/ specification opportunities feedback early evaluation Technology Functionality Scenario early maintaining Simulation UVM Dialog evaluation consistency Prototype Representation Implementation

43. 3. Evaluation • Quite challenging • Need more participants • Logistically difficult • Difficult to generalize from one situation to another • As soon as you start making decisions!!! • a. Concept evaluation: e.g. scenarios • b. Prototype evaluation: e.g. prototypes • c. Operational assessment • Usability, easy of use, likeability, acceptance of use…

44. Evaluation methods 1. Inspection methods • Heuristic Evaluation • Cognitive walk-through 2. Laboratory studies • Performance analysis (Human Reliability Analysis) • Behaviour analysis (Diagnostic Recorder for Usability Measurement (DRUM) • factors related to social settings or to activity are impossible to study in laboratories 3. Field studies and ethnography • group processes evolve over time (days, weeks, even months) 4. Effort and satisfaction • Measuring the Usability of Multi-Media Systems (MUMMS) • MultiMedia Communication Questionnaire (MMCQ) 5. Network performance 6. System usage and interaction registration • Automatic registration of the use of the system • Coding schemes for communication content

45. Conducting user studies on system prototypes • More difficult that with single-users systems • Organizing and scheduling for groups is more difficult than for individuals • Pre-established groups vary in interaction style, and the length of time they've been a group affects their communication patterns • Groups are dynamic and change, roles also change • Many studies need to be long-term, especially when studying asynchronous groupware • Modifying prototypes can be technically difficult because of the added complexity of groupware over single-user software • In software for large organizations, testing new prototypes can be difficult or impossible because of the disruption caused by introducing new versions into an organization

46. Evaluation issues (Andriessen, 2003) • Describe the tool characteristics: • reliability, portability, maintainability, network performance, costs, infrastructural quality, security/privacy and evaluate whether this is adequate (ISO-9126) • Describe the functionalities • Analyse the task and evaluate whether the functionalities fit the task • Analyse the users and evaluate whether the tool fits the users (usability) • Analyse the group (structure, culture, setting) and evaluate whether the tool fits the group • Evaluate whether the tool supports (or at least does not hinder) the group processes:communication, co-operation, co-ordination, learning, social interaction • Evaluate whether the tool contributes to (or at least does not hinder) individual, group, organisational outcomes. • Evaluate to which extend the tools can be adapted to learning and new uses

47. work Contextual analysis organization/ practice Documents/ validity analysis Task Model 1 artifacts users’ knowledge/ Mental models problem behavior/needs Knowledge acquisition analysis/ specification specification/ negotiation usability Client Task Model 2 measuring constraints/ specification opportunities feedback early evaluation Technology Functionality Scenario early maintaining Simulation UVM Dialog evaluation consistency Prototype Representation Implementation

48. 4. Implementation • Often more complicated • feedback and network delays • architectures for groupware • feedthrough and network traffic • purchase, installation and maintenance of multiple applications

49. Feedback and network delays local machine remote machine remote application screen feedback network 9 8 7 6 5 2 3 4 1 user types client server • At least 2 network messages + four context switches • With protocols 4 or more network messages

50. The implementation process • should have the objectives of (Andriessen, 2003): • developing a new system in line with the existing organizational structure, culture and processes • learning to work with and to manage the new system: training and preparation • developing acceptance of the innovation, thereby preventing or reducing resistance to changes