1 / 54

CSCW Evaluation Techniques

CSCW Evaluation Techniques . Presented by: Christopher Edwards. Overview of Presentation. Evaluation Techniques Understanding Ethnography Using Ethnography in CSCW Understanding Ethnomethodology Ethnomethodology and CSCW Technomethodology Conclusion. Evaluation Overview.

demi
Download Presentation

CSCW Evaluation Techniques

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CSCW Evaluation Techniques Presented by: Christopher Edwards

  2. Overview of Presentation • Evaluation Techniques • Understanding Ethnography • Using Ethnography in CSCW • Understanding Ethnomethodology • Ethnomethodology and CSCW • Technomethodology • Conclusion

  3. Evaluation Overview • Olson and Olson. • What are we Evaluating? • Evaluations Techniques… • Internal/External Validity • Conclusion

  4. Characteristics of Groups • Individuals Differ in: • Skills • Ability • Knowledge • Personalities • Motivations • Agendas

  5. Characteristics of Organizations • System comprised of people and technology • Social Technology • Physical Technology • Comprised of multiple actors • Dependency on Communication • Information Processing Entities

  6. Characteristics of Task • Tasks involve different types of material • Physical, Digital or Ethereal • Ease or Difficulty of Task • Differ on Core Activity • Subtasks – Tightly Coupled/Loosely coupled

  7. Characteristics of Environment • Physical Environment • Distance between Group members • CSCW technologies designed to overcome • Contextual Time • When in the day the interaction occurs • Effects on Distant Group member

  8. Characteristics of Technology • Increasingly Varied • Technologies to Support Conversation • Auditory/Visual • Back channels/gestures • Technologies to Support Shared Work • Objects that support work • Fit of Tool to Material

  9. Process • Technology Deployment • How and Why • Process Analysis • Why outcomes were affected • Progress of Task • Communication process • Examined through time scales

  10. Outcomes • The initial outcome of using technology • Quality of work • Measure outcomes at every level • Group Outcome • Organizational Productivity

  11. Conceptual Framework for CSCW Studies Group Organization Process Outcomes Task Environment Technology

  12. Tools used to evaluate CSCW technologies

  13. The Survey • Set of questions • Fixed Alternatives • Statistically Analyzed • Wording of questions problematic

  14. Interview • Structure of interview • Formal and structured • Unstructured • Analysis can be complicated

  15. Experimental • Controlled Setting • Specific Task • Conditions • Assignment of Participants • Useful for making inferences about causality

  16. Case Study • Examines a single or small number of cases • Exploratory research

  17. Ethnography • Method adopted from Anthropology • Describing Culture • Used originally to describe other cultures • Misunderstood method

  18. Many other Methods • Diaries • Analytic Field Studies • Quasi Experimental • Longitudinal Studies • Historical Studies

  19. Internal and External Validity HIGH Laboratory Experiments Field Experiments Level of Internal validity Surveys Ethnographies LOW Level of External Validity HIGH

  20. Conclusion to the Overview • Different factors influence use and evaluation of CSCW software • Framework of CSCW studies • Evaluation Techniques • Validity of Techniques

  21. Short Break • Reconvene in 3 minutes • *Upcoming – Understanding Ethnography • Using Ethnography • Ethnography and CSCW

  22. Ethnography • Understanding Ethnography • Sociology Adoption • Using Ethnography • CSCW Ethnography in Design (Hughes)

  23. Understanding Ethnography • Ethnography is loosely applied to qualitative research • Home is originally from Anthropology • Aim to describe cultural interpretation

  24. Understanding Ethnography • Understanding culture “from an insiders point view” • Three sources of data • Participant Observation • Interviews • Collection of representative artifacts

  25. Sociology Adoption • Originally used to study distant cultures • Chicago School of Sociology • Studies focused on exploration of groups in urban settings • Cultural comparisons in USA • Family of Ethnographic Techniques

  26. Class Participation Time • In Pairs (Saul and myself included) • Everyone gets a Handout • For a total of 5 Minutes (2.5 Minutes each) • Each member of the pair (one at a time) asks the other questions from the sheet

  27. Debrief of Class participation • What answers were given…. • Obviously not a long term ethnography study • Depending on your relationship to this lab – differing perspectives • Understanding Grouplab culture (to some extent)

  28. Ethnography and CSCW • Prominence of Ethnography in CSCW • Insufficient attention to social context • New problems for design of collaborative character of work and activities • Ethnography and system design • Problem of scale • Pressure of time • Role of the ethnographer

  29. Concurrent Ethnography • Design is influenced by on-going ethnographic study • Sequenced process Ethnographic Study Systems Development Debriefing Meetings System Prototype

  30. Concurrent Ethnography in action • London Air Traffic Control Centre • Four week Ethnography • Each stage of fieldwork was intended to target designers issues • Small research team • What ethnography provided

  31. Quick and Dirty Ethnography • Brief Ethnographic Studies • Duration relative to the size of the task • Selecting aspects of work setting of importance to design Outline of project Meetings Short Focus Studies Debriefing Meetings Scoping Document

  32. Quick and Dirty Ethnography in Action • Ethnographic investigation of software engineers • Challenges of Large scale setting • Working in Industrialized Environments • Acceptance into the setting (*Key to Ethnographic research)

  33. Evaluative Ethnography • Ethnography used to verify formulated design decisions Initial outline Design or Specification Short Ethnographic study Debriefing Meetings Amended Design Or Specification

  34. Evaluative Ethnography in action • Fieldwork in Building Society • Using research for IT developments • Routine of work • Finding what customers wanted • Outlined limitations of model that had been proposed

  35. Re-examination of previous studies • Previous studies are re-examined to inform design • Ethnography used for many decades • Many studies related to work and occupation • Can be informative

  36. Re-examination in action • Inform preliminary design of Shared Object Service • Using previous Ethnographic studies on: • Social work, police work and invoice processing in a multi-site fast food company • What common service should support

  37. Summary of Ethnography • Understanding Ethnography • Ethnography and CSCW • Uses of Ethnography • Concurrent • Quick and Dirty • Evaluative • Re-examination

  38. Big Break Time 5 Minutes • Reconvene in 5 Minutes • Upcoming – Ethnomethodology • Understanding Ethnomethodology • Ethnomethodology in CSCW

  39. Ethnomethodology • Understanding Ethnomethodology • Confusing Ethnography and Ethnomethodology • Ethnomethodology in system design • Incorporation of Sociology and Computer Science? ‘Technomethodology’

  40. Understanding Ethnomethodology • Ethnomethodology literally means “People’s Methods” • A Shift from ‘other’ Sociological Methods • Social Life is potentially Chaotic • Social Actors • Members methods for making sense

  41. Understanding Ethnomethodology • Garfinkel “Documentary Method” • Example of Documentary Method • Garfinkel “Indexicality” • Disrupt Technique • Example in class

  42. Understanding Ethnomethodology • We can observe other members methods of construction • Development of Conversation Analysis

  43. Confusing Ethnography and Ethnomethodology • Ethnography is a form of investigative fieldwork • Ethnography focuses on the “Member’s Point of View” • Ethnomethodology is a specific analytical technique

  44. Confusing Ethnography and Ethnomethodology • Confusion arises because: • Ethnomethodologist is likely to use ethnographic techniques • ‘Analytic mentality’-selection of phenomena and topics for investigation

  45. Ethnomethodology in HCI and CSCW • Observations of work activities and interactions help design process • Understanding temporal organization of activities and interactions and implications to design

  46. Learning from Ethnomethodologists • Division of Labour • Field Observation conducted by ethnomethodologists • Act as proxy for end users • Hand off requirements to computer science people

  47. Ethnomethodology for Critique and Design • Ethnomethodology has provided: • Critique of the design • Failure to support the work • Technology doesn’t allow people to engage in their work • Outlines organization of work and communication in the real world

  48. Two Paradoxes • Paradox of system design • Large scale activity • Paradox of technomethodology • Transformational nature of technology • Analysis of practice not invention

  49. Technomethodology • Develop a stance in which ethnomethodology and computer science play equally significant roles • Foundational relationships

  50. Technomethodology • Trying to exploit generalizations from ethnomethodology • Abstractions from both disciplines • Means by which such working practices arise • Dialogical interfaces

More Related