cscw evaluation techniques n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
CSCW Evaluation Techniques PowerPoint Presentation
Download Presentation
CSCW Evaluation Techniques

Loading in 2 Seconds...

play fullscreen
1 / 54

CSCW Evaluation Techniques - PowerPoint PPT Presentation


  • 169 Views
  • Uploaded on

CSCW Evaluation Techniques . Presented by: Christopher Edwards. Overview of Presentation. Evaluation Techniques Understanding Ethnography Using Ethnography in CSCW Understanding Ethnomethodology Ethnomethodology and CSCW Technomethodology Conclusion. Evaluation Overview.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

CSCW Evaluation Techniques


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
    Presentation Transcript
    1. CSCW Evaluation Techniques Presented by: Christopher Edwards

    2. Overview of Presentation • Evaluation Techniques • Understanding Ethnography • Using Ethnography in CSCW • Understanding Ethnomethodology • Ethnomethodology and CSCW • Technomethodology • Conclusion

    3. Evaluation Overview • Olson and Olson. • What are we Evaluating? • Evaluations Techniques… • Internal/External Validity • Conclusion

    4. Characteristics of Groups • Individuals Differ in: • Skills • Ability • Knowledge • Personalities • Motivations • Agendas

    5. Characteristics of Organizations • System comprised of people and technology • Social Technology • Physical Technology • Comprised of multiple actors • Dependency on Communication • Information Processing Entities

    6. Characteristics of Task • Tasks involve different types of material • Physical, Digital or Ethereal • Ease or Difficulty of Task • Differ on Core Activity • Subtasks – Tightly Coupled/Loosely coupled

    7. Characteristics of Environment • Physical Environment • Distance between Group members • CSCW technologies designed to overcome • Contextual Time • When in the day the interaction occurs • Effects on Distant Group member

    8. Characteristics of Technology • Increasingly Varied • Technologies to Support Conversation • Auditory/Visual • Back channels/gestures • Technologies to Support Shared Work • Objects that support work • Fit of Tool to Material

    9. Process • Technology Deployment • How and Why • Process Analysis • Why outcomes were affected • Progress of Task • Communication process • Examined through time scales

    10. Outcomes • The initial outcome of using technology • Quality of work • Measure outcomes at every level • Group Outcome • Organizational Productivity

    11. Conceptual Framework for CSCW Studies Group Organization Process Outcomes Task Environment Technology

    12. Tools used to evaluate CSCW technologies

    13. The Survey • Set of questions • Fixed Alternatives • Statistically Analyzed • Wording of questions problematic

    14. Interview • Structure of interview • Formal and structured • Unstructured • Analysis can be complicated

    15. Experimental • Controlled Setting • Specific Task • Conditions • Assignment of Participants • Useful for making inferences about causality

    16. Case Study • Examines a single or small number of cases • Exploratory research

    17. Ethnography • Method adopted from Anthropology • Describing Culture • Used originally to describe other cultures • Misunderstood method

    18. Many other Methods • Diaries • Analytic Field Studies • Quasi Experimental • Longitudinal Studies • Historical Studies

    19. Internal and External Validity HIGH Laboratory Experiments Field Experiments Level of Internal validity Surveys Ethnographies LOW Level of External Validity HIGH

    20. Conclusion to the Overview • Different factors influence use and evaluation of CSCW software • Framework of CSCW studies • Evaluation Techniques • Validity of Techniques

    21. Short Break • Reconvene in 3 minutes • *Upcoming – Understanding Ethnography • Using Ethnography • Ethnography and CSCW

    22. Ethnography • Understanding Ethnography • Sociology Adoption • Using Ethnography • CSCW Ethnography in Design (Hughes)

    23. Understanding Ethnography • Ethnography is loosely applied to qualitative research • Home is originally from Anthropology • Aim to describe cultural interpretation

    24. Understanding Ethnography • Understanding culture “from an insiders point view” • Three sources of data • Participant Observation • Interviews • Collection of representative artifacts

    25. Sociology Adoption • Originally used to study distant cultures • Chicago School of Sociology • Studies focused on exploration of groups in urban settings • Cultural comparisons in USA • Family of Ethnographic Techniques

    26. Class Participation Time • In Pairs (Saul and myself included) • Everyone gets a Handout • For a total of 5 Minutes (2.5 Minutes each) • Each member of the pair (one at a time) asks the other questions from the sheet

    27. Debrief of Class participation • What answers were given…. • Obviously not a long term ethnography study • Depending on your relationship to this lab – differing perspectives • Understanding Grouplab culture (to some extent)

    28. Ethnography and CSCW • Prominence of Ethnography in CSCW • Insufficient attention to social context • New problems for design of collaborative character of work and activities • Ethnography and system design • Problem of scale • Pressure of time • Role of the ethnographer

    29. Concurrent Ethnography • Design is influenced by on-going ethnographic study • Sequenced process Ethnographic Study Systems Development Debriefing Meetings System Prototype

    30. Concurrent Ethnography in action • London Air Traffic Control Centre • Four week Ethnography • Each stage of fieldwork was intended to target designers issues • Small research team • What ethnography provided

    31. Quick and Dirty Ethnography • Brief Ethnographic Studies • Duration relative to the size of the task • Selecting aspects of work setting of importance to design Outline of project Meetings Short Focus Studies Debriefing Meetings Scoping Document

    32. Quick and Dirty Ethnography in Action • Ethnographic investigation of software engineers • Challenges of Large scale setting • Working in Industrialized Environments • Acceptance into the setting (*Key to Ethnographic research)

    33. Evaluative Ethnography • Ethnography used to verify formulated design decisions Initial outline Design or Specification Short Ethnographic study Debriefing Meetings Amended Design Or Specification

    34. Evaluative Ethnography in action • Fieldwork in Building Society • Using research for IT developments • Routine of work • Finding what customers wanted • Outlined limitations of model that had been proposed

    35. Re-examination of previous studies • Previous studies are re-examined to inform design • Ethnography used for many decades • Many studies related to work and occupation • Can be informative

    36. Re-examination in action • Inform preliminary design of Shared Object Service • Using previous Ethnographic studies on: • Social work, police work and invoice processing in a multi-site fast food company • What common service should support

    37. Summary of Ethnography • Understanding Ethnography • Ethnography and CSCW • Uses of Ethnography • Concurrent • Quick and Dirty • Evaluative • Re-examination

    38. Big Break Time 5 Minutes • Reconvene in 5 Minutes • Upcoming – Ethnomethodology • Understanding Ethnomethodology • Ethnomethodology in CSCW

    39. Ethnomethodology • Understanding Ethnomethodology • Confusing Ethnography and Ethnomethodology • Ethnomethodology in system design • Incorporation of Sociology and Computer Science? ‘Technomethodology’

    40. Understanding Ethnomethodology • Ethnomethodology literally means “People’s Methods” • A Shift from ‘other’ Sociological Methods • Social Life is potentially Chaotic • Social Actors • Members methods for making sense

    41. Understanding Ethnomethodology • Garfinkel “Documentary Method” • Example of Documentary Method • Garfinkel “Indexicality” • Disrupt Technique • Example in class

    42. Understanding Ethnomethodology • We can observe other members methods of construction • Development of Conversation Analysis

    43. Confusing Ethnography and Ethnomethodology • Ethnography is a form of investigative fieldwork • Ethnography focuses on the “Member’s Point of View” • Ethnomethodology is a specific analytical technique

    44. Confusing Ethnography and Ethnomethodology • Confusion arises because: • Ethnomethodologist is likely to use ethnographic techniques • ‘Analytic mentality’-selection of phenomena and topics for investigation

    45. Ethnomethodology in HCI and CSCW • Observations of work activities and interactions help design process • Understanding temporal organization of activities and interactions and implications to design

    46. Learning from Ethnomethodologists • Division of Labour • Field Observation conducted by ethnomethodologists • Act as proxy for end users • Hand off requirements to computer science people

    47. Ethnomethodology for Critique and Design • Ethnomethodology has provided: • Critique of the design • Failure to support the work • Technology doesn’t allow people to engage in their work • Outlines organization of work and communication in the real world

    48. Two Paradoxes • Paradox of system design • Large scale activity • Paradox of technomethodology • Transformational nature of technology • Analysis of practice not invention

    49. Technomethodology • Develop a stance in which ethnomethodology and computer science play equally significant roles • Foundational relationships

    50. Technomethodology • Trying to exploit generalizations from ethnomethodology • Abstractions from both disciplines • Means by which such working practices arise • Dialogical interfaces