Developing and evolving Research Programmes

1. Developing and evolving Research Programmes Professor Gilbert Cockton november 5th 2013, tutorem training school,bled

2. Outline • 21 years of PhD supervision and examination • Developed 4 Research Programmes • Evolved 4 Research Programmes • Workshop • A survey of design managers reveals that projects have mixed experiences when using personas. • Brainstorm to outline a research programme to find out why this variation occurs.

3. PhD supervision & examination • Supervision role for 32 research students since 1993 in computing, art, and design • Examination role in 40 research degrees since 1992 in 7 countries for computing, engineering, design, psychology and forestry science (!) • Key issues for PhDs • Clear research focus • Clear and well-defended claims • Clear, appropriate and credible methodology • Clear, rigorous self-critical practical work and analysis • A thesis, not a chronicle: smooth argument, strong oral defence, not a diary of set backs and disappointments

4. The Research hierarchy

5. PhilosophyProgramme methodology studymethod

6. RESEARCH PROGRAMMES • A research programme is a managed set of projects, including PhDs • Some in sequence, some overlap • Each project involves one or more studies • Each project has a coherent focus, but not necessarily fixed • Projects evolve, programmes evolve • A research programme spans years, often several or more • Typically lead by experienced researchers • Interactiondesign.org chapters, clear evidence of programmes • May involve formal or informal collaborations • Worth-centred design: Microsoft, Finnish VALUTEKES project • User experience evaluation: MAUSE and TwinTide COST projects • A PhD is one project in an (in)formal programme • If your PhD project feels like a programme, shrink it now! Programme methodology studymethod

7. EXAMPLE HCI Programme (GROUP)s • Cognitive Engineering • Co-Design • Pre-attentive Visual Aesthetics • InteractionistAffective Computing • Sustainable HCI • Critical-Empirical HCI • Value-Sensitive Design • Universal Design • Approaches and Resources in Design Work • Ambient Intelligence • Information Visualisation Programme methodology studymethod

8. RESEARCH methodologies • A research methodology can be • a co-ordinated set of research studies • your (evolving) research plan • A coherent set of research philosophies and practices • Chosen research approach, for project or programme • The study of the former (as in biology, geology, ethology, narratology) • I will focus on 1 and 2 • Methodology today means your (evolving) research planfor a co-ordinated set of research studies • Methodologies must be appropriate for research programmes • If you’re not part of a programme, align with someone else’s informally • Research is a contribution to a body of knowledge and of practice Research philosophies refers to the second meaning of methodology Programmemethodologystudymethod

9. Studies and methods • A research study applies one or more research methods • You do not use or follow a method, or even apply it without very careful planning first • You make the method by following a study plan that takes text book accounts of a research method and turns this into concrete practices • Research methods are techniques that include • Guidance on values, best practices, materials and data records • Your own careful planning to ensure that the method as applied plays its proper role in your methodology, and can contribute successfully to the embracing research programme • Studies have to be designed because research • is not about rule following – there are no method guarantees • is about candid self-critical reflection and persuasive practices • involves creative design and individual skills, knowledge and expertise Programmemethodologystudymethod

10. Research Realities

11. We can’t tell you how to … • Choose research programmes • yours was probably chosen for you (if not, choose one quick) • Design your research methodology • that’s too project specific, there’s a lot to take into account here • Design a study • because ‘good‘ design here depends on the embracing programme and methodology • Complete a research method so that it’s correct for your study • because that depends on specific study goals • But we can introduce you to • General principles, attitudes and values for a range of research approaches • Specific tactics, best practices and pitfalls to avoid that will increase your chance of not falling at the last fence • Each instructor’s hard won experience • Looking forward to hearing them all share this Programme methodology studymethod

12. www.flickr.com/photos/travelinio_com/4218547394/sizes/l/in/photostream/www.flickr.com/photos/travelinio_com/4218547394/sizes/l/in/photostream/

13. Methodology LOOKS BEST WHEN

14. http://www.flickr.com/photos/jcuthrell/63028482/sizes/l/in/photostream/http://www.flickr.com/photos/jcuthrell/63028482/sizes/l/in/photostream/

15. Don’t just learn rules: Think! • Can I use this research method? • What relevant studies for my research could it support? • How would I get it to work for a relevant study? • What would I have to do? What would my study plan be? • Do I have the required resources? What would I need, and when? • Can I get them? • What would success look like if I used this research method? • because that depends on specific study goals When I explain what I’m going to do/did, then … • How confident am I? • How well can I defend what I’ve done? • What questions could I have to answer? Programmemethodologystudymethod

16. I’m NOT DOING RULES

17. I’m DOING landscapes

18. Questions?

19. Philosophies and programmes in action

20. RESEARCH Philosophies • Research programmes align themselves to one or more philosophies • Research philosophies are characterised by: • Epistemology – positions on the nature of truth • Ontology – positions on the nature of reality • Axiology – positions on what is important • Axiology dominates • Five research philosophies are commonly distinguished • Positivist • Interpretivist • Rationalist • Critical • Action, including Research through Design • All have their strong and weak points, no one is best • Your disciplinary context can fix your research philosophy Philosophies programmes in action

21. POSITIVIST RESEARCH • Realist ontologies – there is one unchanging world that can be revealed through rigorous systematic practices • Objectivity is possible and subjectivity must be avoided at all costs • Values evidenced based knowledge that we should all rationally accept • Language challenges can all be managed via precise definition • Reliable value-free knowledge requires verification of theories via controlled studies • Predictive knowledge is highly valued, but accurate description is valued too • Explanation typically takes the form of validated predicted theories • Hypothetico-deductive methods, theories logically yield testable hypotheses • Careful design results in convincing experiments where alternate hypothesis must be accepted under specific conditions, null hypothesis accepted otherwise • The facts do not speak for themselves, rather they only have force within specific experimental contexts, and no evidence can rescue a poorly designed experiment Scientism is an extreme form of positivism where ONLY positivist knowledge is valued • Achilles heels of induction (Popper) and argument for experimental designs (Quine) Philosophies programmes in action

22. INTERPRETIVIST RESEARCH • Phenomenological ontologies – we all have unique perspectives on the world, and experience it through individual perspectives, values and conceptual schemata • Objectivity is controversial: inter-subjectivity must be accepted as the best achievable • Language is a resource, not an enemy of positive truths • Valuable knowledge requires critical reflection on collection and analysis of evidence • Predictive knowledge is often unattainable, but accurate description and transparent analysis are highly valued – no tampering with data, no hiding analytical steps • Explanation typically takes the form of well grounded themes in data • Inductive methods • Theories emerge from analysis of data, they do not precede them as systems of conjecture from which testable hypotheses can arise • Inter-rater reliability of coding and theoretical saturation can be empirically grounded Interpretivism can range from strong empiricism to relativism • Strong claims of ethnomethodology, claims for universality weaken once critical perspectives are embraced Philosophies programmes in action

23. RATIONALIST FORMAL RESEARCH • Logic without the positivism • Objectivity is replaced by rigour, (formal) language is the main intellectual resource • Language is formalised, and thus all ambiguity is removed • Analytical philosophy: necessary and sufficient reasons for the use of a term • Formal (mathematical) methods: argument is replace by proof • Formal specifications can be analysed, and principles articulated • Modernist aesthetics, focus on how content and structure make user interfaces work Formal methods need to be well directed • Formal analyses only reveal what the analyst can recognise • Poor analyses lack resonance and expose what was already obvious, but only after extensive elaboration, true but … Philosophies programmes in action

24. CRITICAL RESEARCH • Ideological ontologies – we receive perspectives on the world from those in power • Objectivity is an instrument of repression, as is common sense and realism • Valuable knowledge results from systematic distrust of categories and discourses • Liberating and empowering knowledge is attainable through the adoption of critical perspectives (philosophical criticism, Postcolonialism, Feminism, Queering) • Resonance, insight and revelation are valued, new perspective and paradigms • Critical analysis methods • Theory is unavoidable, and is embedded in all of our concepts and categories • Explicit theory is preferable to implicit theory • Effective critics deploy a range of theoretical perspectives in their criticism Critical analysis can range from genius to boring mechanical insensitive posturing • Critical perspectives can significantly reframe research thinking, but they can also trap analysis in unproductive dead ends • Not always heavy on theory, e.g., ordinary language analysis methods of analytical philosophy, but this moves back towards a rationalist philosophy Philosophies programmes in action

25. ACTION RESEARCH • Pragmatic ontologies – the world is how we make it • Objectivity on paper is less important than effectiveness in the world • Valuable knowledge results from practical engaged committed reflective action • Truly practical knowledge can only result from real world engagement • Collaborative learning and development are valued, as are new local practices and understandings, rather than what is universally and externally true • Action research methods • Bias and subjectivity are accepted, but are subject to reflective critique • Research practices are constantly monitored, evaluated and revised, rather than being rigorously planned and preserved unchanged at all costs Action research can range from outstanding innovation to routine work • Stakeholders are the primary evaluators of action research. Interventions need to succeed for engaged stakeholders. Value takes precedence over academic rigour. • Research through design, co-design, Engineering Design Innovation Philosophies programmes in action

26. Research through Design • Using creative design practice as the basis for knowledge about the world • Koskinen, Binder, Wensveen, Zimmerman, Folizzi, Stolterman, Gaver, … • Constructive Design Research • You can’t preplan creativity • Dealing with Wicked Problems (Rittel and Weber, Conklin) • Rigour is achieved through documentation (Gaver and Bowers, Workbooks) and critical reflection (Schön) • Compare discussion sections of scientific papers (plea bargaining?) • Reflection at the end of phases or stages of activities • Phases or stages typically include more than one activity • Primary research, secondary research, analysis, problem scoping, design generation, evaluation • Activities need to co-ordinated, balanced and integrated Philosophies programmes in action

27. HCI: A Community of communities • Carroll – Encyclopedia of Interaction Design, Three Eras • Multiple communities, multiple philosophies • 1980s Cognitive Engineering – positivist lab. research dominant (infoprocessing) • 1990s Contextual Ethnography – interpretivist field research dominant (agents) • 2000s Critical Interaction Design – humanities and applied arts influences (social and material embedding , Dourish, Where the Action is, 2001) Action Research in all 3 Waves of HCI • 1980s Usability Engineering – user-centred interventions in systems development • 1990s Participative Design – contextually focused co-design practices • 2000s Design Activism – politically motivated community initiatives HCI Research can and does mix all four research philosophies via separated and integrated practices during different stages of research Philosophies programmes in action

28. Questions?

29. Research philosophies in action

30. interactiondesign.orgencyclopedia • Löwgren (1) • Critical and Action Research • Caroll (2) • Examples of all research philosophies, strong emphasis on action research • Höök (12) • Interpretivist, critical and action research • Tractinsky (19) • Positivist, Barzell’s critical response, aided by Tractinsky’s language • Dix (29) • Rationalist (Formal Methods) • Cockton (19) - ? Philosophies programmes in action

31. Five EXAMPLES OF PhD WORK • All references should be available from/via my academia.edu pages • If not, let me know and I’ll add them • DarrynLavery (1993-2000), Glasgow, Computer Science • Critical interpretivist research • Alan Woolrych MPhil 2001, Sunderland, Computing • Interpretivist research • Alan Woolrych (PhD 2012) and Mark Hindmarch, Sunderland • Positivist research • Eamon Doherty (PhD 2001), Sunderland, Computing • Action research • Michael Leitner (PhD write up), Northumbria, Design • Research through Design Philosophies programmes in action

32. CRITICAL RESEARCH: darrynlavery • Lavery, D. Cockton, G. and Atkinson, M.P., "Comparison of Evaluation Methods Using Structured Usability Problem Reports," inBehaviour and Information Technology, 16(4), 246-266. 1997. • Lavery, D. and Cockton, G., “Representing Predicted and Actual Usability Problems”, inProc. Int. Workshop on Representations in Interactive Software Development, QMW London, 97-108, 1997. • Critique of constructs and research methodologies for inspection method development and evaluation • Derived new problem report formats for usability problems • Developed new analysis methods for extracting usability problems from user testing data Philosophies programmes in action

33. Interpretivist RESEARCH: AlAnWoolrych • Cockton, G. and Woolrych, A., “Understanding Inspection Methods: Lessons from an Assessment of Heuristic Evaluation,” in People and Computers XV, eds. A. Blandfordet al., Springer-Verlag, 171-192, 2001, • Application of Lavery’s methodological innovations to Heuristic Evaluation • New explanatory contructs • Discoverability (inferential statistics applied) • Discovery and analysis resources(pilot analysis) Philosophies programmes in action

34. Positivist RESEARCH: Alan Woolrych & Mark HinDMARCH • Cockton, G., Woolrych, A., Hall, L. & Hindmarch, M., “Changing Analysts' Tunes: The Surprising Impact of a New Instrument for Usability Inspection Method Assessment,” in Palanque, P. et al. People and Computers XVII, Springer-Verlag, 145-162, 2003. • Cockton, G., Woolrych, A., and Hindmarch, M., “Reconditioned Merchandise: Extended Structured Report Formats in Usability Inspection”, in CHI 2004 Extended Abstracts, ACM, 1433-36, 2004. • Extension of Lavery’susability report format to expose separate discovery and analysis resources • Doubled evaluation quality on validity and appropriateness • Replicated in deliberate planned hypothesis testing study Philosophies programmes in action

35. ACTION RESEARCH: EAMon DOHERTY • Doherty E.P, Cockton G., Bloor C. & Benigno, D., "Mixing Oil and Water: Transcending Method Boundaries in Assistive Technology for Traumatic Brain Injury," inProc. ACM 1st Conf. on Universal Usability, eds. J. Sholtz and J. Thomas, ACM, 110-117, 2000. • Doherty E.P, Cockton G., Bloor C. & Benigno, D., “Improving the Performance of the Cyberlink Mental Interface with the Yes/No Program,” inProc.CHI2001, ACM, 69-76, 2001. • Co-Design of Brain-Body Interfaces • Different designs for different stakeholders • Compromise design for both • Diagnosis of one participant changed from comatose to Persistent Vegetative State Philosophies programmes in action

36. Research through design: Michael leitner • M. Leitner, M., Cockton, G., Yee, J. and Greenough, T. 2012. The Hankie Probe: a Materialistic Approach to Mobile UX research, in CHI 2012 Extended Abstracts. ACM, 1919-1924.. • Leitner, M., Cockton, G. and Yee, J.S.R. 2013. At the mobile experience flicks: making short films to make sense for mobile interaction design. In Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services (MobileHCI '13). ACM, 304-307. Honourable Mention Award. • Materialising Theory in Probes, Visualising Insights in Film • Two theories of mobility inscribe in hankie probes • Interviews with completed probes • Workshop packs and films for design teams to communicate results of probe usage Philosophies programmes in action

37. Hankie Probe WorkshopsMichael Leitner, Northumbria University, Mobile HCI 2013 • In design sessions the hankie is presented together with additional and focused data, introducing the couples' or the mobile workers' everyday practices and experiences with mobile communication technologies • (1) a short summary of the scenario • (2) a short description of the couple or the person • (3) an annotated version of the hankie highlighting the relevant parts for the scenario • (4) selected quotes taken from the interview • (5) an abstracted and theoretical version of the scenario Philosophies programmes in action

38. SUMMARY • Methodologies, studies and methods all need to be designed • We can’t tell you how to design your PhD research, as that’s a creative activity that draws on individual skill, knowledge and expertise • We can present solid techniques and knowledge associated with them, and explain why some practices are valued and others are not • Values have their roots in research philosophies • The right philosophy in the right place results in good research • The wrong philosophy in the wrong place results in poor research, even when the technical execution is flawless • ‘Correct’ use of a method cannot compensate for flaws in study or methodology design Philosophies programmes in action Programme methodology studymethod

39. Questions?

40. EVOLVINGResearch Programmes

41. RECAP • We can’t tell you how or where to start, but we can keep you going Research programme progresses Methodology contributes to Study supports Method Philosophies programmesin action Programme methodology studymethod

42. WHY Methodologies EVOLVE • Methodologies evolve in the course of a research project for a range of reasons, for example, because … • they can (ethics permitting) • it may not be possible to get around the limits of methods in completed studies, so a different mixed method approach is needed • of paradigm shifts (e.g, from usability method comparison to resource effect studies) • of the task artefact cycle in HCI, new findings change the nature of the research, perhaps invalidating previous studies, through reflection • of insights from pilot studies or replications • of the need for fine tuning, or not so fine oh dear it’s broken mending • Other … Philosophies programmesin action

43. Research evolutionin action

44. Evolving from replication • Gnanayutham, P., Bloor C., & Cockton G., “Discrete Acceleration and Personalised Tiling as Brain‑Body Interface Paradigms For Neurorehabiliation,” inProc. CHI 2005, 261-70, ACM, 2005. • Cassidy, B., Cockton, G., Bloor, C., and Coventry, L., “Capability, Acceptability and Aspiration for: collecting accessibility data with prototypes,” inProc. HCI 2005, Volume 2, 138-43, 7, 2005. • Replication/extension of previous work by predecessor PhD/project sponsor • Brain-Body Interfaces • Tab-select device for ATMs (cash machines) Philosophies programmesin action

45. http://www.freefoto.com/preview/1042-05-23?ffid=1042-05-23 Eamon Doherty, PhD 2001 Paul Gnanayutham, PhD 2007

46. Evolving from pilot studies • Woolrych, A., Cockton, G. and Hindmarch, M., “Knowledge Resources in Usability Inspection,” in Proceedings of HCI 2005, Volume 2, eds. L. Mackinnon, O. Bertelsen and N. Bryan-Kinns, 15-20, 2005. • Analysis of data from HCI 2003 and CHI 2004 studies (pilot and replication) • Developed into COST MAUSE project resource function theory for design and evaluation methods • Woolrych, A.Hornbæk, K. Frøkjær, E. and Cockton, G.. Ingredients and Meals Rather Than Recipes: a Proposal for Research that Does Not Treat Usability Evaluation Methods as Indivisible Wholes, IJHCI, 27(10), 940-970, 2011. Philosophies programmesin action

47. Evolving from critique • Lavery, D. Cockton, G. and Atkinson, M.P., "Comparison of Evaluation Methods Using Structured Usability Problem Reports," Behaviour and Information Technology, 16(4), 246-266. 1997 • Cockton, G. and Lavery, D. “A Framework for Usability Problem Extraction”, inProc. INTERACT 99, eds. A. Sasse& C. Johnson, 347-55, 1999. • Critique of constructs and research methodologies for inspection method development and evaluation • Derived new problem report formats for usability problems • Built on by Alan Woolrych and Mark Hindmarch • Inspired MAUSE COST Project CODE-LIGHTS study, and provided a critical lens for MAUSE Working Group 2 • Developed new analysis methods for extracting usability problems from user testing data • Improved on by Arnold Vermeeren, PhD Delft 2009 Philosophies programmesin action

48. Evolving from action RESEARCH • Cockton, G. Kujala, S., Nurkka, P. and Hölttä, T., Supporting Worth Mapping with Sentence Completion in Proceedings of INTERACT 2009, Part II, (LNCS 5727) eds. Gross, T.; Gulliksen, J.; Kotzé, P.; Oestreicher, L.; Palanque, P.; Prates, R.O.; and Winckler, M, Springer, 566-581, 2009. • Cockton, G. Kirk, D., Sellen, A. and Banks, R., Evolving and Augmenting Worth Mapping for Family Archives in Proceedings of HCI 2009 – People and Computers XXIII, 329-338, BCSeWIC, 2009 • Action Research Projects at MSR Cambridge and within Finnish TEKESVALU project • Evolved worth map formats into simpler versions • Augmented by new practices • Value-focused field data analysis • Sentence completion for value elicitation • User experience frames provide details for experience elements Philosophies programmesin action

49. OF5 Stronger sense of family past OF6 Stewardship obligations discharged OH3 Living Family Heritage: a past you want to revisit OH2 Nurturing: somewhere you want to be OF2 Increased Family Empathy OF3 New Shared Times as a family OT1 Treasures sold or passed on OF7 Stronger roots in the past X1 Reliving (shared) memories OF8 Achievement of closure OT2 Protected Heirlooms X13 Telling my/our story X2 Sharing stories and memories X9 Having fun, playing around X4 Preserving heritage, exercising stewardship X8 Gaining control, making progress X3 Reflecting, taking stock, moving on X5 Being a family, caring & nurturing QA1 Safe, protected, savable QI1 Playful, Fun QT3 Self-explanatory, guiding, suggestive, familiar, intuitive, supportive QA2 Enriched, enhanced, augmented QA4 Respectful, empathic QI2 Doing things together QT2 Inviting QT4 Capable, comprehensive, versatile, inclusive QT1 Accessible, at hand suggesting casual, efficient, calm, easy capture in use QA5 Keeps secrets PRO9 Subtle reminders, safe originals PRO8 Support for Triage PRO1 Moving stuff between boxes PRO5 Edit, Associate, Loose Tag, annotate CAP2 Personal area, access control PRO7 Assets Shared, Individual Curation PRO2 Auto Format Updating PRO3 Rummaging CAP3 Functional object ‘ghosts’ PRO6 Automatic Voice Annotation CAP6 Family Member Identification MN2 WAN back up MT2 Table Form MT1 Multitouch Thinsight, IR, Tagged props MIO5 Microphone MIO6 TBD h/w & s/w for family member ID MT4 OBEX/Bluetooth detection, data transfer MT6 Detachable Camera Worth Sketching and Mapping • Worth as net benefits (benefits – costs) • Connects artefacts (means) to purposes (ends) through experiences

50. VALUE-FOCUSED FIELD DATA ANALYSIS