New Media Research Methods

1. New Media Research Methods Part 2/3 – Qualitative vs Quantitative Data collection Case Study Example Gosia Kwiatkowska

2. Session 2 - Research methods • Recap • Qualitative and Quantitative Research Methods • Interviews and surveys • Designing questionnaires and interviews • Good practice

3. Recap • Variables • Research questions • The research process

4. Qualitative Quantitative • Qualitative - deep understanding of the experience. Not everything can be reduced to numbers. • E.G. Observation, ethnography • Quantitative – objective, measurable. Helps to establish patterns and relationships. • E.G. Survey

5. Qualitative Method? • User perspectives • Questions: how and why? • Captures perceptions, judgements, meanings, processes and reasons • Open ended questions, checklist of topics • Hypothesis and follow up questions are generated during data collection and analysis • Interviews allow to probe

6. Qualitative Method? • Strengths • Participatory • Rich, detailed data • Considers users perspectives and the context for their behaviours • Weaknesses • Hard work • Time consuming • Smaller sample of users • Not easily verifiable • Not easy to group your responses/categorise

7. Quantitative Methods • Designed to ensure objectivity, reliability and ability to generalise • Closed questions • Averages, percentages, ranges, means, frequencies • Can test statistical relationship between variable • Can prove whether or not a particular problem exists

8. Quantitative Methods • Strengths • Robust • Objective • Verifiable • Weaknesses • Out of context – human behaviour, real world settings are not considered • Any variables left out of data collection are not used in analysis

9. Quantitative or Qualitative or Both?

10. Interviews “Qualitative research [such as interviewing] attempts to understand meanings that people give to their deeds, experiences, or to other social phenomena” Silverman D (1997) ‘The purpose of interviewing is to find out what is in and on someone else’s mind. We find out from them those things which we cannot directly observe’ May T (1997)

11. Advantages • Rich data - excellent for complex subjects • Meanings / understandings / perceptions explored • Unforeseen issues / experiences elicited • Clarification / Follow up Q&As possible • Non-verbal communication / Observational opportunities

12. Disadvantages (1) Subjects may: • Conform to expectations – (social desirability) • try to be rational Interviewer may: • Be inconsistent • (Unwittingly) bias respondents’ answers • Mis-interpret answers

13. Disadvantages (2) • Difficult to get quantitative data • Data analysis difficult / complicated • Unrepresentative - reliance on key figures • Time consuming / costly • Low validity / reliability

14. Reliability and validity • Validity: "By validity, I mean truth: interpreted as the extent to which an account accurately represents the social phenomena to which it refers. " (Hammersley, 1992). Pg. 57. (e.g. does the data-gathering measure what you want it to measure?) • Reliability: the degree of consistency with which instances are assigned to the same category by different observers or by the same observer on different occasions". (Hammersley, 1990). Pg. 67 (e.g. does the data-gathering produce the same results if repeated?) .

15. But don’t forget … One case may be enough(!): ‘What is wrong with samples of one? Why should researchers have to apologise for them? Should Piaget apologise for studying his own children, a physicist for splitting only one atom?’ Mintzberg H (1973) The nature of managerial work NY: Harper & Row

16. Preparing the interview (1) Ethical considerations?

17. Preparing the interview (2) • Aims/objectives of the interview • Where these fit with overall research study • Interview style: • Structured  unstructured • Formal  informal • Open  closed • Choose physical setting (if possible) • Atmosphere / environment

18. Preparing the interview (3) Individual questions/areas/themes Fact • objective information (e.g., age, gender, education, behaviour, experience) Opinion / Preference / attitude / feelings • evaluative (e.g., satisfaction, agreement, like\dislike) Intended Behaviour • motivation or intention (e.g., likeliness, willingness)

19. Recording the interview To tape or not to tape?? Consider: • Effect on interviewee • Listening / transcribing time • Reliability of machine / recording But also… • Scribbling whilst listening • Making sense of notes

20. Conducting the interview • Thanks • Set the scene (why / how / where of study) • Give idea of question areas • Give ground rules (can refuse to answer/can terminate interview, anonymity etc.) • Start with demographic and ID questions • (usually) start with general question and then funnel

21. Types of question (1) Open • Opening stages in line of questioning (funnel) • Invites opinion, general knowledge. • Can cover areas where interviewer’s own knowledge lacking • No presumption about response Closed • Elicit hard facts • Control pace/direction of interview

22. Types of question (2) Probing • Extracts more depth • Maintains line of enquiry Leading • Confirm interviewee’s answer • Help interviewee, by rephrasing answer • Bring a line of questioning to an end (summarising)

23. Questionnaires To survey large number of people, to describe/explain characteristics/opinion of a population, usually through a representative sample. Questionnaires measure generalities / the extent to which groups of people behave or think in certain ways.

24. Characteristics of questionnaires • Cross-sectional • Mainly quantitative • ‘Snapshot’ in time • Qs and As standardised

25. Advantages • Reaches large / dispersed populations • Can generalise results • Provides quantitative, authoritative (?) data • Appears easy– work done by respondent • Relatively cheap and quick (per unit) • Removes personal influence • Respondent works in own time • Provides structure for report • Replicable

26. Limitations (1): Questionnaire construction • Low response rate (5-25%?) • Respondents may differ from non-respondents • No way to adapt add/remove questions • Little opportunity for respondents to explain • Can over-simplify issues • Respondents ‘shoehorned’

27. Limitations (2): Respondent issues Respondents may: • Take middle position in polar responses • Lack recall / rely on recent experience • Want to please researcher by being: • socially responsible • compliant

28. Question types (1) • Open questions • For further information (‘tell us more ‘) • For new information (‘what are your view on …’) • Multiple choice checklist • One answer only • All that apply (inc. Guttman scale) • Binary (yes/no) • Good for filtering • Rank order

29. Question types (2) • Likert scale • Agreement (strongly agree, agree, …) • Frequency (always, frequently, …) • Importance (very important, important …) • Quality (excellent, good, average …) • Likelihood (definitely, probably …) • Semantic differential Reactions to stimulus words / concepts in terms of ratings on ‘bipolar’ scale with contrasting adjectives at each end: • Excellent ………………terrible • Helpful …………………unhelpful

30. Surveys - Sampling

31. Process/stages • Formulate study aims • Identify objectives that address aims • Decide what information is required, and from whom • Decide sampling frame (elements making up population) • Research for similar questionnaires • Formulate appropriate questions • Decide distribution method • Postal; Telephone; Clip board/street; Web/Internet; Email; f2f • Pilot (see next slide) and get feedback • Amend • Distribute / administer

32. Good practice • Explain purpose • Catch interest early • Question sequence logical / helpful - broad to specific • Closed questions need "complete" set of response alternatives • Use appropriate language • Include clear instructions • Use clear tick boxes • Keep short

33. Increasing response rate • Name recipient if possible • SAE • Prize (?) • Offer copy of the report • Good design/construction • Engender involvement/interest • Chase non-respondents • Target ignored/committed groups

34. Consider • Question wording: do you like using the Internet and playing online games? • Questions order • Question types: leading, social pressure, presumptions, precision, memory, wish list/hypothetical questions

35. Case Study – Symbol Surfing Project Project team: Nick Weldin, Karen Bunning and Gosia Kwiatkowska Funded by Esmee Fairbairn Trust

36. Symbol Surfing Project - Background • Ubiquity of computer technology and the WWW – access and participation problems • Accessibility issues – keyboard/mouse • Match between the interface and the individual capacity of the user

37. Symbol Surfing Project - Background • The role of the user/the role of the supporter • Interactions: student/supporter; student/computer + supporter/interface/computer • Teacher initiating role to align the student – operational engagement

38. Symbol Surfing Project - Background • Communication – symbols, signs, pictures e.g. Makaton, Widgits etc • Use user experience (using symbols) but for different purpose – control the computer

39. Symbol Surfing Project - Background • Benefits – low cost, using existing skills, control, etc • Exploring the use of freely available symbol recognition software for PMLD community • Accessing and controlling computers by PMLD

40. Research questions • What are the ways in which individuals are able to use and engage with a simple symbol recognition system to access media on a computer? • What role is played by the supporter during user activity with this system?

41. Symbol Surfing Project - Method • Design – an action research, ethics, consent • Settings – specialised collage (2), secondary part of special school (2), residential setting (1). Supporters all knew the participants and understood their communication e.g. gestures, facial expressions etc • Sample – five people with PMLD, age range 15 – 28, difficult to engage, English – first language,

42. Symbol Surfing Project – Data Collection • Four visits, monthly intervals, • Symbol surfing software was installed • Two video cameras were used – one to capture the screen and the other to capture the engagement

43. Symbol Surfing Project – Data Collection • Environment – familiar • Researchers role – marginal participants – guiding the supporter if required • Field notes were recorded • Length of the sessions – varied

44. Symbol Surfing Project – Data Collection • The formats : model, withdraw support, observe • Number of cards used – varied • Video footage – combined and displayed side by side • Sampling of video – 60 sec at 2 min intervals • Data transcribed through repeated viewings 1 3 5 7 9 11 13 15 17

45. Symbol Surfing Project – Data Collection • Transcriptions template – actions participant/ supporter/ researcher • Actions: • vocal, • non vocal communication behaviours, • computer related actions, • contextual information

46. Symbol Surfing Project –Data Analysis • Coding framework – structural linguistics – to capture user attempts at manipulating the symbol tools, but also recording user responses to the activity and the facilitation role performed by the supporter • Initiating moves, response moves, pre-initiating moves and self expression

47. Symbol Surfing Project –Data Analysis • Refining the coding framework – viewing video and observing behaviours • Definition for each category with examples

48. Coding Framework Used