Introduction to Qualitative Research

Introduction to Qualitative Research Aaron Louie, User Experience Architect

PURPOSE • WHY QUALITATIVE RESEARCH? • RESEARCH DESIGN • VALIDITY & RELIABILITY • WRITING INTERVIEW QUESTIONS • ANALYZING DATA • RESOURCES

What this presentation will do • Lay the basic groundwork for gaining skills in user research • Provide useful guidelines for writing interview questions • Point you to resources for deeper study in qualitative research methods

What it won’t do • Substitute for a college course on the subject • Make you an expert

The purpose of qualitative research is…

…to make sense of reality…

…in a valid and reliable fashion.

Describing reality • Reality is complex. • All real problems are “wicked” problems. • Two ways to approach it: • Quantify it. Then construct mathematical models describing it. • Experience it qualitatively. Then make sense of it.

Controlled environment Experimentation Deductive Numbers Researcher is objective Natural/complex environments Interaction Inductive Words Researcher is a participant Quantitative vs. Qualitative

The Scientific Method Hypothesize Experiment Analyze Data Refine Hypothesis

Quantitative Research Hypothesize Experiment Analyze Data Refine Hypothesis Predict outcome & design experiment to test outcome Collect measurements in a controlled environment Apply statistical calculations Identify patterns, revise assumptions

Qualitative Research Hypothesize Experiment Analyze Data Refine Hypothesis Identify assumptions & formulate research questions Collect data via observation, interviews, surveys, etc. Count, classify, & structure data Identify themes, revise assumptions

Research Design • Determine scope • What you’re researching • What you’re not • Identify assumptions • Models from literature • Researcher’s biases • Draft research questions • Choose your methods • Data collection methods • Analysis methods

Framing Research Questions • List what you expect to find out, based on the scope and your assumptions. • For each finding, write a question (or series of questions) that must be answered to support or invalidate the finding. • Determine dependencies (which questions must be answered first).

Qualitative Data Collection Methods • Surveys • Pros: quantifiable, lower chance of affecting subject • Cons: no interaction with subject, nuances of emotion or environment lost • Naturalistic observation • Pros: gain understanding of environment without interfering with subject • Cons: very little interaction with subject, no insight into subject’s mental processes • Contextual inquiry (shadowing, on-site interviews, think-aloud, etc.) • Pros: balanced level of interaction with subject, insight into subject’s thoughts in context of tasks and environment • Cons: researcher’s physical presence can affect subject’s behavior • Ethnography (embedded research) • Pros: maximum level of interaction with subject, researcher gains deep understanding by becoming a subject, subjects treat researcher as peer • Cons: time- and cost-prohibitive

Analysis Methods • Statistical Analysis • Content Analysis • Narrative Analysis • Strategic Analysis • Workflow Analysis • Task Analysis • Business Analysis • Cognitive Work Analysis • Systems Analysis • Grounded Theory • etc.

Good research design • Maximize validity • Conclusions are true • Maximize reliability • Findings are repeatable • Relatively easy to do in quantitative research • Impossible to do in qualitative research

Threats to validity • Changes in environment • Changes in society • Effects of time • Changes in person • Bias in subject • Interaction between subjects (In other words, stuff you can’t control.)

Threats to validity(stuff you CAN control) • Bias in researcher • Interaction between researcher and subject

Validity vs. Understanding

Executive Upper management Supervisor How does the subject view you?

Composing an Interview Script • List what must be known in order to answer each research question • For each item in that list, write a question • Organize the questions into categories • Sort general  specific within each category

Phrasing Questions • Ask open questions for general questions • “what”, “why”, “when”, “how”, “who” • Ask closed questions for specific questions • “how often”, “how many” • Reserve yes/no questions for branching points in interview script (i.e., a line of questioning dependent on a previous answer) • “Have you ever done X? If so, how did you feel about…”

Refining Interview Questions • Remove biased or charged language • “What do you think about the problems with the intranet” • Avoid asking leading questions • “Wouldn’t it be better if you did it this way?” • Be aware of your own biases! • “How have consultants like Ascentium helped you get your job done better?” • Structure questions to facilitate data analysis

Conducting the Interview • Gain informed consent from the interviewee • Record audio/video whenever possible • Remain open-minded – don’t think too much • Listen, listen, listen • Avoid reacting too much (negatively or positively) to the interviewee’s responses • Take prodigious notes • Write up notes immediately after interview

Analyzing The Data • Go through notes & transcript • Identify frequent keywords • Categorize responses • Look for patterns • Match keywords, categories, and patterns to models and assumptions

Resources • Books • The Craft of Research by Wayne C. Booth, Joseph M. Williams, Gregory G. Colomb [link] • Qualitative Research Methods for the Social Sciences by Bruce L. Berg [link] • Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work by Kim J. Vicente [link] • Web sites • Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work by Kim J. Vicente [link] • Qualitative Research Consultants Association: FAQs about Qualitative Research [link] • Association for Information Systems: Qualitative Resarch in Information Systems [link] • UsabilityNet: Tools and methods [link] • UW Libraries: Lit Reviews, Research Methods… [link] • Virginia Tech: Annotated Bibliography on Qualitative Research Methodology [link]

Introduction to Qualitative Research