Analyzing & evaluating qualitative data

1. Analyzing & evaluating qualitative data Kim McDonough Northern Arizona University

2. Epistemology in qualitative research • Theory of knowledge • Constructionist—Meaning/knowledge vary across people and over time • Holistic—complex topic is viewed from multiple perspectives • Subjectivist–Knowledge/meaning is personal • Context-based—Knowledge is negotiated socially and historically

3. Search for knowledge • Truth is relative & context specific • Goal is interpret how people construct meaning • Start with experiences and build theory inductively • Data provides information that must be interpreted • Interpretation is shaped by the researcher’s own experiences

4. “Good” qualitative research • Maintains a single focus/idea/problem • Uses rigorous data collection • Sufficient time in the field • Many sources from multiple perspectives • Detailed summary of each source • Employs rigorous data analysis • Multiple levels of abstraction • Verification of the accuracy of the findings • Engaging report • Clear, detailed writing • Accurately reflexes complexity of the context

5. Appropriate topics for qualitative research • The research question asks how or what • Focus on understanding • The topic needs to be explored • Theory doesn’t exist yet, is insufficient, or hasn’t been tested in a particular context • The small pieces that make up the big picture haven’t been identified yet • Time & resources are available • A receptive audience exists

6. The natural setting is the focus of inquiry • Obtain information from participants in their natural environment • Reveal contextualized patterns/truths • The researcher’s interests • Take a personal role in the project • Serve as a voice for the participants

7. Basic tenets of qualitative analysis • The search for patterns in data and ideas that help explain the existence of those patterns • The goal • Reduce huge amounts of text to manageable units for further analysis • Interpret the contribution of those manageable units to existing knowledge or practice

8. Steps in data analysis • Preliminary steps: Organize the data • Organize the data • Label/identify source of the data • Convert to appropriate text units if necessary • Enter any numeric information into spreadsheet • Determine which data sources are available for each participant • Make decisions about inclusion or exclusion criteria based on completeness of data

9. Form general impressions • Read all the data multiple times • Do not analyze each data source separately • Do not prioritize one source of data over another • Get a sense of the whole before trying to break it into parts • Make notes—short phrases, ideas, or key concepts

10. Analysis: Form initial categories • Search for categories, themes, or dimensions • Identify & name the major themes • Describe each of the themes • Check descriptions to refine overlapping categories

11. Classify data segments by themes • Read the entire data set again • Identify segments in all sources that belong to each theme • Keep track of segments that don’t fit with a theme

12. Evaluation: Assess themes & segment assignment • Evaluate whether the themes are appropriate in light of all the segments • Decide if all the segments fit with an existing theme • Rename/combine/separate themes if necessary • Create new themes if necessary

13. Interpretation • Make sense of the patterns in the data • Step back from the summarizing the data and find links to larger meaning • Based on insights/intuition/hunches • Based on an existing construct, idea, theory, practice • Based on similarity/divergence from previous research findings

14. Verifying accuracy in qualitative data analysis • Prolonged engagement/persistent observation/time on site • Building trust with participants, learning the culture, checking misinformation and distortions, time on site • Triangulation/Diversity of method • Use multiple sources, methods, investigators • Elicit multiple perspectives • Identify corroborating evidence from multiple sources

15. Clarifying researcher bias • State past experiences, biases, prejudices & orientations that may have shaped the inquiry & interpretation • Peer review/debriefing • An external check of the research process • “The devil’s advocate”

16. Member checks • Solicit the informants’ views about the credibility of the findings • External audits • Someone with no connection to the study • Allow an external auditor to examine the process and product of analysis • Determines whether the findings, interpretations, conclusion are supported by the data

17. Rich, thick description • Describe in detail and participants and the setting • Allows readers to make decisions about transferability • Negative case analysis • Refines working hypotheses as data analysis/interpretation unfolds • Revise hypotheses until all cases fit • Account for outliers and exceptions

18. Deciding which techniques to use • Use at least two (Creswell) • Easiest, cost-efficient, most popular procedures • Triangulation • Rich, thick description • Member check

19. Discussion • Procedures you have used for verifying evaluation • Successful ones? • Challenges or concerns? • Current qualitative projects? • Suggestions for data sources & perspectives?