Integrating Quantitative and Qualitative Methods in Evaluation

1. Integrating Quantitative and Qualitative Methods in Evaluation Brad Cousins, Ph.D. University of Ottawa CaDEA Workshop Series, Session 4, Yaoundé, October 2010

2. Quantitative vs. Qualitative Methods

3. Method Mixing Choices • Parallel: quantitative and qualitative methods are used simultaneously • Sequential: one follows the other • Qualitative followed by quantitative: • Little known about domain of interest. • Helps to focus investigation, e.g., clarification of indicators and measures • Test for generalizability of findings • Quantitative followed by qualitative • Clarification of findings through further in-depth inquiry

4. Justifications • Triangulation: • Corroboration of findings • Complementarily • Added value • Initiation • Discovering new constructs Practice observation: • Evaluations fail to really capitalize on the power of multiple lines of evidence

5. A view of mixed methods inquiry (Greene, 2010) • The intentional, and connected or linked, use of more than one social science tradition, methodology, and/or method in service of better understanding • Tradition = philosophical paradigms and assumptions, logics of justification, privileged questions, ways of knowing • Examples: postpositivism, interpretivism, constructivism, feminisms, critical social science

6. A view of MM, continued • Methodology = inquiry logic, including questions, design, sampling, method choice, analysis, quality criteria, and defensible forms of writing • Examples: experimentation, survey research, ethnography, case study, narrative inquiry • Method = a technique or tool for data gathering • Examples: Ask ~ questionnaire, interview, assessment Watch ~ observation Find traces ~ unobtrusive measures

7. A view of MM, continued • A study is an MM study when there is some connection or linkage among the various methods and data sets at one or more stages of inquiry. • Connection in MM can be at any stage in the inquiry, but is generally most common in the interpretation stage

8. Mixing methods at data analysis stage • Data cleaning • Data reduction and description • Data transformation • Data correlation and comparison • Analyses for inquiry conclusions and inferences

9. Integrated analyses – a few principles • Connected to prior methodological decisions. • Highly iterative, spirit of adventure and discovery • Not every creative idea for interactive analyses will generate sensible or meaningful results. • Cognitive activity • Planned stopping point or decision point • Convergence, consistency, and corroboration are overrated: dissonance as stimulus for inquiry • Challenges to data quality and integrity can arise as a result of data transformation and framing

10. Data transformation, enabling joint analyses Data comparison and correlation, looking for patterns Major analyses, leading to inferences and conclusions Data transformation, one form to another (conversion) Data consolidation or merging, multiple data sets into one Data importation Extreme case analysis Integrated data display Warranted assertion analysis Pattern matching Integrated data display Integrated analysis, conceptual framework

11. Selected readings • Greene, J. (2010, September). Snapshots of integrated analyses in mixed methods analyses. Paper presented at the Canadian Evaluation Society, National Capital Chapter, Ottawa. • Greene, J. (2009). Meaningfully Engaging with Difference through Mixed Methods Educational Evaluation. In K. E. Ryan & J. B. Cousins (Eds.), Sage International Handbook of Educational Evaluation. Thousand Oaks: CA: Sage. • Greene, J. C., Benjamin, L., & Goodyear, L. (2001). The merits of mixing methods in evaluation. Evaluation, 7, 25-44.