Science is the belief in the ignorance of the experts

1. Science is the belief in the ignorance of the experts Richard Feynman, 1965 Nobel Laureate, Physics Mark Luborsky MCUAAAR workshop 5/12

2. QualitativeMixed Methods And Design Issues Mark R. Luborsky, PhD Institute of Gerontology Wayne State University, Detroit, MI • MICHIGAN CTR. FOR URBAN AFRICAN AMERICAN AGING RESEARCH Please do not duplicate or use these slides without the express permission of the author.

3. Mark Luborsky MCUAAAR workshop 5/12 m-m  M-M • “Social science is not a science… they have not yet found out anything, they haven’t got anywhere yet, maybe someday…” • “If the theory is right, what should happen, and has that happened needs study… in order to do that I must first understand how the answer probably looks… get a qualitative idea of the phenomena before I could get a good quantitative idea… so I have been working with the hope that in the future that rough understanding can be refined into quantified...” 1999, “The Pleasure of Findings Things Out”, Richard Feynman

4. Ascidians (“sea squirts”) are hermaphrodite marine invertebrate and exhibit traits of the earliest ancestors of chordates and vertebrates. … but with a difference

5. Topics and Take Aways m-m  M-M Mark Luborsky MCUAAAR workshop 5/12 Formulating best practices for M-M (vsbashing disciplines or any particular project or researcher) I. Concepts and considerations • Mixed discourse, multiple methods, Mixed Methods • Conceptual heart of project • Crafting the place of your basic scientific question in the body of knowledge, not the doing (eg aims as: examining, exploring, analysing) mechanical application of sets of procedures and methods • II. Design considerations • III. Succeeding in review and publication Your take aways: • Questions questionsquestions and concepts • Examples to think with • Pragmatic skills, vocabulary and guidelines

6. Mark Luborsky MCUAAAR workshop 5/12

7. Rich productive collaboration stands on firm disciplinary strengths and steps into the shared margins to discover new opportunities Mark Luborsky MCUAAAR workshop 5/12 Envisioning new horizons … the ideal….

8. Mark Luborsky MCUAAAR workshop 5/12 Multi-disciplinary Collaborative Research … closer to the reality

9. Is it true you can now get funding ….. • For a qualitative and mixed methods research project? . . . NO • More easily for epidemiology, physics, and clinical trials? . . . NO • A study of • an important question, • grounded in the scholarship on the topic, with a • full and clearly developed design and methods that fit the aims required to move us forward beyond what we already know. . . . YEStogoodscholarship Mark Luborsky MCUAAAR workshop 5/12

10. work to add to our body of knowledge about factors shaping health and effective human functioning in individuals and groups Mark Luborsky MCUAAAR workshop 5/12

11. Mark Luborsky MCUAAAR workshop 5/12

12. Secret Lives: Moral Aspects of Core Scientific Constructs for Design & Analyses • Power • Validity • Reliability • Sample/Sampling • Representation/representative • Control • Dependent variable • Independent variable The outside “second life” in daily society and talk connotations of these uses bring in enduring cultural dilemmas and challenges for scientists working to achieving adequacy for each of those constructs Luborsky, M. & R. Rubinstein. 1995 Sampling in Qualitative Research: Rationale, Issues, Methods Research on Aging 17(1):89-113 Luborsky, M. 1994 Identifying Themes and Patterns. In J. Gubrium, & A. Sankar (eds) Qualitative Methods in Aging Research. NY: SAGE Mark Luborsky MCUAAAR workshop 5/12

13. Mixed discourse, m-m, M-M Two examples of pervasive substantial confounds Identification of themes and patterns (Luborsky 1994) • Themes • Patterns • Focus group analysis (Agar & Hobbes 1996) • Themes • Negotiation of relationships and values • Consider: William Dressler, arterial hypertension in AA, Mark Luborsky MCUAAAR workshop 5/12

14. “Self rated health” powerful predictor of mortality in later life across the globe. But, what's this really about

15. code = “Physical Body Only” (Idler & Benyamini 2002) [Original] Eleven years ago I had a mastectomy. I’ve had a very bad case of high blood pressure for many years. For 15 years, I’ve had diabetes. I have a sciatic nerve problem, which causes my legs to hurt when I walk. The last doctor I went to wanted to do surgery to remove the nerves around my spine. I refused and said I’d come back when I couldn’t walk at all. He said if I didn’t want surgery, he couldn’t help me. Mark Luborsky MCUAAAR workshop 5/12

16. Mark Luborsky MCUAAAR workshop 5/12

17. Mark Luborsky MCUAAAR workshop 5/12

18. What is the all noise about qualitative and mixed methods? DNA mapping • Socio-cultural distribution of • forms of knowledge construction, and • of disciplinary ‘ethnic’ tension about it Mark Luborsky MCUAAAR workshop 5/12

19. Concepts and Considerations Mark Luborsky MCUAAAR workshop 5/12 • History not new. Stouffer et al. “The American Soldier” 1940s (Likert, etc) • Social science M-M is affliction not in other fields inquiry/discovery • Compatibility and contentions between paradigms within a field, tensions between discipline tensions (NIA/NINR review). • Motivation – trendiness will not carry your. Establish scientific need, acknolwedge plus and minuses and address • Just because you can: quantify qualitative, or qualitative describe experience -- doesn’t mean it will adequately capture the construct or provide an real scientific impact

20. Is your project or article ready for review? Mark Luborsky MCUAAAR workshop 5/12

21. General Considerations • No one preferred. Help reviewers see multiplicity of approaches • Some are more palatable to the NIH’s positivistic quantitative approach • Review panels less biased against a method than incompletely specified proposals • NIH is a public health agency: it is accountable for improving public health • Researcher bears burden of proof regarding accountability and usefulness of methods Mark Luborsky MCUAAAR workshop 5/12

22. What criteria do I use as editor of MAQ and NIH reviewer • I go to the “methods” section to see if quantitative and qualitative data are collected. • I look in the “analyses” “results” and “discussion” sections for some connection, merging, or embedding of one type of data in the other. • I look in the “methods” discussion for rigorous quantitative and qualitative procedures. • I look throughout the manuscript for how the author has positioned the study within the field of mixed methods research and how the study contributes a thoughtful, unique contribution to mixed method literature. • I look in the “background and significance” and throughout the manuscript for the paradigm stance of the author. • I look at the references to see if the author cites and is familiar with the mixed methods literature. • I look in the “introduction” for the rationale for using mixed methods to study the research problem. Mark Luborsky MCUAAAR workshop 5/12

23. A few journals dedicated to mixed methods … Journal of Mixed Methods Field Methods Quality and Quantity International Journal of Multiple Research Approaches (on-line) Other journals that publish mixed methods studies (e.g., International Journal of Social Research Methodology) Special issue journals Mark Luborsky MCUAAAR workshop 5/12

24. Practice Targets for Qualitative M-M Applications to NIH • INTERNAL CONSISTENCY!! Across problem, aims, literature, design, methods • Clarity (simple), specificity, and justification for methods are key • “Watch your language” especially shared terms • Attend to sampling, sample collection and maintenance • Design. Define, then explore strengths and limits • Mixed methods. How are they combined in process and final product • Budget for expertise, resources, time: add if needed • Anticipate questions and answer them in writing Mark Luborsky MCUAAAR workshop 5/12

25. 14 Common Limitations in Proposals With Qualitative Methods • Aims not specific, err by stating procedures (e.g conduct ethnography, do interviews, observe…) • No compelling answer to the “So what” question, need to address the significance of proposed findings • Adversarial and dogmaticvs informed and reasoned discussion about choice for this particular topic and study • Significance too general, lacks detail on specific factors and topics. Over-developed “Background” relevance • State of knowledge used to structure proposal biased by only covering studies using your methods • Methods not well justified: missing pieces, incomplete descriptions - especially “measures” • Methods are internally inconsistent Mark Luborsky MCUAAAR workshop 5/12

26. 14 Common Limitations (continued) • Limitations of procedures not address (sample, data collection, analyses) • Sampling decision not explained with sound rationale • Data collection procedures not explained in adequate detail (when, where, how, by who) and access to population. • Data analyses superficially described • Fail to demonstrate expertise in methods, including technology in using software • Mixed methods fail to adequately link research aims, questions, data, analyses • Unequal attention to qualitative and quantitative methods when both used Mark Luborsky MCUAAAR workshop 5/12

27. Imiganiy’ikinyarwanda / “hardship and opportunities” In season, the lake’s bounty for the fish and edges shrink, once hungry crocodiles enjoy plenty, until the next rains, when lakes grow and again they are hungry while others have plenty Kofi Kankolongo Mark Luborsky MCUAAAR workshop 5/12