Thoughts on the Role of Surveys and Qualitative Methods in Evaluating Health IT

1. Thoughts on the Role of Surveys and Qualitative Methods in Evaluating Health IT National Resource Center for HIT 2005 AHRQ Annual Conference for Patient Safety and Health Information Technology

2. Starting with Surveys… • Design issues • comparison groups • Instrument development • Sampling methodology • probability of selection • size, response rate for significance *and* power • Survey administration • Data analysis

3. Instrument Development: Key Considerations • Internal validity • are you getting what you think you are getting? • Question ordering • easy to difficult • logic to flow and transitions • Keep it short • Pilot test!

4. Sampling Methodology:Key Considerations • Define your population, sampling frame • Random selection of participants • No ‘cherry picking’ (eliminate bias) • Everyone has a non-zero chance to be selected • Stratification and over sampling • Target response rate for a statistically meaningful result • Significance and power

5. Survey Administration:Key Considerations • Selecting your mode of administration • Self-administered questionnaire (SAQ) • Mail, Web • Telephone interviews • In-person • Tips on increasing response rate • Personalize initial outreach • Use the Dillman approach • Questionnaire format • Optimize follow-up: bang for buck

6. “When you can’t express it in numbers … your information is of an unsatisfactory kind” -- Lord Kelvin (1824 – 1907) So, why use surveys *and* qualitative methods?

7. … Because evaluating Health IT is a SOCIAL science

8. What is qualitative methodology? A non-numerical examination and interpretation of observations for the purpose of discovering underlying meanings and patterns. Qualitative methods provide a powerful tool for understanding the context surrounding quantitative data and for explaining phenomena that cannot be measured statistically due to a lack of validated instruments, sample size, or passage of time.

9. Using qualitative methods... • Qualitative methods offer highly valid results through an in-depth understanding of processes, attitudes, and effects. • Qualitative methods do not produce statistically significant results and are generally less reliable than quantitative results.

10. Qualitative Methods: Observation • Direct observation by individual(s) of activities, processes, or behaviors • Is implementation going as planned? • Validating self-reported survey results • Requires skilled observers ($) • Method of observation/data capture varies • E.g., Time and motion studies

11. Qualitative Methods: Semi-structured Interviews • Qualitative Interviews • Use discussion guide or list of themes • Not a static, closed-ended questionnaire • Key stakeholder perspectives • Detailed information, flexibility • Methodology varies • Sampling approach can be scientific • Content analysis on notes or transcripts

12. Qualitative Methods: Focus Groups • Moderated discussion with 10-12 respondents • discussion guides w. pre-selected themes • Improved validity • getting all “on the same page” • Can explain contradictory statistical results

13. Qualitative Methods: Case Studies • An in-depth study of a small set of sites. • Observation • Interviewing • Review and/or content analysis of background information, documents, communications, etc. • Informal conversations and encounters • Provides rich descriptions of activities and the influences of external factors on activities. • Highly valid, but not generalizable. • Case studies are NOT equivalent to program site visits.

14. Analyzing Qualitative Data • Process is variable • but for any study assumptions should be clearly described • General approach • Coding raw qualitative information • Nudist or ATLAS.ti • Identify themes and trends • Identify and describe outliers • Draw conclusions carefully and caveat • Analyses should occur both within and across individual data collections (e.g. within a single interview and across all interviews).

15. Recommended References • Coffey, A., and Atkinson, P. (1996). Making Sense of Qualitative Data: Complementary Research Strategies. Thousand Oaks, CA: Sage. • Edmunds, Holly. The Focus Group Research Handbook. McGraw Hill. 2000. • Hyman, Herbert H., with William J. Cobb, Jacob J. Feldman, Clyde W. Hart, and Charles Herbert Stember. Interviewing in Social Research. University of Chicago Press, 1954. • Krueger, A. Richard. Focus Groups: A Practical guide for Applied Research. Sage Publishing. 1994. • Miles, M.B, and Huberman, A.M. (1994). Qualitative Data Analysis, 2nd Ed., p. 10-12. Newbury Park, CA: Sage. • NSF “User Friendly Handbook for Mixed Method Evaluations.” 1997. • Payne, Stanley L. The Art of Asking Questions. Princeton University Press, 1951. • Turner, Charles F., and Elizabeth M. Martin, eds. Surveying Subjective Phenomena. Russell Sage, 1984.