Session A PROCESS EVALUATION: TOOLS FOR BEGINNERS MPR Cindy Gruman Su Liu University of Connecticut Health Center

1. 1

2. 2 Why Conduct a Process Evaluation? Understand, verify or increase the impact of the program, policy, or initiative Improve delivery mechanisms to be more efficient and less costly Verify that you are doing what you think you are doing It really does lead to policy change CMS is encouraging states to document and understand outcomes

3. 3 Process Evaluation Tools Key Informant Interviews Focus Groups

4. 4 Quantitative vs. Qualitative Data Quantitative research involves analysis of numerical data. Qualitative research involves analysis of data such as words (e.g., from interviews), pictures (e.g., video), or objects (e.g., an artifact). The strengths and weaknesses of qualitative and quantitative research are a perennial, hot debate, especially in the social sciences.� The issues invoke classic 'paradigm war'. Both quantitative and qualitative research rest on rich and varied traditions that come from multiple disciplines and both have been employed to address almost any research topic you can think of.

5. 5 Two Critical Elements Confidentiality Ethics

6. 6 Confidentiality Information obtained in the interviews should not be directly attributed to any particular individual. Assuring confidentiality will make informants feel more comfortable sharing information that may be controversial or of a personal nature.

7. 7 Ethical Principals Respect for Persons The need to respect people�s ability to make independent decisions Beneficence Maximize benefits and minimize risks to subjects Justice Selecting participants through fair opportunity without exploiting the vulnerable

8. 8 Questions to Ask When Using Human Participants Does the purpose of the study justify using human subjects? What methods will be used to gather the data? What population group are you going to study and why?

9. 9 Questions to Ask When Using Human Participants What are the potential benefits to the participants? Do they outweigh the risks? How will you obtain informed consent? How will you handle confidentiality of personal data?

10. 10 Human Subjects Consideration Ensuring that participants are fully informed Obtaining consent Participation is completely voluntary Participants may quit at any time during the interview Participation will not impact benefits (if applicable) Protecting confidentiality Study code used in place of respondent�s name. Identifying list is locked-up. Institutional Review Board Approval

11. 11

12. 12

13. 13 What Are Key Informant Interviews? In depth interviews with 10 or more people (though it is often dependent on time and resources); enough people are needed to generate themes Interview is focused on a topic with which the interviewee has first hand knowledge and/or experience (both good and bad) The goal is to obtain qualitative information about perceptions and experiences with programs, policies, or people Documents the knowledge and experiences of people using their own words

14. 14 Why Conduct Key Informant Interviews? Obtaining qualitative, descriptive, in-depth data Understanding of motivation, behavior and perspectives of participants, administrators, policy makers, providers, advocates Providing examples of successes or failures and attempting to understand why

15. 15

16. 16 How to Conduct Interviews Use an interview guide to standardize administration Questions/prompts should be presented in the same way to all respondents, followed by any needed clarification Use an engaging tone, remain neutral and avoid giving the impression of having a strong view on the topic to avoid bias Tone of voice, body language, or interview style may cause respondents to answer questions in ways that reflect attitudes toward the interviewer rather than answers to the questions 

17. 17 Getting Off Track Although interview questions will exist in a certain order, do not hesitate to deviate. A person may start talking and answer many questions without being asked. A person may introduce a subject not included in the interview guide. Let me talk�the point of this exercise is to allow the person to tell their story, including their particular knowledge, opinions, and experiences.

18. 18 Telephone vs. In-person Often dependent on the topic of interest and feasibility (it is difficult to conduct telephone interviews with individuals who lack privacy--nursing home residents) In-person interviews are often difficult to conduct as they are extremely time consuming Most people are busy and prefer conducting interviews over the phone (it also provides privacy for the respondent)

19. 19 Keys to Conducting Interviews Actively listen Allow sufficient time for interviewee to respond to each question and elaborate on answers Listen for perceptions, ideas and themes Balance note-taking with focused listening Show interest by nodding or saying �Yes� or �I see� Be sensitive to the interviewee by respecting cultural differences and/or background

20. 20 Keys to Conducting Interviews Clarify meanings of response and request detail Repeat part of the question Paraphrase answer back to respondent to confirm interpretation Ask neutral questions: �Could you please tell me more about that?� �Anything else?� �Could you please give me an example?�

21. 21 Key Informant Interview Checklist Interview procedure checklist: Contact interviewee by telephone to schedule interview Familiarize yourself with questionnaire template and conduct a practice interview with at least one individual Anticipate potential questions about the project beforehand

22. 22 Have You Really�..Conducted a Focus Group? Many would say, �Oh yes, I have conducted focus groups.� However, what is more likely the case is that everyone has employed organized discussions. ? While this is one method of data collection, it does not hold up to the incredible rigor associated with scientific focus group methodology and analysis. ? It is understandable why the less rigorous method is often employed. It is much less labor intensive, more efficient, still interesting, and easier. Our goal today is not only to talk about focus groups in general, but guide you in achieving the best results possible given the objectives and budget.

23. Focus Groups:  What? When?Why? Who?How? 

24. 24 What is a Focus Group? A way of listening to people and learning from them A qualitative research method A guided group discussion to generate a rich understanding of experiences and beliefs A group discussion where the conversation flows easily with �nurturing� from the moderator (not a collection of simultaneous individual interviews) Conducted for a well-defined purpose A chance to observe, listen and participate

25. 25 When Should a Focus Group be Conducted? Focus groups are among the most useful and informative tools available. They can be used: at the preliminary or exploratory stages of a study during a study, perhaps to evaluate or develop a particular program after the fact, to assess impact of a program or to generate further avenues of research either as a method in their own right or as a complement to other methods

26. 26 Why Conduct Focus Groups? To draw upon respondents� attitudes, feelings, beliefs, experiences and reactions not feasible using other methods Opportunity to explore a topic as a unit rather than as individuals An effective approach when there is limited knowledge of a topic Interaction enables participants to ask questions of each other, as well as to re-evaluate and reconsider their own understandings of their specific experiences Elicit information to find out why an issue is salient, as well as what is salient about it. Provides a wonderful opportunity to hear the voice of the consumer. �Without this program, I would be unemployed.�

27. 27 Who Should Participate in a Focus Group? The research team uses its judgment to select �purposive samples� of participants who meet the needs of a particular project. Reasonably homogeneous participants--often selected on the basis of specific characteristics (age, gender, occupation, interest, education, group membership). The greatest need is compatibility. A focus group is typically composed of 6 to 10 people. The goal is enough opinions to generate discussion without making people compete for time. Participants must be similar to each other in a way that is important to the researcher (i.e., purpose of study). This similarity is the basis for recruitment and participants are informed of these common characteristics at the beginning of the discussion

28. 28 Is Focus Group Data Generalizable? Focus groups are rarely used to test hypotheses. Data is VIRTUALLY NEVER used in focus groups to generalize to [sub]populations because respondents rarely are drawn from a probability sample. Reports describe main themes. Unlike the complex statistics often used for surveys, focus group reports may do simple counts and give anecdotal examples. Quotes from notes and the tapes are often placed in the report. The purpose of the focus group is not to generalize to the larger body. The purposes are usually insight--gathering a range of dimensions that provide more depth than is typically found in standardized surveys. 

29. 29 What Plans Are Needed for the Focus Group Study? Identify the purpose of the study. Write a general purpose statement that reflects the information you do and do not want to obtain. Identify goals. Write the specific questions to be addressed; determine how the information from the focus group will be used; identify the outcomes required for the focus group to be successful. Determine who to study. Be as specific as possible when selecting individuals for inclusion in a focus group.

30. 30 How Are Participants Selected? What type of people would give you the information you want? Who are the target audiences? What characteristics should the participants have? Identify information-rich people (i.e., �those individuals from whom one can learn a great deal about the issues of central importance to the purpose of the research� (Patton, p. 169, 1990)

31. 31 Planning the Focus Group Session Plan meetings to be 1 to 1.5 hours long Time of day is dependent on the population being interviewed Setting: Hold sessions in a conference room, or other setting with adequate air flow and lighting. Configure chairs so that all members can see each other. From the participants' point of view it is important that people feel relaxed and not intimidated, therefore convenience and comfort are the main concerns: How easy will it be to reach the facility? Is parking available? Do the facilities make it pleasant to sit and talk for up to two hours? Do the facilities convey a pleasant atmosphere?� The most important aspect of a focus group: Provide refreshments!

32. 32 Developing the Moderator Guide Questions should be open-ended in nature. The guide is used to serve as a map to chart the path of the focus group from beginning to end. The wording of the questions should be direct, forthright, comfortable, and above all simple to understand. 5-minute questions occur at the beginning to get participants thinking about the topic before they delve into it in more depth. 10-minute questions are the area of central concern that drives the review. 15-minute questions take up a sizeable amount of time and must be very important.

33. 33 Systematic Notification Procedures Contact potential participants by phone or in person (2 weeks before meeting time) May choose to send a letter from the funding agency first Send a personalized invitation Phone (or contact) each person the day before the focus group

34. 34 Incentives for Participants Money Pizza Tickets Coupons Donation

35. 35 Beginning the Focus Group The first few moments in a focus group discussion are critical. Create a thoughtful, permissive atmosphere Complete informed consent Provide the ground rules Set the tone Recommended introduction pattern: Welcome Overview and topic Ground rules First question

36. 36 What Is the Role of the Moderator? It is necessary for the moderator to probe. Once a topic has been introduced, participants should carry on the discussion with little interruption from the moderator�even if silence is present! When groups move off topic, the moderator becomes more interactive, providing direction and reiterating the reason for conducting the focus group. Eye contact can be made with members who are not participating. It is possible to conduct focus groups without asking a single question, yet remaining very active as the moderator. Some of the best probes are not questions at all: Tell me about... Give me a picture of... Compare with each other how you... I'm going to start a story that I want you to complete.

37. 37 Good Moderator Qualities interested in the topic good communication skills friendliness interested in people openness to new ideas listening skills Participants must feel comfortable with the moderator.� They must feel that the moderator is the appropriate person to ask the questions and that their answers can be openly offered and discussed. There should be no perceived power difference between the moderator and the participants.� The obvious example of this is when a manager tries to hold a focus group with his/her subordinates.

38. 38 Ending Questions All things considered question Ask participants to reflect on the entire discussion and then to offer their positions or opinions or to tell what they think is the most important part of the discussion. Summary question After a brief oral summary, ask participants, "Is this an adequate summary?� Final question "Have we missed anything?�

39. 39 Courtesy Busy people gave time to the project Make sure they get a copy of the interim or final report

40. 40 Compiling, Analyzing, Reporting Results Qualitative data is necessary to collect and analyze as it provides the researchers with insight into thoughts, attitudes, and perceptions of a particular population The data collected will not be in the form of categorical numbers as it would be in quantitative analysis

41. 41 How To Compile Data Read over notes for each question and highlight key issues Make note of insightful or remarkable comments Make a list of key issues/words for each question Number each interviewee in serial order and place that number next to each key issue/word that respondent raised

42. What Do We Do With All of This Data?

43. 43 Features of Empirical Reports Themes Taxonomy Conceptual models

44. 44 Themes Characterize experiences of individual participants by the more general insights that emerge from the whole of the data Recurrent and unifying concepts or statements May reveal relationships among concepts

45. 45 Taxonomy Classification system for describing multifaceted, complex phenomena Catalogue key components and related subcomponents such as: a complex intervention, program or system individual behavior organizational environment or culture

46. 46 Conceptual Model Represents a set of propositions concerning relationships between various determinants, mediating factors, and consequences Examines underlying processes and systematic reasons for events, experiences and phenomena, often in literature

47. 47 Core Principles of Data Analysis Analysis is an ongoing and iterative process Requires continual reflection and writing memos or notes throughout the study Critical to document analytic decisions via an �audit trail� or memo function Analysis is very time and energy intensive

48. 48 Implementing Analysis: General Steps Prepare data Read data for general understanding Code data Analyze data Develop analytic output Today we will briefly review coding, and focus on generating certain types of findings

49. 49 Study Overview Study purpose: To conduct a state-wide needs assessment with a focus on the needs and experiences of people with disabilities and employers. In-depth interviews with 36 individuals; stratified into 3 groupings 35 focus groups (n=286); stratified into 5 groupings 85 codes were classified into nine themes

50. 50 Key Informant Interviews

51. 51 Focus Groups

52. 52 What Are Codes? Codes are tags or labels for assigning units of meaning to descriptive or inferential information Coding is the process of organizing the data into �chunks� that are alike, moving from words and sentences to �incidents�

53. 53 Approaches to Developing Initial Codes Purely inductive, grounded, de novo �Start lists� Integrated approach, combining start lists with guided induction

54. 54 Purely Inductive, Grounded Method Participant experiences drive the analysis Limits possibility of researcher erroneously forcing a preconceived result Develop codes de novo through line by line review of data May employ constant comparisons to build code structure from ground

55. 55 �Start Lists� Method Somewhat deductive in approach Preliminary organizing framework Begin with codes corresponding to concepts in existing literature and from the expertise of the research team content codes - concepts well known in extant scientific literature code types - acts, activities, meanings, participation, relationships, settings

56. 56 Coding Is an Iterative Process Develop initial codes, including operational definitions and their properties Apply the (still evolving) codes to several units of analysis (e.g., transcripts); revise codes as needed Negotiate code definitions and properties in group, developing final code list with some structure (i.e., codes and subcodes) Apply final code list to all data

57. 57 Types of Codes Conceptual codes and subcodes identify key concept domains and essential definitions Relationship codes links between concepts coded with conceptual codes Participant perspective codes negative, positive, indifferent about particular experience Participant characteristic codes gender, type of disability, race/ethnicity; facilitate comparisons Setting codes identify settings or environments; facilitate comparisons

58. 58 Refining Code Structure � Be Open Codes will change as you code data encompass more ideas properties are fleshed out may merge into existing codes may split into new, additional codes Detail your coding decisions (part of audit trail) Updated code structures Updated examples of what has been included within each code Memos about choices in codes

59. 59 Number of Coders? Single coder is sufficient, though not preferred researcher is the instrument data collection and analysis are so intertwined IMP to disclose bias, philosophical approach Team of coders select team members with differing backgrounds improves breadth and depth of analysis increasingly more common in health services research important to conduct careful training, some also compute inter-rater reliability

60. 60 Analysis to Develop Common Themes Use data generated from a particular question: �What prevents X from working? What impeded X? How so? Why?� Code conceptually distinct groups of responses as �themes� or perspectives Themes may be used in two ways: to generate hypotheses for future research to develop and implement effective policies, programs and practices

61. 61 Example: Code Development

62. 62 Example: Code Development

63. 63 Example: Theme Development Question: What prevents people with disabilities or special needs from getting the jobs that they want? Sample Theme: Barriers to employment for people with disabilities Codes: Societal preconceptions and lack of awareness Low expectations Individual attitudes and beliefs Employment discrimination Benefit programs� limitations and complexity Transportation difficulties Lack of satisfying job opportunities Challenges in the hiring process Lack of job accommodations or support Need for skills and training, including social skills Personal care assistance Housing issues Lack of information about resources

64. 64 Quotes to Support the Theme and Codes Code: Societal preconceptions and lack of awareness �It is always assumed they will take entry level jobs. It�s a terrible stigma.� (FG-9, Employer) �Assumptions made by the employer. That they don�t think that you can do the job. I�ve had occasions, I�ve got my masters in social work � it�s always funny and it�s always funny they ask, �What if there�s a fire and you�re on the second floor. Well we worked it out. It�s just this assumption that you are a liability.� (FG-10, pg 4)

65. 65

66. 66

67. 67

68. 68

69. 69

70. 70 Enhancing the Rigor of Moving From Codes to Findings Multiple researchers, with differing disciplines Respondent validation of findings Detailed audit trail Evolution of code definitions and code structure Process used for causal analysis, if relevant Clear, complete explanation of how the research team moved from codes to theory, reported in published work

71. 71 Closing Points Approaches to generating and reporting results from qualitative studies are appropriately diverse Findings particularly useful in health services research include themes, taxonomy and conceptual models A structured approach to coding can facilitate analysis leading to a given form of output

72. 72 References and Resources Access Project. Getting the Lay of the Land On Health: A Guide for Using Interviews to Gather Information (Key Informant Interviews), 1999. Bazzoli, G. J., S. M. Shortell, N. Dubbs, C. Chan and P. Kralovec. (1999). A taxonomy of health networks and systems: Bringing order out of chaos. Health Services Research 33 (6): 1683-717. Bradley, E. H., E. S. Holmboe, J. A. Mattera, S. A. Roumanis, M. J. Radford and H. M. Krumholz. (2001). A qualitative study of increasing beta-blocker use after Myocardial Infarction: Why do some hospitals succeed? Journal of the American Medical Association 285 (20): 2604-11. Crabtree, B. & Miller, W. (1999). Doing qualitative research, 2nd edition. Newbury Park, CA:Sage. Curry L, (2007). Moving from codes to findings. Powerpoint presentation. Department of Epidemiology and Public Health Yale School of Medicine Fahey, K., fanning, L., Fischer, K., Hooper, L. & Kirckpatrick, S. Conducting Key Informant Interviews (2006). http://courses.washington.edu/nutr531/HEBD/KIInterviews/KeyInformantInterviews.ppt Feldman, M. S. (1995). Strategies for Interpreting Qualitative Data. Thousand Oaks: Sage Publications. Fink, A. How to Manage, Analyze, and Interpret Survey Data. (Vol. 9) in The Survey Kit. 2nd ed. Thousand Oaks: Sage, 2003.

73. 73 References and Resources Giacomini M, and Cook D. (2000a). Users� guides to the medical literature:XXIII. Qualitative research in health care A. Are the results of the study valid? Evidence-based Medicine Working Group. Journal of the American Medical Association 284(3):357-62. Giacomini M, and Cook D. (2000b). Users� guides to the medical literature:XXIII. Qualitative research in health care B. What are the results and how do they help me care for my patients? Evidence-based Medicine Working Group. Journal of the American Medical Association 284(3):357-62. Glaser, B. & Strauss, A. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine. Glaser, B. G. (1992). Emergence v Forcing Basics of Grounded Theory Analysis. Mill Valley, CA: Sociology Press. Gubrium, J. F. and J. A. Holstein. (1997). The New Language of Qualitative Method. New York: Oxford University Press Huberman AM & Miles, MB. (2002) The Qualitative Researcher�s Companion, Thousand Oaks, CA. Sage Janesick, V. (2003). The choreography of qualitative research: Minuets, improvisations, and crystallization. In Strategies of Qualitative Inquiry N. Denzin and Y. S. Lincoln. Thousand Oaks, CA, Sage Publications: 46-79. Kirk, J. & Miller, M. (1986). Reliability and validity in qualitative research. Qualitative Research Method Series, 1. London, Sage.

74. 74 References and Resources Krueger, R. A. & Casey, M. A. (2000). Focus groups: A practical guide for applied research. Sage Publications Lincoln, Y. S. and E. G. Guba. (1985). Naturalistic Inquiry. Beverly Hills: Sage. Lofland, J. 1971. Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Belmont, CA: Wadsworth. Mays N. and Pope C. (2000). Assessing quality in qualitative research. British Medical Journal 320:50-52.� Mays N. and Pope C. (1995). Qualitative research: Rigor and qualitative research. British Medical Journal. 311(6997):109-112 Miles, M. & Huberman, AM. (1994). Qualitative data analysis: An expanded sourcebook. Thousand Oaks, CA: Sage Publishers. Morse, J. M. 1999. Qualitative methods: The state of the art. Qualitative Health Research 9 (3): 393-406. Morse, J. M. and L. Richards. (2002). Readme First for a User's Guide to Qualitative Methods. Thousand Oaks, CA: Sage Publications. Mountain States Group, Inc. Conducting Key Informant and Focus Group Interviews. 1999. Nunez-Smith M, Curry L, Bigby JA, Berg D, Krumholz H, Bradley E. Impact of race on the professional lives of physicians of African descent. Annals of Internal Medicine. 146:45-51.

75. 75 References and Resources Oishi, S. How to Conduct In-Person Interviews for Surveys (Vol. 5) in The Survey Kit. 2nd ed. Thousand Oaks: Sage, 2003. Patton, M (1999). Enhancing the quality and credibility of qualitative analysis. Health Services Research 34(5): 1189-1208. Patton, M. Q. (2002). Qualitative Research and Evaluation Methods, 3d ed. Thousand Oaks, CA: Sage Publications. Pope, C., S. Ziebland and N. Mays. (2000).Qualitative Research in Health Care. Analysing Qualitative Data. British Medical Journal 320 (7227): 114-6 Popay J, Rogers A, and Williams G. (1998). Rationale and standards for the systematic review of qualitative literature in health services research. Qualitative Health Research 8(3):341-351. Ryan, G. W. and H. R. Bernard. (2003). Techniques to Identify Themes. Field Methods 15 (1): 85-109. Strauss, A.L. & Corbin, J. (1990). Grounded theory in practice. London: Sage. USAID Center for Development Information and Evaluation. Conducting Key Informant Interviews. 1996. Winzelberg GS, Patrick DL, Rhodes LA, Devo RA. (2005). Opportunities and challenges to improving end-of-life care for seriously ill elderly patients: a qualitative study of generalist physicians. J Palliat Med. Apr;8(2):291-9.