Community-Based Evaluation Methods & Data Analysis Amy Carroll April 22, 2005

1. Community-Based Evaluation Methods & Data AnalysisAmy CarrollApril 22, 2005

2. Opinions, priorities Aspirations, motivations Level of awareness, knowledge, attitudes or beliefs Behaviors, practices Assets, skills Networks, associations Needs, fears, problems, concerns Demographic characteristics Services or resources provided Resident utilization of services or resources provided Numbers or rates of disease, illness, disability, injuries Sales transactions, purchases Policies Pictures, other visuals Maps Different Types of Data

3. Quantitative: Collected in the form of numbers or percentages Closed-ended questions Answers who?, what?, when? and where? Can demonstrate cause and effect Can “represent” a population Cannot collect new ideas or responses, only those considered ahead of time Qualitative: Collected in the form of words, concepts, themes, or categories Open-ended questions Answers how?, why? Can provide richer, more in-depth data Can provide data in a respondent’s own words Can explore new ideas in a dynamic and unstructured way Quantitative vs. Qualitative Data

4. Common Community-Based Evaluation Methods • Secondary data • Activity tracking • Client tracking • Mapping/ Photovoice • Surveys • Focus groups • Key informant interviews

5. Which of these methods collect quantitative data? Which collect qualitative data?

6. Secondary Data • County health department • Vital statistics  Birth & death certificates • City or county government • Schools (including school nurse records) • Hospital or clinic records • Local health promotion, health ed or advocacy agencies • Private foundations which fund community programs Universities or local community colleges • Local surveys or research efforts going on in your community?? • Data you already collect!

7. Activity Tracking • Tracking and documenting program activity implementation • Tracking and documenting number of people reached • Process evaluation method, but important for outcome evaluation • May include: weekly/monthly program implementation log, sign-in sheets at classes or health fairs, counts of materials/ brochures distributed, etc.

8. Client Tracking • Tracking and documenting encounters with clients; describing the encounter and client characteristics • More than just counting • Can be used for process and outcome evaluation • May include: outreach encounter form, home visit form, clinic intake form, etc.

9. Mapping/ Photovoice • Data can also include maps and photos  don’t just have to be numbers or words • Maps and photos can provide visual imagery to give meaning to quantitative data • They can show graphically what cannot be described with numbers or words, such as the conditions of a neighborhood or program facilities, or a community’s proximity to park space or health services. • Make sure your maps and photos are clear and illustrate the points you would like to make

10. Surveys • Surveys are composed of carefully crafted questions on a topic or issue for the purpose of revealing information about community residents or services they utilize. • The most commonly used survey methods are: • Mailed Surveys • Hand-out Surveys • Face to Face Surveys • Telephone Interview Surveys

11. Planning A Survey • Determine your purpose • Determine if the survey data collection is appropriate • Select your target audience • Select a survey method • Design survey questionnaire • Pilot test survey • Publicize and distribute surveys

12. Determine the Data You Need • Draft your research questions – what you want to learn from these interviews. • Identify the data you will need to collect in order to answer these questions. • The type of data you are looking for will help you identify the best persons in the community to survey, and the best survey questions to ask them.

13. Target Population The following are two steps in determining your survey target population. 1) Define your target population --identify the group of individuals from your community from who you want to collect information (single mothers, high school students, the uninsured, etc.) 2) Identify the geographic area of your target population.--Where is your group of interest located?What are the geographic boundaries of your target population?

14. Select A Survey Method Self Administered: These surveys are filled out by respondents themselves without the assistance of trained interviewers. A. Mail Surveys B. Drop-off Surveys Administered by Interviewer: These surveys are filled out with the assistance of trained interviewers. A. Face to Face Surveys B. Telephone Surveys

15. Design Survey Questionnaire • Your survey questions should be in direct relationship with what you want to learn from your target population. • Asking unrelated questions will interrupt the flow of the survey and tire out your respondents. • Be brief and strategic when designing questions! • Open-ended questions are great for answering “why” questions, but only include one or two. • Motivating people to fill out your survey will help you collect the data you need – this can include a benefits statement or incentives such as entry into a lottery or drawing.

16. Pilot Test Survey • It is always a good idea to pilot test the survey with a small group of people from your target population. • The purpose for piloting the survey is to get feedback on the survey question and structure before implementing it. • This will ensure you identify potentially confusing, offensive, leading, or loaded questions from your survey.

17. Publicize and Distribute Surveys • Advertise your survey a couple of weeks before distribution in the community. • Publicizing what you are trying to do and how the information collected will benefit the community may help increase your response rate. • Create a survey distribution timeline and stick to it - postponing survey distribution date may limit the time you have set aside for data analysis.

18. Advantages Data can be collected from a lot of respondents easier than any other method Can get a large enough sample that can be representative of the larger population Findings can be generalized to the larger population Can cover a lot of topics Can easily compare different groups’ data to each other Disadvantages Survey instrument must be carefully constructed to avoid leading questions, and to make sure the appropriate responses are available Response rates can be low for self-administered surveys, especially mailed ones Response will be low if survey is too long Advantages and Disadvantages of Surveys

19. Focus Groups Focus group are a qualitative research method designed to learn more about how people think, feel, or make decisions (attitudes, perceptions, opinions, experiences) through focused discussions.

20. Focus Groups (cont.) • A series of discussions involving 8-12 people, selected to share their perceptions of a defined topic • Stimulates participants to share their opinions openly in a group discussion • Captures rich data in participants’ own words • Flexible to capture new ideas and issues • Can be difficult to summarize and interpret results across groups • Need to be creative when recruiting busy people

21. Planning a Focus Group • Determine the data you need • Determine and select your target audience • Develop a focus group discussion guide • Schedule focus groups • Recruit focus group participants • Recruit moderator and note-taker • Plan and conduct focus groups • Compile and organize focus group data

22. Determine the Data You Need • Draft your research questions – what you want to learn from these focus groups. • Identify the data you will need to collect in order to answer these questions. • The type of data you are looking for will help you identify the best persons to recruit for your focus groups, as well as the best questions to develop for your focus group discussion.

23. Target Audience • To whom do you want to talk? • What types of experiences and perspectives you are hoping to capture in your discussion? • Design and use a screening tool to recruit participants: ask questions that qualify people to participate. • It is important that you include participants who will feel comfortable talking to each other. • Participants should not know each other.

24. How Many Focus Groups? • Conduct 2-3 focus groups for each sub-population of interest (i.e. gender, age, race/ethnicity, language, etc.) • 2-3 ensures that the perspectives and experience you capture are more representative of the larger population and not just by “chance”. • Length: 1.5 hrs.-2 hrs. • Focus Group Size: 6-12 participants

25. Focus Group Discussion Guide • Discussion guide: An outlined script to help guide the moderator and focus the discussion • Organization: Introduction Purpose Ground rules Focus group questions Closing statements

26. Discussion Guide Questions • Introductory questions: getting people to think about the general topic. • Transition questions: questions that get participants to logically transition between the introduction topics and the ideas contained in the key questions. • Key questions: 2-5 questions important to getting the information you have set out to collect. • Ending questions: questions that provide closure and reflection. • Summary question: note-taker summarizes the major themes heard throughout the discussion in a short oral summary (2-3 minutes), then will ask the participants if the summary covered all the major points.

27. Scheduling Focus Groups • Location: The focus groups should take place in aconvenient and accessible location. It is also important to select a neutral environment. • Time of day: most convenient for your target audience. Set time earlier than intend to start. • Incentives: Food or refreshments, money or gifts, drawings, child care, transportation, anything that is meaningful to those community members.

28. Conducting & Moderating • The discussion guide will help guide the moderator with the questions you want answered. • Moderator Responsibilities:setting up ground rules, keeping the discussion focused and on track, drawing out comments from quiet participants, and asking for clarification or additional information from participants when needed. • Note-taker Responsibilities: takes comprehensive notes, operates audio equipment (tape-recorder), keeps track of time, handles environmental conditions and logistics, unexpected interruptions, and incentives.

29. Skills of the Moderator Extremely important to select moderators based on skills and experience: • Comfortable/familiar with leading meetings or groups • Able to exercise mild control over the group • Able to guide conversation back on target • Maintain group enthusiasm and interest • Curious about the topic • Respects participants and their comments • Creates and maintains a comfortable environment • Adequate background knowledge • Good oral and written skills • Remains neutral

30. Advantages Flexible Captures rich, in-depth data Immediate results Encourages and stimulates individuals to share more openly Data can be combined with quantitative data to provide a complete picture about an issue Disadvantages May be challenging to recruit participants Need to schedule at least 2-3 focus groups to capture diversity Difficult to generalize results to the larger population because of small numbers of participants Difficult to compare results across groups Advantages and Disadvantages of Focus Groups

31. Key Informant Interviews • Key informant interviews are qualitative, in-depth interviews with key informants • Key informants are individuals who know what is going on in the community (community leaders, professionals, residents, advocates, elected officials, doctors, etc.) • The following are two common techniques used to conduct key informant interviews: • Telephone interviews • Face-to face interviews

32. Planning Key Informant Interviews • Determine the data you need • Determine target population and brainstorm possible key informants • Select key informants • Select interview technique • Develop an interview guide • Determine documentation method • Select designated interviewer(s) • Conduct key informant interviews • Compile and organize key informant interview data

33. Determine the Data You Need • Draft your research questions – what you want to learn from these interviews. • Identify the information you will need to collect to answer these questions. • The type of data you are looking for will help you identify the best persons in the community to interview, and the best questions to ask them.

34. Target Audience and Key Informants • Determine your target population (program clientele, racial/ethnic minority groups, adolescents, women, the elderly, HIV+ persons, etc.) • Brainstorm and create a list of possible key informants who are knowledgeable and closely involved with your target population. • Select key informants: • Who have first-hand knowledge about the target population, the program, and the issues you are trying to investigate. • Who represent different experiences, backgrounds, and viewpoints.

35. Select Interview Technique Telephone interviews (15-25 min): • May be the most convenient and least time intensive way to interview busy key informants. • Disadvantage: not having the personalized interaction otherwise possible through a face-to-face interview. Face-to-face interviews (20-30 min): • Most frequently used format. But time intensive. • Advantages: provides a free-exchange of ideas and lends itself to ask more complex questions.

36. Develop Interview Guide Introduction- Introduce yourself and your purpose and/or benefits statement. Key questions- Draft 5-10 questions important to getting the information you have set out to collect. Probing questions – Probing questions encourage participants to reflect more deeply on the meaning of their comments. Closing question—Provide an opportunity for the key informant to give any additional information or comments. Summary—Quickly summarize the major comments made.

37. Determine Documentation Method • Note-taking:manually write the key informant’s comments while conducting the interview, on a copy of the interview guide. • Tape-recording: use a tape recorder to document what key informants say. This approach may allow the interviewer to freely engage in the conversation without worrying about note-taking.

38. Conduct Interviews • Thank respondents for their time and stress the importance of the interview. • Interviewers should listen carefully for recurring and new opinions or beliefs. • In order to compare with other interviews and identify emerging themes, it is important to get answers from every person interviewed on key questions.

39. Advantages Rich data can be gathered relatively cheaply & easily Allows interviewer to establish rapport with the respondent, clarify questions, and draw out responses Allows for discussion of topic without group dynamic of focus groups Provides an opportunity to build relationships with important community stakeholders Disadvantages Selecting the “right” key informants may be difficult so they represent diverse backgrounds and viewpoints May be challenging to reach and schedule interviews with busy and/or hard-to-reach respondents Difficult to generalize results to the larger population unless interviewing many key informants Advantages and Disadvantages of Key Informant Interviews

40. Things to Consider Before You Select Your Methods • What data collection method(s) would best collect the type of data you need from your target population • What data collection method(s) would help you to best answer your research question(s)? • Think of your available resources: time, staffing, skills, funding, computer technology • Prior experience • Potential benefits • Potential challenges

41. How to Understand the Data You Collect (Data Analysis)

42. Why It’s a Good Idea to Plan Ahead for Data Analyses • To determine if the data you collect are practical for analyses • To review the appropriateness of your chosen data collection method(s) • To inform the data collection instrument you develop • To decide if you will need help with analyses

43. 1: Check the Data • Make sure it’s all there • Make sure it makes sense • Catch any mistakes that happened while filling out the survey or entering the data • Perform data quality checks throughout your data collection process • Common mistakes: • A missing or incomplete response • An impossible response • A response unrelated to the question • A response that contradicts an earlier response

44. 2: Go Back to the Research Questions • Be sure to analyze only the data that will help you answer your research question(s) • Don’t get side-tracked into analyzing other pieces of data with your limited time and resources • It is normal to collect more data than you will need for your evaluation • You can return to any other interesting data after completing your evaluation

45. 3: Reduce the Amount of Data • Save your data in its original form so that you can return to it if you need to • Look at it in small chunks at a time • Eliminate any irrelevant data from your analysis • Create summary documents to help you “eyeball” the data more easily to look for patterns or themes: • Numeric spreadsheet • Qualitative summary document

46. 4: Analyze the DataQuantitative Data Analysis: Looking for Patterns in the Data • Calculate averages • Used with “continuous” data: infinite number of values • Add numbered responses • Divide by number of responses • 2 + 4 + 6 + 10 = 22 22/4 = 5.5 • Count frequencies • Used with “discrete” data: set number of response categories • Count number of responses • Number “yes”, number “no” • Number of choice “A”, Number of choice “B”, etc.

47. 4: Analyze the DataQuantitative Data Analysis: Looking for Patterns in the Data • Calculate proportions • Is standardized • Equation: % = # of responses/ total # of respondents X 100 • Example: 45 females/ 100 respondents X 100 = 45% of the respondents were female • Calculate rates • Similar to proportions; also standardized • Easier to compare • Equation: Rate = freq of event in pop/ total pop X 100,000 • Example: 15 cases of lung cancer/ population of 3,500 X 100,000 = 428 persons per 100,000 have lung cancer

48. 4: Analyze the DataQuantitative Data Analysis: Looking for Patterns in the Data • Compare averages, frequencies, proportions, rates • Compare data from different populations: • County vs. County • County vs. State • State vs. US • Compare data from different segments of a population: • Male vs. female • African American, white, Latino, Asian, American Indian, etc. • Children, teens, adults, seniors • Clientele of your program vs. those who are not • Different income or health insurance levels • Married vs. single • Smokers vs. not

49. 4: Analyze the DataQuantitative Data Analysis: Looking for Patterns in the Data • Present the data in different ways to see additional patterns and relationships: • Chart/ table – shows averages, counts, proportions, or rates side-by-side • Pie graph – demonstrates percentages of the whole • Bar graph – compares quantities • Line graph – shows trends over time • Determine your findings • Interesting results? • Interesting patterns or relationships? • These are your findings

50. Exercise 5.4A(Using Step 5 of Performing a Community Assessment) • Look at sample survey and sample spreadsheet in Appendix C • Answer questions on page 51 and 52