Community based evaluation methods data analysis amy carroll april 22 2005
Download
1 / 56

community-based evaluation methods data analysis amy carroll - PowerPoint PPT Presentation


  • 194 Views
  • Uploaded on

Community-Based Evaluation Methods & Data Analysis Amy Carroll April 22, 2005. Opinions, priorities Aspirations, motivations Level of awareness, knowledge, attitudes or beliefs Behaviors, practices Assets, skills Networks, associations Needs, fears, problems, concerns

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'community-based evaluation methods data analysis amy carroll' - Audrey


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Different types of data l.jpg

Opinions, priorities

Aspirations, motivations

Level of awareness, knowledge, attitudes or beliefs

Behaviors, practices

Assets, skills

Networks, associations

Needs, fears, problems, concerns

Demographic characteristics

Services or resources provided

Resident utilization of services or resources provided

Numbers or rates of disease, illness, disability, injuries

Sales transactions, purchases

Policies

Pictures, other visuals

Maps

Different Types of Data


Quantitative vs qualitative data l.jpg

Quantitative:

Collected in the form of numbers or percentages

Closed-ended questions

Answers who?, what?, when? and where?

Can demonstrate cause and effect

Can “represent” a population

Cannot collect new ideas or responses, only those considered ahead of time

Qualitative:

Collected in the form of words, concepts, themes, or categories

Open-ended questions

Answers how?, why?

Can provide richer, more in-depth data

Can provide data in a respondent’s own words

Can explore new ideas in a dynamic and unstructured way

Quantitative vs. Qualitative Data


Common community based evaluation methods l.jpg
Common Community-Based Evaluation Methods

  • Secondary data

  • Activity tracking

  • Client tracking

  • Mapping/ Photovoice

  • Surveys

  • Focus groups

  • Key informant interviews


Slide5 l.jpg

Which of these methods collect quantitative data?

Which collect qualitative data?


Secondary data l.jpg
Secondary Data

  • County health department

  • Vital statistics  Birth & death certificates

  • City or county government

  • Schools (including school nurse records)

  • Hospital or clinic records

  • Local health promotion, health ed or advocacy agencies

  • Private foundations which fund community programs Universities or local community colleges

  • Local surveys or research efforts going on in your community??

  • Data you already collect!


Activity tracking l.jpg
Activity Tracking

  • Tracking and documenting program activity implementation

  • Tracking and documenting number of people reached

  • Process evaluation method, but important for outcome evaluation

  • May include: weekly/monthly program implementation log, sign-in sheets at classes or health fairs, counts of materials/ brochures distributed, etc.


Client tracking l.jpg
Client Tracking

  • Tracking and documenting encounters with clients; describing the encounter and client characteristics

  • More than just counting

  • Can be used for process and outcome evaluation

  • May include: outreach encounter form, home visit form, clinic intake form, etc.


Mapping photovoice l.jpg
Mapping/ Photovoice

  • Data can also include maps and photos  don’t just have to be numbers or words

  • Maps and photos can provide visual imagery to give meaning to quantitative data

  • They can show graphically what cannot be described with numbers or words, such as the conditions of a neighborhood or program facilities, or a community’s proximity to park space or health services.

  • Make sure your maps and photos are clear and illustrate the points you would like to make


Surveys l.jpg
Surveys

  • Surveys are composed of carefully crafted questions on a topic or issue for the purpose of revealing information about community residents or services they utilize.

  • The most commonly used survey methods are:

    • Mailed Surveys

    • Hand-out Surveys

    • Face to Face Surveys

    • Telephone Interview Surveys


Planning a survey l.jpg
Planning A Survey

  • Determine your purpose

  • Determine if the survey data collection is appropriate

  • Select your target audience

  • Select a survey method

  • Design survey questionnaire

  • Pilot test survey

  • Publicize and distribute surveys


Determine the data you need l.jpg
Determine the Data You Need

  • Draft your research questions – what you want to learn from these interviews.

  • Identify the data you will need to collect in order to answer these questions.

  • The type of data you are looking for will help you identify the best persons in the community to survey, and the best survey questions to ask them.


Target population l.jpg
Target Population

The following are two steps in determining your survey target population.

1) Define your target population --identify the group of individuals from your community from who you want to collect information (single mothers, high school students, the uninsured, etc.)

2) Identify the geographic area of your target population.--Where is your group of interest located?What are the geographic boundaries of your target population?


Select a survey method l.jpg
Select A Survey Method

Self Administered: These surveys are filled out by respondents themselves without the assistance of trained interviewers.

A. Mail Surveys

B. Drop-off Surveys

Administered by Interviewer: These surveys are filled out with the assistance of trained interviewers.

A. Face to Face Surveys

B. Telephone Surveys


Design survey questionnaire l.jpg
Design Survey Questionnaire

  • Your survey questions should be in direct relationship with what you want to learn from your target population.

  • Asking unrelated questions will interrupt the flow of the survey and tire out your respondents.

  • Be brief and strategic when designing questions!

  • Open-ended questions are great for answering “why” questions, but only include one or two.

  • Motivating people to fill out your survey will help you collect the data you need – this can include a benefits statement or incentives such as entry into a lottery or drawing.


Pilot test survey l.jpg
Pilot Test Survey

  • It is always a good idea to pilot test the survey with a small group of people from your target population.

  • The purpose for piloting the survey is to get feedback on the survey question and structure before implementing it.

  • This will ensure you identify potentially confusing, offensive, leading, or loaded questions from your survey.


Publicize and distribute surveys l.jpg
Publicize and Distribute Surveys

  • Advertise your survey a couple of weeks before distribution in the community.

  • Publicizing what you are trying to do and how the information collected will benefit the community may help increase your response rate.

  • Create a survey distribution timeline and stick to it - postponing survey distribution date may limit the time you have set aside for data analysis.


Advantages and disadvantages of surveys l.jpg

Advantages

Data can be collected from a lot of respondents easier than any other method

Can get a large enough sample that can be representative of the larger population

Findings can be generalized to the larger population

Can cover a lot of topics

Can easily compare different groups’ data to each other

Disadvantages

Survey instrument must be carefully constructed to avoid leading questions, and to make sure the appropriate responses are available

Response rates can be low for self-administered surveys, especially mailed ones

Response will be low if survey is too long

Advantages and Disadvantages of Surveys


Focus groups l.jpg
Focus Groups

Focus group are a qualitative research method designed to learn more about how people think, feel, or make decisions (attitudes, perceptions, opinions, experiences) through focused discussions.


Focus groups cont l.jpg
Focus Groups (cont.)

  • A series of discussions involving 8-12 people, selected to share their perceptions of a defined topic

  • Stimulates participants to share their opinions openly in a group discussion

  • Captures rich data in participants’ own words

  • Flexible to capture new ideas and issues

  • Can be difficult to summarize and interpret results across groups

  • Need to be creative when recruiting busy people


Planning a focus group l.jpg
Planning a Focus Group

  • Determine the data you need

  • Determine and select your target audience

  • Develop a focus group discussion guide

  • Schedule focus groups

  • Recruit focus group participants

  • Recruit moderator and note-taker

  • Plan and conduct focus groups

  • Compile and organize focus group data


Determine the data you need22 l.jpg
Determine the Data You Need

  • Draft your research questions – what you want to learn from these focus groups.

  • Identify the data you will need to collect in order to answer these questions.

  • The type of data you are looking for will help you identify the best persons to recruit for your focus groups, as well as the best questions to develop for your focus group discussion.


Target audience l.jpg
Target Audience

  • To whom do you want to talk?

  • What types of experiences and perspectives you are hoping to capture in your discussion?

  • Design and use a screening tool to recruit participants: ask questions that qualify people to participate.

  • It is important that you include participants who will feel comfortable talking to each other.

  • Participants should not know each other.


How many focus groups l.jpg
How Many Focus Groups?

  • Conduct 2-3 focus groups for each sub-population of interest (i.e. gender, age, race/ethnicity, language, etc.)

  • 2-3 ensures that the perspectives and experience you capture are more representative of the larger population and not just by “chance”.

  • Length: 1.5 hrs.-2 hrs.

  • Focus Group Size: 6-12 participants


Focus group discussion guide l.jpg
Focus Group Discussion Guide

  • Discussion guide: An outlined script to help guide the moderator and focus the discussion

  • Organization:

    Introduction

    Purpose

    Ground rules

    Focus group questions

    Closing statements


Discussion guide questions l.jpg
Discussion Guide Questions

  • Introductory questions: getting people to think about the general topic.

  • Transition questions: questions that get participants to logically transition between the introduction topics and the ideas contained in the key questions.

  • Key questions: 2-5 questions important to getting the information you have set out to collect.

  • Ending questions: questions that provide closure and reflection.

  • Summary question: note-taker summarizes the major themes heard throughout the discussion in a short oral summary (2-3 minutes), then will ask the participants if the summary covered all the major points.


Scheduling focus groups l.jpg
Scheduling Focus Groups

  • Location: The focus groups should take place in aconvenient and accessible location. It is also important to select a neutral environment.

  • Time of day: most convenient for your target audience. Set time earlier than intend to start.

  • Incentives: Food or refreshments, money or gifts, drawings, child care, transportation, anything that is meaningful to those community members.


Conducting moderating l.jpg
Conducting & Moderating

  • The discussion guide will help guide the moderator with the questions you want answered.

  • Moderator Responsibilities:setting up ground rules, keeping the discussion focused and on track, drawing out comments from quiet participants, and asking for clarification or additional information from participants when needed.

  • Note-taker Responsibilities: takes comprehensive notes, operates audio equipment (tape-recorder), keeps track of time, handles environmental conditions and logistics, unexpected interruptions, and incentives.


Skills of the moderator l.jpg
Skills of the Moderator

Extremely important to select moderators based on skills and experience:

  • Comfortable/familiar with leading meetings or groups

  • Able to exercise mild control over the group

  • Able to guide conversation back on target

  • Maintain group enthusiasm and interest

  • Curious about the topic

  • Respects participants and their comments

  • Creates and maintains a comfortable environment

  • Adequate background knowledge

  • Good oral and written skills

  • Remains neutral


Advantages and disadvantages of focus groups l.jpg

Advantages

Flexible

Captures rich, in-depth data

Immediate results

Encourages and stimulates individuals to share more openly

Data can be combined with quantitative data to provide a complete picture about an issue

Disadvantages

May be challenging to recruit participants

Need to schedule at least 2-3 focus groups to capture diversity

Difficult to generalize results to the larger population because of small numbers of participants

Difficult to compare results across groups

Advantages and Disadvantages of Focus Groups


Key informant interviews l.jpg
Key Informant Interviews

  • Key informant interviews are qualitative, in-depth interviews with key informants

  • Key informants are individuals who know what is going on in the community (community leaders, professionals, residents, advocates, elected officials, doctors, etc.)

  • The following are two common techniques used to conduct key informant interviews:

    • Telephone interviews

    • Face-to face interviews


Planning key informant interviews l.jpg
Planning Key Informant Interviews

  • Determine the data you need

  • Determine target population and brainstorm possible key informants

  • Select key informants

  • Select interview technique

  • Develop an interview guide

  • Determine documentation method

  • Select designated interviewer(s)

  • Conduct key informant interviews

  • Compile and organize key informant interview data


Determine the data you need33 l.jpg
Determine the Data You Need

  • Draft your research questions – what you want to learn from these interviews.

  • Identify the information you will need to collect to answer these questions.

  • The type of data you are looking for will help you identify the best persons in the community to interview, and the best questions to ask them.


Target audience and key informants l.jpg
Target Audience and Key Informants

  • Determine your target population (program clientele, racial/ethnic minority groups, adolescents, women, the elderly, HIV+ persons, etc.)

  • Brainstorm and create a list of possible key informants who are knowledgeable and closely involved with your target population.

  • Select key informants:

    • Who have first-hand knowledge about the target population, the program, and the issues you are trying to investigate.

    • Who represent different experiences, backgrounds, and viewpoints.


Select interview technique l.jpg
Select Interview Technique

Telephone interviews (15-25 min):

  • May be the most convenient and least time intensive way to interview busy key informants.

  • Disadvantage: not having the personalized interaction otherwise possible through a face-to-face interview.

    Face-to-face interviews (20-30 min):

  • Most frequently used format. But time intensive.

  • Advantages: provides a free-exchange of ideas and lends itself to ask more complex questions.


Develop interview guide l.jpg
Develop Interview Guide

Introduction- Introduce yourself and your purpose and/or benefits statement.

Key questions- Draft 5-10 questions important to getting the information you have set out to collect.

Probing questions – Probing questions encourage participants to reflect more deeply on the meaning of their comments.

Closing question—Provide an opportunity for the key informant to give any additional information or comments.

Summary—Quickly summarize the major comments made.


Determine documentation method l.jpg
Determine Documentation Method

  • Note-taking:manually write the key informant’s comments while conducting the interview, on a copy of the interview guide.

  • Tape-recording: use a tape recorder to document what key informants say. This approach may allow the interviewer to freely engage in the conversation without worrying about note-taking.


Conduct interviews l.jpg
Conduct Interviews

  • Thank respondents for their time and stress the importance of the interview.

  • Interviewers should listen carefully for recurring and new opinions or beliefs.

  • In order to compare with other interviews and identify emerging themes, it is important to get answers from every person interviewed on key questions.


Advantages and disadvantages of key informant interviews l.jpg

Advantages

Rich data can be gathered relatively cheaply & easily

Allows interviewer to establish rapport with the respondent, clarify questions, and draw out responses

Allows for discussion of topic without group dynamic of focus groups

Provides an opportunity to build relationships with important community stakeholders

Disadvantages

Selecting the “right” key informants may be difficult so they represent diverse backgrounds and viewpoints

May be challenging to reach and schedule interviews with busy and/or hard-to-reach respondents

Difficult to generalize results to the larger population unless interviewing many key informants

Advantages and Disadvantages of Key Informant Interviews


Things to consider before you select your methods l.jpg
Things to Consider Before You Select Your Methods

  • What data collection method(s) would best collect the type of data you need from your target population

  • What data collection method(s) would help you to best answer your research question(s)?

  • Think of your available resources: time, staffing, skills, funding, computer technology

  • Prior experience

  • Potential benefits

  • Potential challenges



Why it s a good idea to plan ahead for data analyses l.jpg
Why It’s a Good Idea to Plan Ahead for Data Analyses

  • To determine if the data you collect are practical for analyses

  • To review the appropriateness of your chosen data collection method(s)

  • To inform the data collection instrument you develop

  • To decide if you will need help with analyses


1 check the data l.jpg
1: Check the Data

  • Make sure it’s all there

  • Make sure it makes sense

  • Catch any mistakes that happened while filling out the survey or entering the data

  • Perform data quality checks throughout your data collection process

  • Common mistakes:

    • A missing or incomplete response

    • An impossible response

    • A response unrelated to the question

    • A response that contradicts an earlier response


2 go back to the research questions l.jpg
2: Go Back to the Research Questions

  • Be sure to analyze only the data that will help you answer your research question(s)

  • Don’t get side-tracked into analyzing other pieces of data with your limited time and resources

  • It is normal to collect more data than you will need for your evaluation

  • You can return to any other interesting data after completing your evaluation


3 reduce the amount of data l.jpg
3: Reduce the Amount of Data

  • Save your data in its original form so that you can return to it if you need to

  • Look at it in small chunks at a time

  • Eliminate any irrelevant data from your analysis

  • Create summary documents to help you “eyeball” the data more easily to look for patterns or themes:

    • Numeric spreadsheet

    • Qualitative summary document


4 analyze the data quantitative data analysis looking for patterns in the data l.jpg
4: Analyze the DataQuantitative Data Analysis: Looking for Patterns in the Data

  • Calculate averages

    • Used with “continuous” data: infinite number of values

    • Add numbered responses

    • Divide by number of responses

    • 2 + 4 + 6 + 10 = 22 22/4 = 5.5

  • Count frequencies

    • Used with “discrete” data: set number of response categories

    • Count number of responses

    • Number “yes”, number “no”

    • Number of choice “A”, Number of choice “B”, etc.


4 analyze the data quantitative data analysis looking for patterns in the data47 l.jpg
4: Analyze the DataQuantitative Data Analysis: Looking for Patterns in the Data

  • Calculate proportions

    • Is standardized

    • Equation: % = # of responses/ total # of respondents X 100

    • Example: 45 females/ 100 respondents X 100 = 45% of the respondents were female

  • Calculate rates

    • Similar to proportions; also standardized

    • Easier to compare

    • Equation: Rate = freq of event in pop/ total pop X 100,000

    • Example: 15 cases of lung cancer/ population of 3,500 X 100,000 = 428 persons per 100,000 have lung cancer


4 analyze the data quantitative data analysis looking for patterns in the data48 l.jpg
4: Analyze the DataQuantitative Data Analysis: Looking for Patterns in the Data

  • Compare averages, frequencies, proportions, rates

  • Compare data from different populations:

    • County vs. County

    • County vs. State

    • State vs. US

  • Compare data from different segments of a population:

    • Male vs. female

    • African American, white, Latino, Asian, American Indian, etc.

    • Children, teens, adults, seniors

    • Clientele of your program vs. those who are not

    • Different income or health insurance levels

    • Married vs. single

    • Smokers vs. not


4 analyze the data quantitative data analysis looking for patterns in the data49 l.jpg
4: Analyze the DataQuantitative Data Analysis: Looking for Patterns in the Data

  • Present the data in different ways to see additional patterns and relationships:

    • Chart/ table – shows averages, counts, proportions, or rates side-by-side

    • Pie graph – demonstrates percentages of the whole

    • Bar graph – compares quantities

    • Line graph – shows trends over time

  • Determine your findings

    • Interesting results?

    • Interesting patterns or relationships?

    • These are your findings


Exercise 5 4a using step 5 of performing a community assessment l.jpg
Exercise 5.4A(Using Step 5 of Performing a Community Assessment)

  • Look at sample survey and sample spreadsheet in Appendix C

  • Answer questions on page 51 and 52


Step 4 analyze the data qualitative data analysis looking for themes in the data l.jpg
Step 4: Analyze the DataQualitative Data Analysis: Looking for Themes in the Data

  • Qualitative data analysis can be deceptively trickier

    • Qualitative data is by nature “bigger”

    • Analysis requires more analytical thinking and interpretation

  • Qualitative data is more open to “bias”

    • Numbers don’t lie

    • Analysts bring their values, assumptions and opinions

    • Analysts may think they “know” how people feel

  • Include at least 2, preferably 3 people in each stage of data analysis to avoid bias

  • Create as structured of a process as possible to avoid bias


Step 4 analyze the data qualitative data analysis looking for themes in the data52 l.jpg
Step 4: Analyze the DataQualitative Data Analysis: Looking for Themes in the Data

  • Read through all of the data at least twice

    • Stay “close” or “grounded” in the data

  • Create categories

    • Focus group or interview questions

    • Or specific assessment health issues

  • List themes that emerge from discussion

    • Sub-categories

    • Ideas

  • Rank order according to frequency of appearance

  • Summarize discussion around each theme

  • Determine your findings


Exercise 5 4b using step 5 of performing a community assessment l.jpg
Exercise 5.4B(Using Step 5 of Performing a Community Assessment)

  • Look at sample focus group discussion transcript in Appendix C

  • Answer questions on page 56


5 verify findings l.jpg
5: Verify Findings

  • You now have:

    • Some patterns and relationships you found in your quantitative data

    • Some themes and categories you found in your qualitative data

  • Based on the results you tallied and summarized, pull out the main findings of each method you used

  • Verify these findings by re-tallying and re-summarizing the data to make sure you get the same results

  • You can also verify your findings by comparing the results reached by 2 analysts independently


6 interpret findings and draw conclusions l.jpg
6: Interpret Findings and Draw Conclusions

  • Determine what interpretations can be drawn from each finding

  • Invite different perspectives from different Partners and community members, as this will make your conclusions stronger

  • Are the results similar to what you expected? If not, discuss why you think they are different

  • Brainstorm alternative explanations for your results to make sure you have considered all possibilities

  • Make sure the conclusions answer the original assessment questions

  • Draw conclusions and recommendations that can be shared with external audiences


Exercise 5 6 using step 5 of performing a community assessment l.jpg
Exercise 5.6(Using Step 5 of Performing a Community Assessment)

  • Look at the secondary data tables in Appendix C

  • Answer questions on page 59


ad