collecting and analyzing qualitative data all you wanted to know but were afraid to ask n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Collecting and Analyzing Qualitative Data: All You Wanted To Know, But Were Afraid To Ask PowerPoint Presentation
Download Presentation
Collecting and Analyzing Qualitative Data: All You Wanted To Know, But Were Afraid To Ask

Loading in 2 Seconds...

play fullscreen
1 / 68

Collecting and Analyzing Qualitative Data: All You Wanted To Know, But Were Afraid To Ask - PowerPoint PPT Presentation


  • 175 Views
  • Uploaded on

Collecting and Analyzing Qualitative Data: All You Wanted To Know, But Were Afraid To Ask. January 10, 2008 Presented by: Yvonne M. Watson, Evaluation Support Division National Center for Environmental Innovation Office of Policy, Economics and Innovation

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Collecting and Analyzing Qualitative Data: All You Wanted To Know, But Were Afraid To Ask' - micah


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
collecting and analyzing qualitative data all you wanted to know but were afraid to ask

Collecting and Analyzing Qualitative Data: All You Wanted To Know, But Were Afraid To Ask

January 10, 2008

Presented by:

Yvonne M. Watson, Evaluation Support Division

National Center for Environmental Innovation

Office of Policy, Economics and Innovation

U.S. Environmental Protection Agency

and

John McLaughlin

McLaughlin Associates

workshop objectives
Workshop Objectives

Participants will learn: 1) when to use qualitative data; 2) what data collection methods are available; 3) how to select participants for qualitative data collection; and 4) the steps for analyzing qualitative data.

session overview
Session Overview

Module 1: Data Collection

I. Overview

II. Qualitative Data Collection Methods:

  • Interviews
  • Focus Groups
  • Survey/Questionnaire (Open-ended questions)
  • Document/File Review
  • Observation

Module 2: Data Analysis

III. Steps for Analyzing Qualitative Data

IV. Assessing the Rigor of Qualitative Data

Module 3: Appendix, Resources and References

orientation exercise
Orientation Exercise

As a group, discuss your perceptions regarding qualitative data versus quantitative data with respect to:

  • Quality
  • Collection
  • Analysis
  • Utility
quantitative and qualitative data
Numerical data

Highly structured

Creates precise measures

Relatively easy to analyze

May not explain “why”

Closed

Risk of bias

Quantitative and Qualitative Data

Quantitative Qualitative

  • Text (Descriptions of reactions, opinions, behaviors, experiences)
  • Structured Unstructured
  • Creates lots of rich data regarding perceptions
  • Challenging to analyze
  • Labor intensive to collect
  • Risk of bias (evaluator and subject)

(World Bank , Module 6: Data Collection Methods, Slides 20 and 21)

quantitative data
Quantitative Data
  • Answers questions about: How much? How many? How often?
  • Use quantitative data when you:
    • Want to do statistical analysis
    • Want to be precise
    • Know exactly what you want to measure
    • Want to cover a large group or population
  • Quantitative Methods:
    • Examples: Survey questionnaires, tests, checklists, monitoring data.
    • Often used to obtain information on outcomes and causal relationships.
qualitative data
Qualitative Data
  • Answers questions which begin with: Why? How? In what way?
  • Use qualitative data when you:
    • Are concerned with opinions, experiences and feelings of individuals producing subjective data.
    • Want anecdotes or in-depth information
    • Are seeking understanding, themes, issues
    • Are not sure what you want to measure
    • There is no need to quantify
    • Are unable to collect quantitative data
  • Qualitative Methods:
    • Examples: Interviews, focus groups, document review, direct observation.
    • Often used to obtain information on processes, meanings, in-depth understanding.
levels types of qualitative information
Levels/Types of Qualitative Information
  • Different levels/types of information can be gathered from respondents.
  • Formulate questions that yield information regarding:
    • Reactions, feelings and emotions
    • Opinions and values
    • Knowledge and learning
    • Changes in skills
    • Behaviors/experiences
    • Effectiveness
    • Background/history/context

(Hancock 1998)

considerations in selecting a data collection method
Considerations in Selecting a Data Collection Method
  • Your evaluation or study question
  • Stakeholders’ desired sources of data
  • Resources (Financial and Skills)
  • Time (available to collect data)
  • Access to and availability of subjects/respondents
  • Information Collection Request (ICR)
interviews things to consider
Format

Structured

Semi-structured

Unstructured

Questions

Open-ended

Closed-ended

Sequencing

Location

In-person

Telephone

Duration

Selection of Interviewees

Equipment/Supplies

Recorder (tape or digital)

Laptop

Note Paper

Schedule

Interviewer Skills

Resources

Financial, Staff

Interviews: Things to Consider
interviews format
Interviews: Format
  • Structured Interview - Interviewer asks a specific set of questions of each respondent in the same way. This allows the interviewer to obtain a uniform set of data from each respondent.
  • Semi Structured- Includes a series of open ended questions based on the subject of interest to the interviewer but provides flexibility to explore issues in greater detail.
  • Unstructured Interview – General sets of questions are asked so that subjects respond in a free flowing manner resembling a conversation. The interview is designed to find out more information about a topic.

(Hancock 1998)

interviews questions
Interviews: Questions

Open-ended Questions:

Solicit additional information from the respondent and will require more than one or two word responses. Respondents are encouraged to explain their answers.

  • Advantages:
    • Respondents can provide more information about a subject.
    • Researchers can better understand respondents true feelings, reactions about an issue.
    • Allows for an unrestricted response
  • Disadvantages:
    • Time-consuming
    • Challenging for respondents that are less articulate

(Hancock 1998)

interviews questions cont d
Interviews: Questions (Cont’d)

Close-ended Questions:

Limit interviewee’s responses to a pre-existing set of answers e.g., yes/no, true/ false, or multiple choice with an option for other or a ranking scale response option can be used. Questions can be restrictive and can be answered in a few words.

  • Advantages:
    • More easily analyzed
    • Answers can be assigned a numerical value
    • Questions can be more specific
  • Disadvantages:
    • Can yield incomplete responses
    • Discourages disclosure
    • Results could be misinterpreted
interviews questions cont d1
Interviews: Questions (Cont’d)

Example 1:

  • Open-ended: Tell me about your relationship with the program’s Project Officer.
  • Closed-ended: Do you have a good relationship with the program’s Project Officer?

Example 2:

  • Open-ended?: Can you describe your satisfaction with the program?
  • Closed-ended?: How satisfied are you with the program?

□ Very satisfied

□ Somewhat satisfied

□ Dissatisfied

interviews location duration and schedule
Interviews: Location, Duration and Schedule
  • Location
    • Decide whether to conduct in-person or telephone interviews.
    • Select a time and place that is quiet and free of distractions.
  • Duration
    • Schedule the interview for no more than 1 hour.
  • Schedule
    • Leave ample time to review transcripts and notes after each interview (at least one hour).
interviews selection of interviewees how should participants be selected
Interviews: Selection of Interviewees How Should Participants Be Selected?
  • Snowball sampling: Identify a few members of the community of interest, and then ask them for additional contacts.
  • Contrasting cases: Select cases with high contrast to learn about what underlies the differences between them.
  • Typical cases: Select cases that appear to represent the average, normal, typical situation.
  • Critical cases: Select cases that are considered to be crucial to understanding the study/ evaluation topic or which are assumed to represent the perspective of many other cases.

(Kakoyannis 2007)

interviews equipment supplies needed
Interviews: Equipment/Supplies Needed
  • Equipment/Supplies
    • Note paper, recorder (tape or digital) or laptop to record/document responses
  • Note taking tips
    • Take good notes without detracting from the conversation
    • Write while maintaining eye contact
    • If interviewee says something you want to capture, it is OK to ask them to repeat it or to finish what you are writing before asking the next question.

(World Bank, Module 6: Data Collection Methods, Slides 53 and 54)

interviews skills and resources needed
Interviews: Skills and Resources Needed
  • Interviewer Skills
    • Identify an experienced Interviewer
    • Interviewer should be aware of any cultural norms: eye contact, direct questions, gender issues
    • Stick to the script:
      • If asking close-ended questions, ask exactly the way written.
      • If asking open-ended questions go with the flow, not too directive.
    • Avoid asking yes/no questions. Ask, how, who and why*
    • Don’t step outside of your role as an interviewer
    • Good listener
  • Resources
    • Ideally, have a second person to help take notes or use a recorder

*In some instances, when the interviewer consistently asks the respondent why, it may be interpreted as aggressive

(World Bank, Module 6: Data Collection Methods, Slides 53 and 54)

focus groups things to consider
Format

Group Size

Number of Groups

Questions

Open-ended

Closed-ended

Location

In-person

Conference Call

Duration

Selection of Focus Group Participants

Equipment/Supplies

Recorder (tape or digital)

Laptop

Note Paper

Schedule

Skills

Resources

Financial, Time, Staff

Focus Groups: Things to Consider
focus groups format and questions
Focus Groups: Format and Questions
  • Size
    • Recommended size of group is 6-10
    • Focus group members should have something in common
  • Number of Focus Groups
    • No rules here. However, more than one is recommended to ensure sufficient information is collected.
  • Questions
    • Start broad and then be specific

(Hancock 1998)

focus groups location duration and schedule
Focus Groups: Location, Duration and Schedule
  • Location
    • Comfortable, neutral, safe environment
    • Free from distractions and accessible
    • Setting: around a table or in a circle
  • Duration
    • Typically 1-2 hours (clear start and stop times)
  • Schedule
    • Piggy back on existing meetings/conferences
    • Do not over schedule: 2 or 3 in a day is plenty for one moderator/facilitator.
focus groups selecting participants
Focus Groups: Selecting Participants
  • May need to have homogeneous groups with respect to gender, race, social class, managers vs. staff etc.
  • Cultural norms are important.
  • Things to Consider:
    • What is the geographical spread of your potential participants?
    • Are there any specific inclusion criteria for selecting participants
    • Where or how could you obtain a list of potential participants?
    • Are there any pre-existing groups and what are the advantages and disadvantages of using members?

(World Bank , Module 6: Data Collection Methods, Slide 73), (Hancock 1998)

focus groups equipment supplies and resources needed
Focus Groups: Equipment, Supplies and Resources Needed
  • Equipment
    • Bring equipment and supplies needed to document/record the focus group. Note paper, recorder (tape or digital), laptop
  • Resources
    • Facilitator and note-taker
  • Other
    • Consider providing food, incentives, childcare, transportation etc., to respondents
focus groups skills needed
Skilled facilitation is essential.

Facilitator should know the script so focus group appears conversational.

Ensure that everyone is heard.

Ask: “What do other people think?”

State: ”We have heard from a few people, do others have the same views or different views?”

Active listener

Develops and adheres to ground rules

Accept all views while managing differences of opinion.

So we have different perspectives

Probe for elaboration

Tell me more.

Manage time

Closing off discussion and moving to next topic.

Invisible: say as little as possible

Let conversation flow across the table with minimal direction.

Keep personal views outside the room.

Focus Groups: Skills Needed

(World Bank, Module 6: Data Collection Methods, Slides 78, 80 and 81

survey questionnaire open ended questions things to consider
Format

Type of Questions

Open-ended

Closed-ended

Method of Administration

Self-Administered vs. Guided by Interviewer

Mail, Telephone, Electronic, In-person

Length

Duration

Consider 10 – 20 minutes

Confidentiality

Response rate

May decrease if mailed.

Survey/Questionnaire (Open-ended questions): Things To Consider
document file review
Meeting minutes

Organizational mission statements

Letters, records and laws

Memoranda

Correspondence

Official publications and reports

Personal diaries

Photographs and memorabilia

Progress reports

Studies

Document/File Review

Collection and examination of documents produced in daily life as a means for better understanding the values of people in the study.

document file review things to consider
Document/File Review: Things To Consider
  • Type of information
    • What data are you looking for? (context, process, outcome, satisfaction)
  • Accuracy
    • Were data accurately recorded? Is it trustworthy? Has it undergone QA?
  • Access/Availability
    • Is permission needed to access files?
    • Are files in a central location or dispersed geographically?
  • Completeness
    • Are data available for appropriate years, stakeholders
document file review things to consider1
Document/File Review: Things To Consider
  • Confidentiality
    • Can data be shared publicly? Do legal restrictions exist? (e.g., CBI, personnel data)
  • Informative
    • Will data collected from the files help provide information to answer the study question?
  • Time
    • Does the volume of documents/files increase the level of effort needed to complete the review?
document file review1
Document/File Review
  • Advantages:
    • Unobtrusive
    • Analysis can easily be replicated because the data are stable
    • Documents can allow broader coverage of data by giving insight into past events that form the context within which the current study is operating in
    • Often less expensive and faster than collecting original data
  • Disadvantages:
    • Difficult to access and retrieve certain documents
    • Data gaps exist
    • Data do not explain why something is occurring/ happening
    • Data may not be “exactly” what is needed
    • Selection of documents might be biased if researcher does not collect a broad range of data
slide33
Exercise 1:

Selecting a Qualitative Data Collection Method

tribal gap data collection efforts
Tribal GAP Data Collection Efforts
  • Reviewed a sample of files for 111 Tribes in 9 EPA Regions w/ Federally Recognized Tribes
    • GAP Accountability Tracking System
    • Grants Information and Control System
    • Audit Database
    • Strategic Goals Reporting System
  • Reviewed Regional Files (e.g., quarterly reports submitted by Tribes)
  • Conducted Interviews with GAP Project Officers in 8 Regions
  • Organized Panel Discussions w/Tribal Representatives
    • United South and Eastern Tribes (USET) Impact Week, Arlington, VA
    • EPA Region 5, Indian GAP Conference, Chicago, IL
    • EPA Region 8, Tribal Operations Committee, Denver, CO
qualitative data analysis
Qualitative Data Analysis

Analysis and interpretation are employed to bring meaning, order, and understanding to the data. (Taylor-Powell and Renner 2003)

The purpose of qualitative data analysis is to describe, interpret, explain and understand data that are collected. (Dey 1993)

what is content analysis
What is Content Analysis?

A systematic process for identifying themes and patterns in the data, coding and characterizing the themes in order to understand the issue being studied. (Russ-Eft and Preskill 2001)

steps for analyzing qualitative data
Steps for Analyzing Qualitative Data

Step 1: Focus the analysis

Step 2: Get to know the data

  • Refocus the analysis if necessary

Step 3: Create Code/Categorize the data

  • Check validity of codes

Step 4: Identify patterns and themes using codes

  • Check categorization of coding

Step 5: Interpret the data

Step 6: Conduct member check

(Taylor-Powell and Renner 2003), (McNamara 1998)

step 1 focus the analysis
Step 1: Focus the analysis
  • Review the purpose of the evaluation
  • Review the key study questions
    • Using the research question as a guide, think about which parts of the text help inform that question
  • Consider a framework for analyzing the data
      • Processes – Data are organized to describe an important process
      • Issues – Data are organized to illuminate key issues (often equivalent of primary evaluation questions)
      • Questions – Responses to data are organized question by question
      • Concepts – Data organized by key concepts

(Patton 2007)

step 1 focus the analysis1
Step 1: Focus the Analysis
  • Inductive analysis
    • Involves discovering patterns, themes, and categories in one’s data. Findings emerge out of the data, through the analyst’s interactions with the data.
  • Deductive analysis
    • Involves analyzing data according to an existing framework, e.g., the program’s logic model.
  • Use both
    • Build on the strengths of both kinds of analysis. For example, once patterns, themes, and/or categories have been established test the appropriateness of the categories.

(Patton 2007)

step 2 get to know the data
Step 2: Get to know the data
  • Transcribe the data
    • Listen to audio/recorded tapes
    • Read notes and develop a transcript
  • Read through the transcript first as a whole
    • Make brief notes (in the margin) of interesting or relevant information you are seeing in the data

(McNamara 1998)

step3 create codes and categorize the data
Step3: Create codes and categorize the data

Codes are labels, abbreviations or symbols that are used to identify a particular concept, theme, idea or behavior reflected in the data.

Coding involves breaking down, labeling, comparing and organizing data in order to group them into similar categories.

(Taylor-Powell and Renner 2003)

step3 create codes and categorize the data1
Step3: Create codes and categorize the data
  • Preset Categories:
    • Start with a list of themes or categories in advance, and then search the data for these topics.
  • Emergent Categories:
    • Rather than using preconceived themes or categories, you read through the text and find the themes or issues that recur in the data.

(Taylor-Powell and Renner 2003)

step 3 create codes and categorize data
Step 3: Create codes and categorize data
  • Review margin notes, and make a list of the different types of information found.
  • Review the list of data items and categorize them in a way that describes what it is about.
  • Categorize the code words into similar groups
    • As you read, add or modify the descriptive code words so they better reflect the newer data.
    • Consider whether they can be linked in some way. Develop major and minor categories if needed
  • Examine the list of minor and major categories of data. Compare and contrast the categories.

(Taylor-Powell and Renner 2003)

step 4 identify patterns and themes
Step 4: Identify patterns and themes
  • Identify common, recurring patterns and themes, ideas, words or phrases
    • Look for associations, connections and causal relationships in the themes
  • Display summaries of data to enhance/illuminate interpretation e.g., compilation sheets, flowcharts, diagrams, matrices;

(Taylor-Powell and Renner 2003), (McNamara 1998)

step 5 interpret the data
Step 5: Interpret the data
  • Reflect on the themes and patterns and data collected to make sense of the data and to find meaning and significance
  • Draw conclusions
  • If possible relate these to other data sets

(Taylor-Powell and Renner 2003), (McNamara 1998)

step 6 conduct member check
Step 6: Conduct member check
  • Share theories and conclusions with respondents to verify the accuracy of your interpretation
slide48
Exercise 2:

Analyzing Qualitative Data

assessing rigor of qualitative data
Assessing Rigor of Qualitative Data

Demonstrating data analysis is rigorous is important given criticism and skepticism associated with qualitative data. The rigor of qualitative data may be addressed by assessing:

  • Reliability (of the methods employed)
  • Validity (of the interpretation of the data)
    • Internal validity (credibility) – Extent to which the findings are credible and the “reality” that is described are credible to the people interviewed.
    • External validity (transferability) – Extent findings can be generalized to a larger population of people, settings, or situations.
  • Objectivity

(Lacey and Luff 2001)

strategies for increasing reliability and validity
Strategies for Increasing Reliability and Validity

Reliability

  • Describe the approach to and procedures for data analysis
  • Clearly document the process of generating themes, concepts or theories

Validity

  • Consider and discuss alternative interpretations of the findings
  • Carefully consider and discuss cases and data that don’t fit overall patterns and themes,
  • Triangulate the analysis (use of multiple data sources)

(Lacey and Luff 2001), (Patton 2007)

strategies for increasing reliability and validity1
Strategies for Increasing Reliability and Validity
  • Check for representativeness of data
  • Check of bias
  • Cross-check data with evidence from other, independent sources
  • Compare and contrast data
  • Use extreme (groups of) informants to the maximum.
  • Do additional research to test the findings of your study.
  • Respondent validation (Get feedback from your informants.
  • Triangulate analysis, methods, sources

Module 23: Analysis of Qualitative Data, International Development Research: http://www.idrc.ca/en/ev-56451-201-1-DO_Tpoic.html, pg. 3-28

increasing validity through triangulation
Increasing Validity through Triangulation
  • Triangulation Options:
    • Check out the consistency of findings generated by different data-collection methods, i.e., methods of triangulation
    • Check out the consistency of different data sources within the same method, i.e., triangulation sources
    • Use multiple analysts to review findings , i.e. analyst triangulation; and
    • Use multiple perspectives or theories to interpret the data, i.e. theory/perspective triangulation

(Patton 2007)

contact
Contact

Yvonne M. Watson

202-566-2239

watson.yvonne@epa.gov

appendix
Appendix
  • Interviews: General Guidelines
  • Focus Group: General Guidelines
  • Interviews and Focus Groups: Final Thoughts…
  • Survey/Questionnaire: General Guidelines
  • Document/File Review: General Guidelines
interviews general guidelines
Interviews: General Guidelines
  • Define purpose
    • Link to study/evaluation objectives
  • Decide whether you want to ask open-ended or closed-ended questions
  • Draft interview questions
    • Sequence questions so they flow
  • Prepare introduction and closure
    • Purpose of the interview
    • How and why interviewees were selected
    • Close with asking whether interviewees have questions or comments
    • Thank you and follow-up
  • Prepare a record of responses
  • Pre-test the instrument

(World Bank, Module 6: Data Collection Methods, Slides 47 and 48)

interviews general guidelines1
Interviews: General Guidelines
  • Let interviewees know:
    • Why they are being interviewed
    • How they were selected
    • How the data will be used
    • Whether it is confidential
    • How long the interview will take
    • Whether you might want to talk to them again
  • Additional touches:
    • Share interview questions ahead of time.
      • - No surprises.
    • Offer to share a summary of what you understand from the interview
      • This might be especially useful to give the interviewee (especially if high ranking official) a greater feeling of control.
    • Thank you note afterwards.

(World Bank, Module 6: Data Collection Methods, Slides 49 and 50)

interviews general guidelines2
Interviews: General Guidelines
  • Every word and idea is valuable.
  • Take time to write up notes as carefully and in-depth possible.
  • Do at least a brief clean-up of notes immediately afterwards (leave an hour between interviews).
  • Write up full notes within a day of the interview: memory decay sets in quickly.

(World Bank, Module 6: Data Collection Methods, Slides 53 and 54)

focus group general guidelines
Focus Group: General Guidelines
  • Introduction:
    • Purpose of focus groups
    • Sponsor
    • Why participants were selected
    • How the information will be used
    • The ground rules
    • Overview of the process
  • Have participants introduce themselves
  • First question: easy, ice-breaker.
  • Ask main questions.
  • Last questions:
    • Summary question: most important think that was said here that we should take with us.
    • Other comments or questions?

(World Bank, Module 6: Data Collection Methods, Slides 78 and 79)

focus group general guidelines1
Focus Group: General Guidelines
  • Write up impressions immediately after focus group: major issues and major points of discussion.
  • Compare notes with your partner.
  • Ideally, the focus group tape will be transcribed verbatim.
  • If not, listen to the tape afterwards while writing in-depth notes.
    • You will be surprised how much you did not hear during the actual focus group.
  • Leave time to prepare write-up immediately following the focus group.
  • Capture anything unusual that happened during the focus group.

(World Bank, Module 6: Data Collection Methods, Slides 83 and 84)

focus group ground rules
Focus Group: Ground Rules
  • What is said here, stays here.
  • Everyone is encourage to participate but note everyone has to answer every question.
  • Respect different viewpoints.
  • There are no right or wrong answers.
  • Only one person speaks at a time.

(World Bank, Module 6: Data Collection Methods, Slide 78 and 79)

interviews and focus groups final thoughts
Interviews and Focus Groups: Final Thoughts…

When gathering data from people:

- Keep it simple, clear, easy, short

- Respect respondents time and intelligence

- Tell them how they were selected and why their participation is important

- Do no harm: keep responses confidential

Consider privacy issues- permission needed to access files?

  • Is permission needed to record the session?
  • Are you planning to attribute comments directly to individual interviewees?
survey questionnaire
Survey/Questionnaire
  • Advantages:
    • People are familiar with surveys
    • Some respondents prefer surveys to interviews
    • Can reach respondents in several geographic locations
  • Disadvantages:
    • Respondents may not complete the survey
    • Can’t probe for additional information/details from respondents
    • Respondents can misinterpret questions when a set of choices is not available.
    • Increased number of open-ended questions may lower the response rate.

(Russ-Eft and Preskill 2001)

survey questionnaire general guidelines
Survey/Questionnaire: General Guidelines
  • Include a brief introduction statement
  • Avoid abbreviations and acronyms
  • Avoid biases words and phrases
  • Exercise caution when asking about personal information
  • Assess personal biases of the interviewer
  • Ensure only one thought is expressed in each question

(Fink and Kosecoff 1998)

document file review general guidelines
Document/File Review: General Guidelines
  • Review the evaluation/study question of interest.
  • Identify the type of information needed to answer the study question and identify a code for each possible answer.
  • Determine which documents/files contain the information needed to answer the question.
  • Develop a data collection form, instrument/matrix or table that will assist you in collecting the specific information needed.
    • The instrument should be clear, simple to use and code
  • Establish procedures for using the instrument
  • Conduct training to ensure everyone codes the same way.
  • Review documents/files.
  • Have a second person review the files.
resources publications
Resources: Publications
  • Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change. 2001. Russ-Eft, D. and H. Preskill. Cambridge, MA: Perseus Publishing.
  • Real World Evaluation: Working Under Budget, Time, Data, and Political Constraints. Bamberger, M., Rugh, J. and L. Mabry. 2006. Thousand Oaks, CA: Sage Publications.
resources websites and online texts
Resources: Websites and Online Texts
  • International Program for Development Evaluation Training (IPDET) Course Modules. http:///www.worldbank.org/ieg/ipdet/modules.html
  • University of Wisconsin-Extension Cooperative Extension, Madison, Wisconsin, Program Development & Evaluation, Analyzing Qualitative Data. G3658-12, Taylor-Powell, E. and Renner M. 2003. http://learningstore.uwex.edu/pdf/G3658-12.pdf
  • Online QDA (Qualitative Data Analysis) http://onlineqda.hud.ac.uk/Intro_QDA/index.php
references
References
  • Hancock B. Trent Focus for Research and Development in Primary Health Care: An Introduction to Qualitative Research. Trent Focus, 1998
  • Kakoyannis C. Qualitative Data Collection & Analysis Notes. 2007
  • Lacey, A. and Luff D. Trent Focus for Research and Development in Primary Health Care: An Introduction to Qualitative Analysis. Trent Focus, 2001
  • McNamara, C. 1998. Basic Guide to Program Evaluation: Analyzing and Interpreting Information. Available online at:http://www.managementhelp.org/evaluatn/fnl_eval.htm; accessed August 126, 2005).
  • Patton, M.Q. 2007. 2007 American Evaluation Association Qualitative Methods Workshop, November 5-6, 2007. Baltimore, Maryland.
  • Russ-Eft, D. and H. Preskill. 2001. Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, and Change. Cambridge, MA: Perseus Publishing.
  • University of Wisconsin-Extension Cooperative Extension, Madison, Wisconsin, Program Development & Evaluation, Analyzing Qualitative Data. G3658-12, Taylor-Powell, E. and Renner M. 2003. http://learningstore.uwex.edu/pdf/G3658-12.pdf
  • The World Bank Group. Carleton University, IOB/Ministry of Foreign Affairs, Netherlands. International Program for Development Evaluation Training (IPDET), Module 6. Data Collection Methods. Power Point Slides and Narrative Text
  • The World Bank Group. Carleton University, IOB/Ministry of Foreign Affairs, Netherlands. International Program for Development Evaluation Training (IPDET), Module 8. Data Analysis and Interpretation. Power Point Slides and Narrative Text