1 / 45

21 tips for conducting evaluation research on a boot lace

Learn valuable advice on how to conduct evaluation research, including setting objectives, choosing survey methods, and measuring impacts. Get help from university departments, market research associations, and online resources.

Download Presentation

21 tips for conducting evaluation research on a boot lace

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. David Geddes, Ph.D. Senior Vice President & Partner Fleishman-Hillard Institute for Public Relations Measurement Commission 21 tips for conducting evaluation research on a boot lace

  2. Today’s objectives • Advice on doing it yourself • Advice on being a good client

  3. Where to get help • University departments • Social work, Sociology, Psychology, Political Science, Business, and others • Contacts via board of directors • American Marketing Association • Market Research Association • Market research firms • LinkedIn (www.linkedin.com) • National Opinion Research Center (University of Chicago) http://www.whatisasurvey.info/

  4. http://www.italladdsup.gov/community_partners/dc_teachmehow_02.asphttp://www.italladdsup.gov/community_partners/dc_teachmehow_02.asp • Introduction to evaluation research • Process evaluation examples • Telephone survey with annotations • Intercept and self-administered survey

  5. Build a community • Vision • Lutheran Social Services Research Community • Share best practices • Methods • Facebook or other collaboration tool

  6. 1 Set your objectives • Program objectives • Specific information needs

  7. 2 Do your background research

  8. 3 Think about different types of evaluation and measurement • Formative evaluation • Helps planning • Process evaluation • How well is the program operating? • Impact evaluation • Building awareness and understanding? • Outcome evaluation • Concrete program or “business” metrics

  9. 4 Don’t assume you know audience attitudes and opinions • You need formative research

  10. 5 Set up your process metrics

  11. 6 Measure impacts • Awareness • Knowledge • Attitudes and perceptions • Intended actions

  12. Known respondents Population =Sample frame Members Donors Clients Employees Civic organization leaders Defined population Sample frame derived from population General public Congregation members Potential donors Potential adoptive parents 7 Identify survey targets

  13. 8 Pick the right survey method to reach targets • Telephone • Mail • Online • Mall or store intercept

  14. Telephone surveys • Advantages • Speed, in some cases • Control • Probe and clarify • Cost if you do it yourself • Disadvantages • Cost if you go outside • Hard to reach some socio-economic groups

  15. Telephone surveys • Case: General public, 10 minute survey of 500 adults in Saint Louis metro area. • $8,000 = $16 cost per interview • Review cost parameters • Case: Survey of 200 adult children or guardians of seniors who receive services from LSS in St. Louis metro area. LSS provides names and phone list. • $4,000= $20 cost per interview • Review cost parameters

  16. Telephone surveys • Case: Survey of 200 married adults in the St. Louis metro area aged 18 to 45 who are at least somewhat open to the idea of adoption. 10 minute survey. 30% incidence based on other surveys. • $7,000 = $35 cost per interview

  17. Doing your own telephone surveys • Sample list • Staff • Interns, volunteers, students, etc. • Call management system • Data recording system • Discipline • Persistence

  18. Going outside for telephone surveys • Smaller local firms • Be flexible on timing

  19. Mail surveys • Advantages • Lower cost • Easy if you have a mailing list • Easier to do yourself • Disadvantages • Slow • Lower control • Lower response rates • Need a mailing list

  20. Doing your own mail surveys • Clear questionnaire layout • Sample list • Staff • Interns, volunteers, students, etc. • Lots of manual paper handling • Discipline • Patience

  21. Online surveys • Advantages • Speed, in some cases • Control • Cost if you do it yourself (SurveyMonkey) • Disadvantages • Purchase a sample list • Unrepresentative of some socioeconomic groups

  22. Online surveys • Case: General public, 10 minute survey of 500 adults in Saint Louis metro area using an online survey tool. • $3,300 = $6.60 cost per interview • Review cost parameters

  23. Doing your own Online surveys • SurveyMonkey programmer • Sample list • Staff • Interns, volunteers, students, etc.

  24. Mall intercept surveys • Advantages • Control • Probe and clarify • Cost if you do it yourself • Disadvantages • Slow • Need mall or store cooperation • Unrepresentative of some socioeconomic groups

  25. Doing your own mall intercept surveys • Cooperation of mall or Wal-Mart • Staff • Interns, volunteers, students, etc. • Discipline • Patience .. Figure on one interview/hour

  26. 8a Select an appropriate sample size • What are possible sources of error? • How far off could we be? • What sample size do we need?

  27. 9 Give advance notice • If you know the target respondents • Newsletter • E-mail • Postcard • Benefit: improves response rate

  28. 10 Remember you are asking for their time • This applies to… • Pre-survey letters or cards • Introduction to the survey • And the questions themselves • Don’t ask meaningless questions

  29. 11 Write a good introduction to the questionnaire • Introduce yourself • “Research” not “survey” • Survey topic: social services • Not a sales call … not a fundraising call • You want their opinion / input

  30. 12 Move from general to specific

  31. 13 Lead with an interesting question • Engage the respondent • Ask interesting questions and people will talk

  32. 14 Make judicious use of open-ended questions • Example: • “Based on what you have read or heard, what are your personal perceptions of Lutheran Social Services? [PROBE] Anything else?” • 2 to 3 per questionnaire maximum • Finish with a “final thoughts” question

  33. 15 Use good response scales • Building a scale • Clearly distinguish good and bad • 1 Agree • 2 Disagree

  34. Use good response scales • Building a scale • Add a polite positive and negative • 1 Strongly Agree • 2 Agree • 3 Disagree • 4 Strongly disagree

  35. Use good response scales • Building a scale • Add a top positive response • 1 Very strongly agree • 2 Strongly Agree • 3 Agree • 4 Disagree • 5 Strongly disagree

  36. Use good response scales • Building a scale • Add a don’t know/no opinion • 1 Very strongly agree • 2 Strongly Agree • 3 Agree • 4 Disagree • 5 Strongly disagree • 6 Don’t know / no opinion

  37. Use good response scales • Building a scale • Add a mid-point if you really need it ... and balance the scale • 1 Very strongly agree • 2 Strongly Agree • 3 Agree • 4 Neither agree nor disagree • 5 Disagree • 6 Strongly disagree • 7 Very strongly disagree • 8 Don’t know / no opinion

  38. 16 Make sure you cover all possible responses • Multiple choice questions

  39. 17 “No opinion” does not equal “Don’t know”

  40. 18 Always include a “refused” option

  41. 19 Be careful with numeric responses • “About how much do you spend on gas each month” $__________ • Which of the following best describes the amount you spend on gas each month: • Under $25 • $26 to $50 • $51 to $100 • Over $100

  42. 20 Be careful about “do you floss your teeth” items • Socially desirable responses • Responses that may imply guilt

  43. 21 Test your questionnaire

  44. Wrap-up • Begin with your objectives and information needs • Define the people you want to survey • Select an appropriate survey method • Write good questions • Test • Launch the survey • Analyze the results

  45. Exercise

More Related