Tell me what to ask Designing Surveys: Practice makes pretty good Research & Evaluation Strategic Planning & Implementation Kamehameha Schools Presented by: Brandon Ledward, Wendy Kekahio, Summer Keli’ipio, Nolan Malone and Kathy Tibbetts 10 December 2007
Activity 1 Please stand if you’ve ever: (No shame! Get prizes, you know.)
Agenda • Activity 1 • How do we collect data? • Activity 2 • Before you get started • Activity 3 • Creating a survey • Item Types • Activity 4
Quantitative Surveys/evaluations Forms Roll books/ participation counts Assessments/tests Collaborators Qualitative Observations Interviews Artifacts Open-ended survey questions How do we collect data? Do you really need/want a survey?
Activity 2 • Envelope contains 5 scenarios • What method of data collection would you use? Why? • Interview • Focus Group • Survey • Observation There’s no wrong answer
Before you get started What is a survey? • Method of gathering information • Sample vs. Census • Every individual is asked the same question in more or less the same way • Create a composite profile of a population • Types of surveys: • Mail • Telephone • In-person (“paper & pencil” or “show of hands”) • Internet
Before you get started How do I create a good survey? The quality of a survey is largely determined by its purpose and the way it is conducted. Scheuren, Fritz. “What is a Survey?” 2004.
Before you get started How do I begin? • Focus on the purpose of the survey • Where does the purpose come from? • See Workshop #1: Logic Models Are Our Friends • Know how you will use the information • Allocate resources accordingly • Do you need to give makana? • What time of day is best? • Can the survey take a long time?
Activity 3 • With someone at your table discuss one program outcome you each wish to measure • Why do you need to measure this? • How are you going to use the information? • What are some challenges? Be prepared to share with the group
Creating a survey Step 1: Define the purpose • Why do you need a survey? • What are you going to do with the information? (i.e. reporting) • Have you verified the purpose with leadership and staff?
Creating a survey Step 2: Plan the administration of the survey • What method? (mail, in-person, internet, telephone, etc.) • How many people will take the survey? • How long will the survey take? • When will the survey be administered? • What logistical issues should be considered? (age group, materials, makana)
Creating a survey Step 3: Write questions & then pilot those questions Step 4: Review & revise your questions and fully implement your survey easier said than done! Checklist Revisions Approval START!
Survey Basics: Item Types Two item types: • Open-ended • Close-ended • Ratings • Multiple choice(s)
Survey Basics: Item Types Examples: Open-ended questions 1. What was the single best part of your experience today? ______________________________ ______________________________ 2a. Did participating in this program impact your future career goals? • Yes • No • Not sure 2b. Please explain: ______________________________ ______________________________ ______________________________
Strongly Disagree Strongly Agree 1 2 3 4 5 6 7 8 9 10 Survey Basics: Item Types Examples: Close-ended questions Rating scales 1. Rate each activity: 2. I learned educational strategies I can use in my classroom.
Survey Basics: Item Types • Examples: Close-ended questions Multiple Choice(s) • What language is primarily spoken in • your home? (Choose one only). ○ English ○ Pidgin ○ Hawaiian ○ Some other language (specify):__________
Activity 4: Writing your questions Go back to the program outcome that you wanted to measure: Write down 3-5 questions to measure your program outcome. With someone at your table, discuss how you would turn one of these questions into a survey item. Be prepared to share with the group
Survey Basics: Reporting • Begin at the first step • If you cannot find a reason to report a piece of information don’t waste a question on your survey • Know beforehand how info will fit into your report • Be brief in reporting all findings
Mahalo a nui loa Additional questions? Brandon Ledward Summer Keli‘ipio Wendy Kekahio 541-5325 534-3964 523-6373 email@example.com firstname.lastname@example.org email@example.com Nolan Malone Kathy Tibbetts 534-3849 541-5321 firstname.lastname@example.org email@example.com