Writing survey questions
Download
1 / 47

- PowerPoint PPT Presentation


  • 214 Views
  • Updated On :

Writing Survey Questions. Research Methods for Public Administrators Dr. Gail Johnson. Survey Options: Amount of Structure . Structured Close-ended questions Semi-structured Open-ended questions. Structured Questions.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about '' - kaia


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Writing survey questions l.jpg

Writing Survey Questions

Research Methods for Public Administrators

Dr. Gail Johnson

Dr. G. Johnson, www.reseachdemystified.org


Survey options amount of structure l.jpg
Survey Options: Amount of Structure

  • Structured

    • Close-ended questions

  • Semi-structured

    • Open-ended questions

Dr. G. Johnson, www.reseachdemystified.org


Structured questions l.jpg
Structured Questions

  • How satisfied or unsatisfied are you with the graduates of the teachers college?

    • Very Unsatisfied <---- > Very Satisfied

  • How helpful or unhelpful have the agricultural consultants been in working with you in the past year?

    • Very Unhelpful <------- > Very helpful

  • How useful, if at all, has the program evaluation workshop been in helping your evaluate your program?

    • Of little usefulness <------- > Very Useful

Dr. G. Johnson, www.reseachdemystified.org


Semi structured questions l.jpg
Semi-Structured Questions

  • What are the greatest assets you have observed in the MPA graduates?

  • What knowledge and skills should MPA graduates have for the 21st century?

  • What are the three things that you learned from the MPA program that you use most frequently?

Dr. G. Johnson, www.reseachdemystified.org


Structured vs semi structured which to choose l.jpg
Structured vs Semi-structuredWhich to Choose?

  • Structured, check-a-box questions are easier for participants to complete and easier for researchers to analyze.

    • But hard to develop and sometimes feels impersonal.

    • Unforgiving of mistakes: if you ask the wrong question, use the wrong word or have a type or grammatical error on a written survey: there is no way to change it. You have to live with it.

Dr. G. Johnson, www.reseachdemystified.org


Structured vs semi structured which to choose6 l.jpg
Structured vs Semi-structuredWhich to Choose?

  • Semi-structured, open-ended questions are harder for participants to complete because it takes much more time to write-in the answers.

    • But they get to tell their story in their own words.

  • Harder for researchers to analyze because the English words can have many meanings

    • But it may provide researchers with new insights: ideas and views that had not considered

Dr. G. Johnson, www.reseachdemystified.org


Structured vs unstructured which to choose l.jpg
Structured vs UnstructuredWhich to Choose?

  • My advice:

    • If you have a lot of open-ended questions, better to interview in person than ask them to complete a self-administered survey.

Dr. G. Johnson, www.reseachdemystified.org


Writing closed ended survey questions l.jpg
Writing Closed-Ended Survey Questions

  • Harder than it might appear

  • Takes more time that people think

  • Requires an in-depth knowledge of the topics to be covered

  • Requires a comprehensive awareness of the words that people are likely to use when talking about the topics

Dr. G. Johnson, www.reseachdemystified.org


Writing closed ended survey questions9 l.jpg
Writing Closed-Ended Survey Questions

  • “Every questionnaire, must finally be handcrafted. It is not only that questionnaire writing must be ‘artful’; each questionnaire is also unique, an original.”

    --Converse and Presser

Dr. G. Johnson, www.reseachdemystified.org


Close ended check a box l.jpg
Close-ended: Check a Box

  • Single question with a response set:

  • How frequently, if at all, have you visited the community center in the last 3 months?

    • Did not visit (0 times)

    • 1-3 times

    • 4-6 times

    • 7-9 times

    • 10 or more times

Dr. G. Johnson, www.reseachdemystified.org


Close ended check a box11 l.jpg
Close-ended: Check a Box

  • Single question with a response set:

  • Including all sources of income, what was your gross household income last year?

    • Less than $15,000

    • $15,000 to $25,000

    • $25,001 to $50,000

    • $50,0001 to $75,000

    • $75,001 to $100,000

    • $100,001 to $125,000

    • $125,001 to $150,000

    • Greater than $150,000

Dr. G. Johnson, www.reseachdemystified.org


Using intensity scales l.jpg
Using Intensity Scales

  • Multiple-choice answers to survey questions are sometimes called response sets

  • Intensity scales are better than simple agree or disagree responses, better than simple yes or no responses

  • Intensity scales provide a way to gage the intensity of feeling:

    • Distinguish between strong feelings from the mediocre/whatever feeling

Dr. G. Johnson, www.reseachdemystified.org


One way intensity scale l.jpg
One-Way Intensity Scale

  • One-Way Scale: tries to capture a single dimension (or construct) that goes from low to high with a meaningful middle category

    • Always or Almost Always

    • Mostly

    • About Half of the Time

    • Occasionally

    • Never or Almost Never

Dr. G. Johnson, www.reseachdemystified.org


Soften the ends of the scales l.jpg
Soften the Ends of the Scales

  • Soften the ends of the scales by phrasing “Always or Almost always” and “Never or Almost never”

  • You want to encourage people to use the full range of the scale

    • It is rare for something to be absolutely never or always

      • I almost never watch TV but it would be untrue to say that I never watch TV

      • In my mind, I watch TV less than occasionally but that would be truer than saying never

      • Absolute ends limits the usefulness of using a scale

Dr. G. Johnson, www.reseachdemystified.org


A goldilocks intensity scale l.jpg
A Goldilocks Intensity Scale

  • Would you say the amount of elective courses offered are too many, too few or about right?

    • Much too many

    • Somewhat too many

    • About right

    • Somewhat too few

    • Much too few

Dr. G. Johnson, www.reseachdemystified.org


A goldilocks intensity scale16 l.jpg
A Goldilocks Intensity Scale

  • “About right” is meaningful information—if fact, the program director would want to see a majority of students reporting that the number of elective courses are “about right”

Dr. G. Johnson, www.reseachdemystified.org


Intensity scales with a neutral middle l.jpg
Intensity Scales with a Neutral Middle

  • Likert Scale

    • Strongly Agree

    • Agree

    • Neither Agree or Disagree

    • Disagree

    • Strongly Disagree

Dr. G. Johnson, www.reseachdemystified.org


Intensity scales with a neutral middle18 l.jpg
Intensity Scales with a Neutral Middle

  • Effectiveness/Ineffectiveness Scale

    • Very Effective

    • Generally Effective

    • Neither Effective Or Ineffective

    • Generally Ineffective

    • Very Ineffective

Dr. G. Johnson, www.reseachdemystified.org


Survey intensity scales l.jpg
Survey: Intensity Scales

  • Yes/No Intensity Scale

    • Definitely Yes

    • Probably Yes

    • Neither Yes or No

    • Probably No

    • Definitely No

Dr. G. Johnson, www.reseachdemystified.org


Exits no opinion not applicable don t know l.jpg
Exits: No Opinion, Not Applicable, Don’t Know

  • Questionnaires/survey instruments are designed for everyone to answer

  • But sometimes people are asked questions that they cannot answer because the question does not apply or they truly have no opinion

Dr. G. Johnson, www.reseachdemystified.org


Exits no opinion not applicable don t know21 l.jpg
Exits: No Opinion, Not Applicable, Don’t Know

  • For example, all MPA students might be asked about the quality of elective courses but some may have not yet taken any

  • You do not not want them offering an opinion

Dr. G. Johnson, www.reseachdemystified.org


Exits no opinion not applicable don t know22 l.jpg
Exits: No Opinion, Not Applicable, Don’t Know

  • So, you add an exit: “no opinion, or not applicable, or don’t know” to the response set

  • Do not ever delude yourself into thinking you can force people to answer your question if you do not give them an exit

    • Remember, their participation is always voluntary!!

Dr. G. Johnson, www.reseachdemystified.org


Why exits are needed l.jpg
Why Exits are Needed

  • The middle category conceptually is trying to capture an opinion.

  • A person may not really care, one way or another, if the MPA program decides to change its name. The person is indifferent but that is still an opinion

  • The exit allows the person to say I have no information about the proposed name change to have an opinion. They are not indifferent.

  • Subtle distinction but in some situations it might matter

Dr. G. Johnson, www.reseachdemystified.org


Adding exits to scales l.jpg
Adding “Exits” to Scales

  • Likert Scales

    • Strongly Agree

    • Agree

    • Neither Agree or Disagree

    • Disagree

    • Strongly Disagree

    • Don’t know, No Opinion

Dr. G. Johnson, www.reseachdemystified.org


Do exits work l.jpg
Do Exits Work?

  • Sometimes

    • At best, the provide

      • A way for people to complete the survey with minimal frustration (and one goal is to make it easy for people to complete the survey)

      • Meaningful information (and avoid junk data)

Dr. G. Johnson, www.reseachdemystified.org


Do exits work26 l.jpg
Do Exits Work?

  • But some people may not want to admit they don’t know

    • People have been asked their opinions about public programs that do not exist

      • Rather than say “don’t know”, they find the middle category: “I give a 3”; This is junk data

      • This suggests that analysts need to be very careful about how they analyze middle categories

Dr. G. Johnson, www.reseachdemystified.org


Big exits skip questions l.jpg
Big Exits: Skip Questions

  • Researchers can use skip questions when they want to target a handful of questions to specific participants.

    • For example, in an organizational survey, the researchers might want to ask a few questions that specifically apply to supervisors.

Dr. G. Johnson, www.reseachdemystified.org


Big exits skip questions28 l.jpg
Big Exits: Skip Questions

  • They ask: do you supervise people.

    • This is a time to use a simple yes or no response set.

  • If yes, please answer the following 5 questions (questions 20-25).

  • If no, please skip to question 26.

Dr. G. Johnson, www.reseachdemystified.org


Big exits skip questions29 l.jpg
Big Exits: Skip Questions

  • Another example: if conducting a citizen survey, the researcher might ask:

    • Did you use the library in the past year?__Yes ___ No

      • If yes, please answer questions 10-13.

      • If no, please skip to question 14.

Dr. G. Johnson, www.reseachdemystified.org


Writing good closed ended questions l.jpg
Writing Good Closed-Ended Questions

  • Give people permission to provide answers on either side of scale

    • Phrasing can seem awkward but needed to avoid bias:

      • “How likely or unlikely are you to vote in the election” sends the signal that it is OK to answer either likely or unlikely.

Dr. G. Johnson, www.reseachdemystified.org


Avoid ranking questions l.jpg
Avoid Ranking Questions!!

  • They appear to be easy and meaningful but appearances are deceiving

  • They are very hard to analyze

  • They give very limited information

  • They can even distort information

  • See text for my full and complete rant!!

Dr. G. Johnson, www.reseachdemystified.org


Writing survey questions my best advice l.jpg
Writing Survey Questions: My Best Advice

  • Locate surveys others have used and see if there are questions that will work for your purpose

  • Use clear, simple language

  • Encourage a range of responses using a 5-point scale (or even a 7-point scale)

    • Avoid simple yes or no responses when asking about opinions

Dr. G. Johnson, www.reseachdemystified.org


Writing survey questions my best advice33 l.jpg
Writing Survey Questions: My Best Advice

  • Ask only one question at a time

    • Avoid double or triple-barreled questions

      • “Do you think your supervisor communicates well with staff about their performance as well as about what is happening in the agency?”

      • She might do one well but not the other. How would employee answer this question?

Dr. G. Johnson, www.reseachdemystified.org


Writing survey questions my best advice34 l.jpg
Writing Survey Questions: My Best Advice

  • Close all the “gates”

    • Provide instructions for skip questions

    • Provide mutually exclusive responses (eg. No overlapping categories for age or income)

Dr. G. Johnson, www.reseachdemystified.org


Writing survey questions my best advice35 l.jpg
Writing Survey Questions: My Best Advice

  • Focus on current experiences: memory decays over time!

    • The more distant the time, the more likely their recall will be inaccurate.

  • Leave exits (no opinion, not applicable)

  • Ask only the demographic questions that you actually need, keeping in mind that some people may be identifiable (eg. the only woman on the faculty will be identifiable)

Dr. G. Johnson, www.reseachdemystified.org


Writing survey questions my best advice36 l.jpg
Writing Survey Questions: My Best Advice

  • Pre-test, pre-test, pre-test

    • With real participants, find out:

      • Is each question clear?

      • Are there unknown words or phrases?

      • Is there a better way to ask each question?

      • What questions did they expect to see but did not?

      • Remember: ask them for suggestions

Dr. G. Johnson, www.reseachdemystified.org


Practice writing survey questions l.jpg
Practice: Writing Survey Questions

  • Thinking about the observation of the room exercise:

    • Develop 5 closed-ended questions that would enable everyone in the class to assess the key characteristics of the room

  • Develop 10 questions to assess student opinions about the quality of the MPA program?

Dr. G. Johnson, www.reseachdemystified.org


Practice writing survey questions38 l.jpg
Practice: Writing Survey Questions

  • Compare with partners

  • Notice:

    • Similarities and differences?

    • How many different topics were covered?

    • How many different scales were used?

  • While there are guidelines that help, there is tremendous diversity things we each think matters most and the words we prefer

Dr. G. Johnson, www.reseachdemystified.org


Survey questions in the news l.jpg
Survey Questions In The News

  • Washington Post-ABC survey (11/09) asked:

  • “On another subject, you may have heard about the idea that the world's temperature may have been going up slowly over the past 100 years. What is your personal opinion on this - do you think this has probably been happening, or do you think it probably has not been happening?”

Dr. G. Johnson, www.reseachdemystified.org


Right concept l.jpg
Right Concept?

  • Is it global warming or climate change?

  • Do some people think climate change is happening but not global warming?

  • How would they answer this question?

Dr. G. Johnson, www.reseachdemystified.org


Survey questions in the news41 l.jpg
Survey Questions In The News

  • “There's a proposed system called "cap and trade." The government would issue permits limiting the amount of greenhouse gases companies can put out. Companies that did not use all their permits could sell them to other companies. The idea is that many companies would find ways to put out less greenhouse gases, because that would be cheaper than buying permits. Would you support or oppose this system?”

Dr. G. Johnson, www.reseachdemystified.org


Right question l.jpg
Right Question?

  • Is this the most accurate description of cap and trade?

  • For those who do not support cap and trade but do want the government to take action to mitigate climate change—how would they answer?

Dr. G. Johnson, www.reseachdemystified.org


Dilemma short but biased l.jpg
Dilemma: Short but Biased?

  • My belief: to only ask about a single option for a complex policy issue is a form of bias

    • But you can also see that the survey would get quite long if all options were covered in the midst of a survey about many policy issues

    • It is a dilemma: need to keep it short and simple, but need to be comprehensive to avoid bias

Dr. G. Johnson, www.reseachdemystified.org


Writing survey questions final words l.jpg
Writing Survey Questions: Final Words

  • Writing questionnaires are harder than might appear

  • The English language is sometimes difficult: the same words can mean different things to different people

  • It is sometimes hard to get a complex issue nailed down in one question

    • It may make take ten.

  • Pre-test, Pre-test, Pre-test

  • Folks: you won’t write a great survey in a week!

Dr. G. Johnson, www.reseachdemystified.org


Credibility of survey results l.jpg
Credibility of Survey Results

  • If the issue matters to you, look at the exact wording of the question and the possible answers

    • A news report may not give an accurate rendering

    • Make sure the question did not try to lead to desired answers (bias)

    • Make sure there are an equal number of positive and negative response options

    • Consider whether they asked the right questions for the topic

    • Be on the look-out for fatally flawed questions

Dr. G. Johnson, www.reseachdemystified.org


Credibility of survey results46 l.jpg
Credibility of Survey Results

  • At the same time, keep in mind, it really is hard to write good survey questions

  • The appearance of bias might be the result of how they decided to handle the comprehensive vs keep in short dilemma

  • It is easy to criticize any survey but after you have done a few surveys, you will have more compassion for the work of others

  • Perfection is not the appropriate criteria

Dr. G. Johnson, www.reseachdemystified.org


Creative commons l.jpg
Creative Commons

  • This powerpoint is meant to be used and shared with attribution

  • Please provide feedback

  • If you make changes, please share freely and send me a copy of changes:

  • Visit www.creativecommons.org for more information

Dr. G. Johnson, www.reseachdemystified.org


ad