THE BASICS OF CRAFTING GOOD QUESTIONS. Damon Burton University of Idaho. Smyth, Dillman & Christian (2007) studied how question construction influenced responses on the number of hours a day students studied. 3 types of response patterns were tested:
University of Idaho
3 types of response patterns were tested:
A low range with 5 categories from .5 or less to 2.5 hours & a 6th category of more than 2.5 hours
A high range consisting of 2.5 hours or less and 5 categories between 2.5 and 4.5 hours
An open-ended question with no categories.
Responses for students studying more than 2.5 hours varied by response categories, including: 30% for the low range, 71% for the high range and 58% for the open-ended responses.QUESTIONCONSTRUCTION CASE STUDY
Respondents often look to the question and its accompanying response options for clues.
Typically they assume that the range emphasized by the scale represents how many hours most students study.
The middle option is often seen as representing the amount the average student studies.
To save time, students may simply estimate whether they study more or less than an average student.RATIONALE FOR FINDINGS
Is this question being repeated from another survey, and/or will answers be compared to previously-collected data?
Will respondents be willing and motivated to answer accurately?
What type of information is the question trying to obtain?ISSUES IN CRAFTING SURVEY QUESTIONS
Telephone interview respondents give and receive information through spoken words and their hearing system. Memory also becomes a more important factor.
Web and mail questionnaires transmit information through the visual system.
In mail and internet surveys, visual design elements become more critical.1. SURVEY MODE USED
Government surveys and longitudinal studies often repeat items.
In self-administered studies, not only should the question be the same but also visual design and layout of the item.
Can items be modified to remain contemporary in wording and focus?2. COMPARING TO PREVIOUS SURVEYS
Assume the worst in designing items.
Poor item design may reduce motivation.
Items may be (a) difficult to read and understand, (b) instructions are hard to find, or (c) the response task is too vague.3. ITEM MOTIVATES ACCURATE RESPONSE
Broad categories are more readily answered when dealing with financial information.
For sensitive questions, wording and putting the question into context improve response rates.3. ITEM MOTIVATES ACCURATE RESPONSE
For attitude or opinion questions, elements of the item can influence the answering process,
type of response requested,
visual layout.4. TYPE OF INFORMATION REQUESTED
For behaviors or events, if the item requires remembering too many details, respondents rely on survey design for answers.
Memory fades over time, so asking questions about easily-recalled or memorable behaviors enhances recall (e.g., days walked to work this week or recent trip to Yellowstone).4. TYPE OF INFORMATION REQUESTED
Choose appropriate question format(s).
Understand the anatomy of a survey question and use that information to design effective items.
Question design benefits from informed, multiple and diverse perspectives.A WHOLISTIC APPROACH TO QUESTION DESIGN
Closed-ended questions are used to force all respondents to make a choice from a limited range of options (e.g., multiple choice questions or rating agreement on a 5-point Likert scale.
The choice of format needs to be based on the purpose of the question.A WHOLISTIC APPROACH TO QUESTION DESIGN
allows respondents to freely answer without limitations,
numerical responses can be reported exactly without vague labels or ranges,
Weaknesses of open-ended formats
more respondents skip this item format,
biases arise more often,
more variations in responses making it more difficult to analyze, and
short answers more likely.OPEN-ENDED QUESTION FORMATS
can be analyzed statistically (i.e., ordinal or interval data),
data results are produced quickly,
numerical responses can be reported exactly without vague labels or ranges,
Weaknesses of closed-ended formats
ordering and grouping may have unintended effects on answers and make response harder,
Responses are constrained by possible answers.
Hard to identify new categories of information.CLOSED-ENDED QUESTION FORMATS
Ordinal data has an order to the categories of response choices (i.e., good, average and poor) but can’t assume the size of the scaled magnitude between categories is the same or represents equal distance.
Interval data has both order and equal intervals between response choices (i.e., 5 categories from very dissatisfied to very satisfied).TYPES OF DATA
Additional instructions help respondents comprehend the meaning of questions or concepts.
Crafting survey questions involves both choosing words and visually displaying questions and responses.ANATOMY OF A SURVEY QUESTION
Make sure the item is technically correct.
Ask one question at a time.
Use simple and familiar words.
Select specific, concrete wording.
Use as few words as possible.
Use simple, complete sentences.
Make sure “yes” means yes & “no” = no.
Be sure the question specifies how to respond.GUIDELINES FOR CHOOSING WORDS & FORMING QUESTIONS
Avoid “skip” instructions,
Avoid the word “if” which implies a response is not needed.
Avoid providing only 2 bad options, especially if respondents have to answer.
Provide a range of categories so every person has to respond to every question.1. QUESTION APPLIES TO ALL
For example, the question “How many feet tall is your horse?” is hard to answer because horses’ height is typically measured in “hands” (i.e., a hand = 4 inches).
Credibibility and trust come with using the appropriate terminology for the audience and being technically correct.2. ITEM TECHNICALLY CORRECT
For example, the question “Do you subscribe to and read any magazines related to your job?” could be answered based on (a) subscribing, (b) reading or (c) both.
Responses are quicker and more accurate when the item has a single topic.
Credibibility and trust come keeping questions simple and easy to respond to.3. ONE QUESTION AT A TIME
Avoid formalized wording with too complex or technical wording.
For example, use “tired” instead of “exhausted,” “work” instead of “employment,” and “correct” instead of “rectify.”
Replace specialized terms and cliches.
Credibility comes from simplicity.4. USE SIMPLE, FAMILIAR WORDS
For example, the question “How many times did you get together as a family last week?” is somewhat vague.
What does “get together as a family mean?”
“How many meals did you sit down to eat as a family last week?” is much more specific.
Specificity in questions allows much more accurate responses.5. USE SPECIFIC WORDING
Replace complex words or rewrite item more simply.
Eliminate redundant words.
However, keeping readability low and content specific is more important than question length.6. USE FEW WORDS PER ITEM
Write complete sentences that use a simple sentence structure.
Avoid convoluted sentence structure with confusing wording.
Keeping readability low requires simple sentences that are easily understood.
Compound sentences work better than complex ones.7. USE COMPLETE SENTENCES
Specify meaning of words such as “favor” and “oppose” so “for” and “against” may be better terms.
Ballet initiatives are often worded in a confusing ways so you are not sure whether you are “for” or “against.” Avoid this problem in your surveys.
Credibility and trust come from making decisions easy to understand.8. MAKE SURE “YES” MEANS “YES”
Make the task clear to the respondent after have read the question once.
The question stem has to clearly state the response task.
The response format and/or options provided must match the task as stated in the question stem.
Mismatched question and response makes item unusable.9. SPECIFY THE RESPONSE TASK
4 types of visual design elements communicate meaning and can be manipulated to draw attention to or take away from meaning.
Words – the primary source of meaning.
Numbers – convey meaning and sequence.
Symbols – figures that add special meaning.
Graphics – simple and complex shapes and visual images that convey meaning.VISUAL PRESENTATION OF SURVEY QUESTIONS
Gestalt principles of pattern perception assign meaning and provide a sense of grouping and separation of elements.
Proximity – locating elements together denotes relatedness.
Common Region – boxes and highlighted areas designate connectedness.
Connections – linking elements thru lines.
Continuity – multiple elements layered continuously creates a perception of a group.VISUAL PRESENTATION OF SURVEY QUESTIONS
Creates clear subgrouping and separation,
Contrast helps delineate questions from answers.
Use spacing to help create subgrouping within a question because proximity states that items located together are perceived as a group.
Visually standardize all response spaces.
Similarity states that items appear regular and similar are perceived as belonging together.VISUAL PRESENTATION GUIDELINES
Manipulating size, contrast and location will emphasize or de-emphasize element importance.
5. Use design properties with consistency.
Use each design element for only one purpose such as underlining to draw attention to important words or bolding for questions.
6. Visual elements in question must send consistent messages.
Avoid conflicting messages communicated by incongruous visual elements.VISUAL PRESENTATION GUIDELINES
When special instructions are outside the question stem, they may be ignored.
8. Separate optional or occasionally needed instructions from the question stem by font or symbol variation.
Distinguish between words that are essential for everyone to read and those that may be needed by only some respondents.
9. Organize questions in ways that minimize the need to reread portions of instructions in order to comprehend the response task.
Emphasize respondent efficiency so they are always clear about what to do.VISUAL GUIDELINES
Need appropriate font so avoid script and use serif or sans serif.
Use 10-12 point font for most populations and larger fonts for older populations.
Line length of 3-5 inches is recommended.VISUAL PRESENTATION GUIDELINES