1 / 29

Advanced survey design

Tuesday, February 7, 2019 10:00 – 11:30am. Advanced survey design. Jen Sweet Center for Teaching & Learning DePaul University Shannon Milligan Institutional Research & Market Analytics DePaul University. Workshop Outline. Introductions Conflicting Recommendations for Survey Design?

elma
Download Presentation

Advanced survey design

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Tuesday, February 7, 2019 10:00 – 11:30am Advanced survey design Jen Sweet Center for Teaching & Learning DePaul University Shannon Milligan Institutional Research & Market Analytics DePaul University

  2. Workshop Outline • Introductions • Conflicting Recommendations for Survey Design? • Cognition and Survey Design • Recommendations to Reduce Cognitive Load ACTIVITY! • Affective Assessment and Surveys ACTIVITY! • Survey Design & Distribution Tools

  3. Workshop Outcomes By the end of this workshop, participants will be able to: • Apply knowledge of the cognitive processes students use to respond to surveys to design effective survey items and instruments. • Identify non-cognitive variables that can be assessed with surveys • Use the tools available to them at their respective institutions to design and distribute surveys.

  4. Conflicting Recommendations for Survey Design?

  5. Recommendations for Survey Design that seem to Conflict Examples: Neutral Point • Always include? • Never use? • Sometimes yes; sometimes no? Number of Scale Points to Include • 2? • 3? • 4? • 5? • 6? • 7? • 9? • The more the better? All of This is in the Literature!

  6. So, What’s Up with the Literature? All of these recommendations may be appropriate depending on the specific context: • respondent attributes • nature of the items in the survey • length of the survey • Etc. Generally looking at things like: • Reliability • *Validity • Survey Outcomes • Response Rate (high) • Use of Response Sets (low) Survey Design is as Much Art as Science! “There is always a well-known solution to every human problem - neat, plausible, and wrong.” H.L. Hencken

  7. Cognition and Survey Design

  8. Cognitive Load Paas and Van Merrienboer, 1994 The amount of cognitive effort (or thinking) students need to exert to respond to a survey item. If cognitive load exceeds the student’s working memory capacity, they will take some sort of shortcut (Paas & Van Merrienboer, 1994), or satisfice (Krosnick, 1991) • Read questions less carefully (skim) • Use a response set • Give same response for all questions, regardless of content • Overuse neutral or N/A response option • Skip the question (provide no response) • Respond randomly • Decide not to complete the survey

  9. Cognitive Steps to Respond to Surveys (Tourangeau, 1984) • Interpretation • Retrieval • Judgment • Response

  10. Recommendations to Reduce Cognitive Load

  11. Step 1: Interpretation • Use language that is clear and familiar to survey respondents. • Avoid cognitively taxing wording. • Avoid unfamiliar words and phrasing. • Avoid jargonand acronyms • Ensure that question stems are clear and explicit. • Do not use concepts that are unclear or unfamiliar to respondents. • Avoid complex sentence structures. • Ask about only one concept in each stem; avoid double-barreledquestions • Use questions that do not make assumptions. • Ask for information in a direct manner by avoiding double negatives • Ensure question stems are succinct, including only as much information as is necessary for respondents to properly interpret what is being requested of them.

  12. Interpretation (Continued) • Include clear instructions that clarify the purpose of the survey instrument, and provide respondents with expected procedures for responding to the survey instrument. • Ensure that every portion of a survey instrument is visible without the need for additional action by the respondent. • Use radio buttonsinstead of drop-down boxesto display response options. • Do not “hide” definitions respondents may need to interpret and respond to survey items. • Use easy-to-read font size and type. • Use high-contrast font and background colors.

  13. Step 2: Retrieval • Use stems that request information with which respondents have primary experience and avoid asking for second-hand information (i.e., information that the respondent has heard about, but not experienced personally) or hypothetical information. • Group conceptually similar items together.

  14. Step 3: Judgment • Use the smallest number of response options necessary to encompass all meaningful divisions of what you are asking about. • General Guideline: four or five response options, depending on whether or not there will be a neutral option. • Include a neutral option if you reasonably expect participants to have no opinion, but otherwise, they should be avoided • Neutral responses can be difficult to interpret • Offering a neutral option may encourage satisficing

  15. Step 4: Response • Use the smallest number of response options necessary to encompass all meaningful divisions of what you are asking about. • Label the scale options. • May only need to label the most extreme options • Include a neutral option if you reasonably expect participants to have no opinion, but otherwise, they should be avoided • Neutral responses can be difficult to interpret • Offering a neutral option may encourage satisficing

  16. Activity!

  17. Practice Evaluating Survey Items! Individually: Complete the worksheet In Groups: Compare Responses Did Everyone Identify the same items for Improvement? Are there differences in ways you edited items?

  18. Surveys and the Rise of Affective Assessment

  19. Growing Emphasis on Non-Cognitive Abilities • Grit • Growth • Social-emotional development • Self-awareness/management/efficacy • General affect • Engagement • Mattering • Climate • Research shows strong relationships between these variables and overall success From NPR: “Nonacademic Skills Are Key to Success. But What Should We Call Them?” (May 28, 2015)

  20. The Role of Surveys From NPR: “To Measure What Tests Can’t, Some Schools Turn to Surveys” (December 2, 2015) From that article: “A growing battery of school leaders, researchers and policymakers think surveys are the best tool available right now to measure important social and emotional goals for schools and students” Why? • Easy to administer • Easier to collect and analyze than reflection papers (and the like) • Many surveys/survey questions already exist • Faster data sharing = faster decision-making/implementation (maybe)

  21. Activity!

  22. Non-Cognitive Assessment and You Individually: • What non-cognitive variables might be of interest to your program? • Are you already assessing any of these variables? In Groups: • Share what you are assessing and what you might be interested in assessing • What variables might be important to look at on an institutional-level?

  23. Survey Design/Distribution Tools

  24. Three Main Tools Used at DePaul • Qualtrics • Google Forms • Survey Monkey

  25. Qualtrics Advantages: • Free to DePaul faculty and staff • Supported by Information Services • Very Flexible and Comprehensive System • Lots of Features Disadvantage: • Reporting Features aren’t Great • Steeper Learning Curve than Other Systems

  26. Google Forms Advantages: • Free to Everyone • Data is Collected in Excel Format • Easier to Learn than Qualtrics Disadvantage: • Not Nearly as many Features as Qualtrics

  27. Survey Monkey Advantages: • ? Disadvantage: • Most Advanced Features Cost $ • To get all the features available in Qualtrics, cost is $780/year • Limited to 10 questions and 100 responses on the free version

  28. Questions??

  29. Contact Information Jen Sweet DePaul University Office for Teaching, Learning & Assessment jsweet2@depaul.edu Shannon Milligan DePaul University Institutional Research & Market Analytics shannon.milligan@depaul.edu

More Related