1 / 23

Data Collection & Technology

Data Collection & Technology. EDUC 894 Week 6. Plan for Today. Next Week No class, consultations available E-mail me by next Monday to set up a time Reading handouts Instrument Focus: WebSurveys What NOT to do: An Example How many problems can you find? What To Do : Principles

mare
Download Presentation

Data Collection & Technology

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Collection & Technology EDUC 894 Week 6

  2. Plan for Today • Next Week • No class, consultations available • E-mail me by next Monday to set up a time • Reading handouts • Instrument Focus: WebSurveys • What NOT to do: An Example • How many problems can you find? • What To Do: Principles • Designing a Valid & Reliable Survey • The Wario Survey • Working with WebSurvey data • Instrument Focus: Eye-tracking & Brainwaves • Field trip to the Engrammetron -----------------------------Break----------------------------- • Group Work • Honing down your instruments

  3. Survey Design • (Most) surveys are conducted from an objectivist perspective • Writing a survey is all about making decisions: • Recognizing when a choice is being made • Making choices that will create a reliableinstrument that will help you make validinferencesto answer your research questions • It’s (usually) not about making the “best” decision in some abstract sense, but making a choice that will be most useful for your purposes

  4. Limitations of Surveys • Self-report data • Notoriously unreliable • Indicates only what we think, not reality • Data at one point in time • Limited Sample • Retrospective questions rely on memory • The Devil is in the Details • Question form, wording, and answer choices can inadvertently dictate responses • Sampling • Who responds isn’t random Image Source: http://www.nynpa.com/images/NYNPA0081.jpg

  5. Survey Decisions • Topics Covered • Number & Focus of Items per Topic • Item Sequence & Presentation • Item Type / Format • Item Wording • Always think about what you will do with the answers you get – i.e. how will you analyze the data?

  6. What NOT to do • Take the survey posted here (link sent via email) https://my.sfu.ca/cgi-bin/WebObjects/WebSurvey.woa/wa/survey?17437110 • How many problems in each category can you find? • Topics Covered • Number & Focus of Items per Topic • Item Sequence & Presentation • Item Type / Format • Item Wording

  7. Survey Decisions Topics Covered • Remember the idea of content-related validity? • Goal:Determine the most popular animals • Assumed “like” indicates popularity • What pet you had as a child may or may not relate • Are the plants in you garden a better indicator? • Feelings about gardening don’t give relevant info • Caveat: Questions to lay a foundation • Define your terms • Are you concerned with popularity of animals (in general) or only pets? • Can ask a similar question about plants / garden plants • What about ability self-esteem (in general) or self-esteem as related to academic work? • How would this affect your survey design?

  8. Survey Decisions Topics Covered Survey Academic Self Esteem Social Athletic

  9. Survey Decisions Number & Focus of Items per Topic • Number: Do you have enough questions on each topic to give a reliable measure? • Three questions is usually the absolute minimum given for a scale • Focus: Do questions give you info you can actually triangulate? • How will you compare the names of plants reported to be in their garden with Likert scales given for specific plants? • What will you do with the question about liking “trees without flowers”?

  10. Survey DecisionsItem Sequence & Presentation • Question sets and introductions • Broad to narrow progression • How satisfied are you with service in the food court? • Do servers in the food court smile when taking your order? • Will you group or distribute similar content items? • Putting similar items together will help learners think about what you want them to • But, if you have multiple items try and get convergence on a construct, putting them near each other will lead to a false reliability • Pagination • No one likes to scroll forever • No one likes unnecessary clicking • The “required item” dilemma

  11. Survey DecisionsItem Type / Format • Open versus Closed Ended Questions • It all depends on what you are looking for, but from an objectivist perspective: “Use close ended questions where at all possible, if you can limit the possible answers it can really speed up the data crunching when you are finished” • You may need open-ended questions when you don’t know the range of possible answers • But then you will have to go through and code these answers for themes • Thus, don’t ask an open-question with a limited number of possible answers or where answers range predictably • “Do you like to garden?” • “How much do you like dogs?”

  12. Survey DecisionsOpen Ended Question Guidelines • Be specific • What did you think of class? BAD • What 3 things did you like best about class? GOOD • What 3 things did you like best about the class content? BETTER • Leave people enough space to write • A philosophy of life, love and happiness is not 64 characters or less! • Don’t ask too many open-ended questions • Your participants will tire and it will show in their responses • G. W. Bush example

  13. Survey DecisionsClosed Ended Question Guidelines • Question Types 1 • Yes/No • Lists • Single choice (“best answer”, radio button) • Multi (“all that apply”, checkbox) • Can do a series of these in a grid, but dangers of confusion and auto-pilot

  14. Survey DecisionsClosed Ended Question Guidelines • Question Types 2 • Rating Scales • Ordinal • E.g. Rank the following from best to worst • Likert • E.g. Levels of agreement • Likert-style • E.g. Levels of approval, satisfaction etc. • Frequency • E.g. Often, sometimes, almost never • Bipolar • E.g. Outgoing to introverted

  15. Survey DecisionsClosed Ended Question Guidelines • Focus on Rating Scales • Graphic vs. Finite Choice • Think about the following: • Does your scale goes to the extremes? • Does your scale have enough choices? • Participants are less likely to choose the anchors (n-2) • Does your scale imply interval spacing? • Will your scale have a “neutral” option? Mark an X on the line to show your level of agreement Disagree Agree Indicate your level of agreement by circling your choice Strongly Disagree Disagree Neutral Agree Strongly Agree

  16. Survey DecisionsItem Wording 1 • Clear & Concise • What do you do when you get a headache?UNCLEAR • What headache medicine do you use? CLEARER • Concrete • Did you enjoy the book? ABSTRACT • Did you recommend the book to a friend? CONCRETE • One idea per question • Should BC continue its urban density and needle exchange programs? CONFOUNDED • Be careful when asking personal questions • Use categories to avoid emotional responses

  17. Survey DecisionsItem Wording 2 • Avoid leading questions • Do you think Barak Obama will continue to be a successful candidate? • Do you think Barak Obama will continue to be plagued by challenges about his level of experience? • The challenge of reversed questions • Can be used to break up question sequence or to get at something in a different way, but big reliability issues • I do not like venus fly-traps EXPLICIT • Self-Esteem Survey: Sometimes I doubt myself IMPLICIT

  18. Survey DecisionsThe Big Picture • Every time you have to make a decision, always think about 3 things • How will your decision affect participant responses? • How will your decision affect your analysis? • What does your decision imply about the underlying construct?

  19. The Wario Survey

  20. Working with WebSurvey Data

  21. Field trip to the Engrammetron

  22. Break

  23. Group Work

More Related