1 / 32

Designing for Reality

Designing for Reality. QMSS2 Seminar 3 on Survey Quality Leinsweiler , Dec. 2010 Peter Ph. Mohler. Doing surveys is a privilege and great fun!. Overview. Prolegomena Designing for Reality Conclusion . Prolegomena. Social Surveys are very, very robust measurement devices

novia
Download Presentation

Designing for Reality

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. QMSS2 Peter Mohler 2010

  2. Designing for Reality QMSS2 Seminar 3 on Survey Quality Leinsweiler, Dec. 2010 Peter Ph. Mohler

  3. QMSS2 Peter Mohler 2010

  4. Doing surveys is a privilege and great fun! QMSS2 Peter Mohler 2010

  5. Overview • Prolegomena • Designing for Reality • Conclusion QMSS2 Peter Mohler 2010

  6. Prolegomena • Social Surveys are very, very robust measurement devices • Screening of ALLBUS showed over a period of 20 years only one ‘deviation’ (due to a mis-design in item sequence • Exception: web-panels of sorts used in marketing due to an extreme fluctuation of the panel participants QMSS2 Peter Mohler 2010

  7. Prolegomena • This seminar discussed “problems” on a very high quality and precision level reflecting the needs of government agencies QMSS2 Peter Mohler 2010

  8. Prolegomena • Accuracy/Precision/Micro-census Benchmarking • The quality of surveys is often defined by checking core demographics using a Micro-Census as benchmark • For instance Age, Gender, Education, or Regional Distribution • Three surveys • A) Personal address, age, gender, citizenship from residents lists • B) Like 1, but more calls per respondent • C) Address of household with substitution on the primary sampling unit (PSU) level by interviewer QMSS2 Peter Mohler 2010

  9. Prolegomena - Benchmarking • Survey C is best • Note Micro-Census is also a survey with MSE! QMSS2 Peter Mohler 2010

  10. Reality Check – Why to Break Your Neck? • Survey A and B cost much, much more than Survey C (including exhausting interviewers and staff) • Survey A and B need more time for data collection • So, why should you break your neck, when you can get better benchmarking by less effort and costs? (Attention: Herecy!) • However, the real question is, which effort and costs for what survey? (MSE) QMSS2 Peter Mohler 2010

  11. Analysis in a Real World • Marketing • 20% vs- 80% market share • Election polls • +/- 2% precision for predicting election outcomes • Health prevalence • 1% point estimate error may result in millions of dollars mis-allocated • Comparative research • 1 minute lost due to mis-localisation results in a one time loss of ~150.000€ multiplied by long term loss in analytical power QMSS2 Peter Mohler 2010

  12. Last Prolegomon Scientific Engineering • Scientific work requires transparent, inter-subjective methods • Not all methods are fully based on scientific proof • Some of them are based on experience • they “work”, but are not completely understood why • Doing surveys is scientific engineering • Improving survey research methods requires better scientific understanding why something works or works not (scientific laws of causality) QMSS2 Peter Mohler 2010

  13. Designing for Reality QMSS2 Peter Mohler 2010

  14. Doing Data Collection • Is like driving 100 Quadrigae simultaneously • Is a complex logistic enterprise • Needs lots of engineering QMSS2 Peter Mohler 2010

  15. What is the Object of our Desire? QMSS2 Peter Mohler 2010

  16. The Object of Desire QMSS2 Peter Mohler 2010

  17. The Obscure Object of Desire • Is that single measurement (system – see Bo) • Everything else is environment (in systems terminology to that measurement system) • The “environment” is the big machine that supports the single measurement (asking a question and getting a response • Questionnaires are interconnected chains of single measurements (that is a system in itself) QMSS2 Peter Mohler 2010

  18. The Environment of the Object of Desire QMSS2 Peter Mohler 2010

  19. Target Measurement in a Nano Dimension QMSS2 Peter Mohler 2010

  20. And its Environment (looks like data collection QMSS2 Peter Mohler 2010

  21. Designing for Reality • Identify your research question • Identify your analytical needs • Apply parsimony • Observe cumulation QMSS2 Peter Mohler 2010

  22. Define Your Research Question QMSS2 Peter Mohler 2010

  23. See a Survey as a Production Process • “Survey Life Cycle” as a metaphor • CCSG build on that metaphor QMSS2 Peter Mohler 2010

  24. Survey Life Cycle QMSS2 Peter Mohler 2010

  25. Survey Production Process II • Introduce Analysis as a core production process item • Combine several items from the life-cycle QMSS2 Peter Mohler 2010

  26. Production System QMSS2 Peter Mohler 2010

  27. Identify Your Analytical Needs • Be very specific about the precision you really need • Be even more specific about the precision you really can achieve • Be suspicious about the volatility of your analytical devices (ask Paul) QMSS2 Peter Mohler 2010

  28. Precision? How Fine is our Measurement Graded? QMSS2 Peter Mohler 2010

  29. Apply Parsimony • AGE plus what other demographics? • 20%-30% of social surveys are about demographics • Experimental pre-research identifying strong predictor variables • Of the shelf surveys or special study ) QMSS2 Peter Mohler 2010

  30. Observe Cumulation • Re-analyse, re-analyse, re-analyse • Drop items that were filtered out in re-analysis • Provide documentation that allows for cumulation QMSS2 Peter Mohler 2010

  31. Reducing Complexity • see Bo’s presentation QMSS2 Peter Mohler 2010

  32. Enjoy your survey! QMSS2 Peter Mohler 2010

More Related