Optimal adaptive survey design
Sponsored Links
This presentation is the property of its rightful owner.
1 / 18

Optimal Adaptive Survey Design PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Optimal Adaptive Survey Design. Lars Lyberg, Frauke Kreuter, and James Wagner ITSEW 2010 Stowe, VT, USA, June 16. What Should Be Designed?. Requirements+specifications+operations Ideal goal+ Defined goal+Actual results

Download Presentation

Optimal Adaptive Survey Design

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Optimal Adaptive Survey Design

Lars Lyberg, Frauke Kreuter, and James Wagner

ITSEW 2010

Stowe, VT, USA, June 16

What Should Be Designed?

  • Requirements+specifications+operations

  • Ideal goal+ Defined goal+Actual results

  • Good survey design means control of accuracy through the specs (QA) and control of operations (QC)

Some Early Thinking

  • Hansen-Hurwitz-Pritzker 1967

    • Take all error sources into account

    • Minimize all biases and select a minimum-variance scheme so that Var becomes an approximation of (a decent) MSE

    • The zero defects movement that later became Six Sigma

  • Dalenius 1969

    • Total survey design

Some More Thinking

  • Textbook on total survey design

    • Hansen-Hurwitz-Cochran-Dalenius

  • Survey models and specific error sources

  • Cochran’s comment from 1968

Alternative Criteria of Effectiveness

  • Minimizing MSE for a given budget while meeting other requirements

  • Maximizing fitness for use for a given budget

  • Maximizing comparability for a given budget

  • All these reversed

  • Something else?

The Elements of Design

  • Assessing the survey situation (requirements)

  • Choosing methods, procedures, “intensities”, and controls (specifications)

  • Allocating resources

  • Assessing alternative designs

  • Carry out one of them or a modification of it

  • Have a Plan B

So, What’s the Problem?

  • No established survey planning theory

  • Multi-purpose, many users

  • The information paradox

  • Uninformed clients/users/designers

  • Much design work is partial, not total

  • Limited knowledge of effects of measures on MSE and cost

More Problems

  • Decision theory and economics theory not used to their potential

  • New surveys conducted without sufficient consideration of what is already known

  • No one knows the proper allocation of resources put in before, during and after

  • The literature is small

Various Skills Needed Which Calls for a Design Team

  • Survey methodology

  • Subject-matter

  • Statistics (decision theory, risk analysis, loss functions, optimization, process control)

  • Economics (cost functions, utility)

  • IT

The Adaptive Element

  • The entire survey process should be responsive to anticipated uncertainties that exist before the process begins and to real time information obtained throughout the execution of the process


  • Use process data (paradata) to check, and if necessary, adjust the process

We Should Assemble What We Know

  • Assessment methods

  • Design principles

  • Trade-offs and their effects

  • The potential offered by other disciplines

  • We shouldn’t accept partial designs

Apply Design Principles

  • If pop is skewed then….

  • If pop is nested then….

  • If questions are sensitive then….

  • If a high NR rate is expected then…

Apply SOPs, CBMs or Best Practices

  • Part of the design is to use known, dependable methods

Examples of Trade-offs

  • Accuracy vs timeliness

  • Response burden vs wealth of detail

  • Conduct survey vs other information collection

  • Large n vs smaller n

  • Mixed vs single mode

  • NR bias vs measurement error

  • NR vs interpretation by family members

Process view

  • Upstream thinking (prevention)

  • Understanding variation

  • Measure cost of poor quality and waste

  • Intervention or improvement actions should be based on good data and statistical analysis

  • Continuous monitoring

Tentative Course Syllabus

  • The elements of design

  • Real world examples (e.g., CPS Technical Paper 63, PIAAC, the Monthly Retail Trade Survey, the Annual Survey of Hale Mountain Fish & Game Club, VT)

  • The literature on optimal decisions

  • Theory for adaptive treatment design and risk management

Course syllabus continued

  • Data for monitoring and decision making

  • Analysis of such data

  • Design lessons learned

  • Examples of bad designs and not so great trade-offs

  • Student project with TSE perspective

  • Student presentations

  • Login