Optimal Adaptive Survey Design - PowerPoint PPT Presentation

Optimal adaptive survey design
1 / 18

  • Uploaded on
  • Presentation posted in: General

Optimal Adaptive Survey Design. Lars Lyberg, Frauke Kreuter, and James Wagner ITSEW 2010 Stowe, VT, USA, June 16. What Should Be Designed?. Requirements+specifications+operations Ideal goal+ Defined goal+Actual results

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Optimal Adaptive Survey Design

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Optimal adaptive survey design

Optimal Adaptive Survey Design

Lars Lyberg, Frauke Kreuter, and James Wagner

ITSEW 2010

Stowe, VT, USA, June 16

What should be designed

What Should Be Designed?

  • Requirements+specifications+operations

  • Ideal goal+ Defined goal+Actual results

  • Good survey design means control of accuracy through the specs (QA) and control of operations (QC)

Some early thinking

Some Early Thinking

  • Hansen-Hurwitz-Pritzker 1967

    • Take all error sources into account

    • Minimize all biases and select a minimum-variance scheme so that Var becomes an approximation of (a decent) MSE

    • The zero defects movement that later became Six Sigma

  • Dalenius 1969

    • Total survey design

Some more thinking

Some More Thinking

  • Textbook on total survey design

    • Hansen-Hurwitz-Cochran-Dalenius

  • Survey models and specific error sources

  • Cochran’s comment from 1968

Alternative criteria of effectiveness

Alternative Criteria of Effectiveness

  • Minimizing MSE for a given budget while meeting other requirements

  • Maximizing fitness for use for a given budget

  • Maximizing comparability for a given budget

  • All these reversed

  • Something else?

The elements of design

The Elements of Design

  • Assessing the survey situation (requirements)

  • Choosing methods, procedures, “intensities”, and controls (specifications)

  • Allocating resources

  • Assessing alternative designs

  • Carry out one of them or a modification of it

  • Have a Plan B

So what s the problem

So, What’s the Problem?

  • No established survey planning theory

  • Multi-purpose, many users

  • The information paradox

  • Uninformed clients/users/designers

  • Much design work is partial, not total

  • Limited knowledge of effects of measures on MSE and cost

More problems

More Problems

  • Decision theory and economics theory not used to their potential

  • New surveys conducted without sufficient consideration of what is already known

  • No one knows the proper allocation of resources put in before, during and after

  • The literature is small

Various skills needed which calls for a design team

Various Skills Needed Which Calls for a Design Team

  • Survey methodology

  • Subject-matter

  • Statistics (decision theory, risk analysis, loss functions, optimization, process control)

  • Economics (cost functions, utility)

  • IT

The adaptive element

The Adaptive Element

  • The entire survey process should be responsive to anticipated uncertainties that exist before the process begins and to real time information obtained throughout the execution of the process


  • Use process data (paradata) to check, and if necessary, adjust the process

We should assemble what we know

We Should Assemble What We Know

  • Assessment methods

  • Design principles

  • Trade-offs and their effects

  • The potential offered by other disciplines

  • We shouldn’t accept partial designs

Apply design principles

Apply Design Principles

  • If pop is skewed then….

  • If pop is nested then….

  • If questions are sensitive then….

  • If a high NR rate is expected then…

Apply sops cbms or best practices

Apply SOPs, CBMs or Best Practices

  • Part of the design is to use known, dependable methods

Examples of trade offs

Examples of Trade-offs

  • Accuracy vs timeliness

  • Response burden vs wealth of detail

  • Conduct survey vs other information collection

  • Large n vs smaller n

  • Mixed vs single mode

  • NR bias vs measurement error

  • NR vs interpretation by family members

Process view

Process view

  • Upstream thinking (prevention)

  • Understanding variation

  • Measure cost of poor quality and waste

  • Intervention or improvement actions should be based on good data and statistical analysis

  • Continuous monitoring

Tentative course syllabus

Tentative Course Syllabus

  • The elements of design

  • Real world examples (e.g., CPS Technical Paper 63, PIAAC, the Monthly Retail Trade Survey, the Annual Survey of Hale Mountain Fish & Game Club, VT)

  • The literature on optimal decisions

  • Theory for adaptive treatment design and risk management

Course syllabus continued

Course syllabus continued

  • Data for monitoring and decision making

  • Analysis of such data

  • Design lessons learned

  • Examples of bad designs and not so great trade-offs

  • Student project with TSE perspective

  • Student presentations

  • Login