Report on the 2008 international total survey error workshop itsew
Sponsored Links
This presentation is the property of its rightful owner.
1 / 23

Report on the 2008 International Total Survey Error Workshop (ITSEW) PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Report on the 2008 International Total Survey Error Workshop (ITSEW). Presented at Q2008 Conference Rome, Italy July 11, 2008 Presented by Paul Biemer. Acknowledge. Other Organization Committee members: Roeland Beerten, ONS Lilli Japec, Statistics Sweden Mary Mulry, Census Bureau

Download Presentation

Report on the 2008 International Total Survey Error Workshop (ITSEW)

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Report on the 2008 International Total Survey Error Workshop (ITSEW)

Presented at Q2008 Conference

Rome, ItalyJuly 11, 2008

Presented by Paul Biemer


Other Organization Committee members:

Roeland Beerten, ONS

Lilli Japec, Statistics Sweden

Mary Mulry, Census Bureau

Alan Karr, National Institute for Statistical Science

Jerry Reiter, Duke University

Clyde Tucker, BLS

Brian Meekins, BLS


  • Definition of Total Survey Error

  • Highlights of the Recent Workshop on Total Survey Error Workshop

  • Other Emerging Areas of Research

  • Future Directions of Research on TSE

Purpose of the ITSEWs

  • To stimulate new research on total survey error (TSE)

  • Provide a forum for researchers to exchange ideas on current research projects

  • Initiate new projects to advance the TSE field

  • To generate interest in the field among students and new researchers

Organization of the Workshops

  • Attendance by invitation only

    • although no one has ever been refused a seat at the workshop

  • Focus is on on-going research and undeveloped ideas

    • Midcourse research

    • Ideas still in incubation

    • Exploratory research

  • ITSEW series began 2005 in Washington, DC

  • ITSEW 2009 will be held in Tälberg, Sweden

  • National Institute of Statistical Science (NISS) provides funding and staff support

  • Support from other organizations is welcome

2008 Total Survey Error WorkshopTheme: Multiple Error Sources and their Interactions

  • Held in Research Triangle Park, NC on June 2-4, 2008

  • Organizing committee:

    • Paul Biemer, RTI and UNC-CH

    • Roeland Beerten, ONS

    • Lilli Japec, Statistics Sweden

    • Mary Mulry, Census Bureau

    • Alan Karr, NISS

    • Jerry Reiter, Duke University

    • Clyde Tucker, BLS

    • Brian Meekins, BLS

  • Sponsored by the National Institute for Statistical Science, RTI International and the Survey Research Methods Section of ASA

Joop Hox

Annica Isaksson

Peter Lundquist

Noelle Molinari

Donsig Jang

Scott Fricker

Ting Yan

Steven Cohen

Wendy Hicks

Barry Schouten

Emilia Peycheva

Kristen Olson

John Dixon

Paul Biemer

Catharine Burt

Andy Peytchev

Brian Meekins

Mary Mulry

Roger Tourangeau

Jerry Reiter

Lars Lyberg

Rita Thissen

Oztas Ayhan,

Mojca Bavdaz,

Steven Machlin

Eric Slud

27 Presenters at the 2008 ITSEW

Total Survey Error

  • Sampling Error

  • Nonsampling Error

    • Specification

    • Nonresponse

    • Noncoverage

    • Measurement

    • Data Processing

Mean Squared

Error (MSE)

TSE: the Concept

  • Includes both measurement, analysis and reporting of multiple sources of error

    • Quantify the major components of survey error for a survey

    • Assess contributions of each source to TSE

    • Combine these to estimate total MSE

  • Does not include error reduction methods

    • E.g., questionnaire design, nonresponse reduction methods, etc.

Arguments for Estimating TSE

  • To correct inferences based upon faulty theory

    • Biased estimates

    • Biased measures of uncertainty

  • To optimize the design of future surveys

  • To provide quality declarations to users

Arguments Against Estimating TSE

  • Cost

  • Intractability

  • Inaccessible methodology

  • Lack of motivation

  • Fear

Topic Coverage


  • Interaction between nonresponse and measurement error

    • Effect on measurement error of nonresponse followup

    • Statistical models for simulating the measurement error and nonresponse interaction

    • Meta-analysis of literature on nonresponse-measurement error interaction

    • Empirical studies of the relationship between nonresponse and measurement error

    • Causal factors influencing both response propensity and measurement error

    • Nonresponse propensity and under-reporting of sensitive topics

Other common themes

  • Frame stratification errors and sampling errors

  • Frame coverage errors and measurement errors (e.g., cell phone samples and measurement error)

  • Interviewer effects on nonresponse and measurement errors

Gaps in the Research

  • Data processing errors

    • Questionnaire design and editing errors

    • Information content and coding errors

  • Establishment surveys

  • Specification error vs. measurement error

    • i.e., measuring the wrong concept accurately vs. measuring the right concept inaccurately

  • Need for meta-analysis of existing studies

  • Better indicators of data quality

    • Schouten, Bethlehem, et al

  • More attention to costs of error reduction methods

Gaps in the Research (cont’d)

  • Methods for developing countries

  • TSE issues in multi-cultural surveys

  • Methods for estimating variances that reflect multiple error sources, not just sampling errors

  • Standardized indicators for comparing the quality of survey data

  • A common language for TSE

Need for Data

  • How can “true values” be obtained?

  • Paradata for TSE analysis

    • E.g. level of effort to obtain a response in the broadest sense, condition or value of housing, encounters with gatekeepers

    • need a systematic / theory-based framework

    • quality of paradata / “metadata for paradata”

  • Data sets for TSE analysis are scarce

Emerging Research Areas

  • Interviewer effects in surveys and their effects on TSE

  • Methods for predicting poor reporters (i.e., bad respondents)

  • Language translation issues for TSE

  • Completing the feed-back loop between data processors and survey designers

  • Methods for estimating the nonresponse-response error interaction

Some Initiatives Underway for ITSEW 2009

  • Lars Lyberg is organizing ITSEW 2009 in Talberg, Sweden for June 14-17, 2009

  • Biemer and Lyberg will edit a special issue of Public Opinion Quarterly in 2010 on TSE.

  • Special issue on TSE is also being planned for the Journal of Official Statistics in 2010.

  • Alan Karr has offered NISS services to host a moderated Wikipedia on Total Survey Error

Some Initiatives Underway for ITSEW 2009 (cont’d)

  • Special Topic Sessions for ITSEW 2009

    • Nonresponse and measurement error interaction and practical implications (Paul Biemer)

    • Shared data sets for TSE research (Barry Schouten)

    • Coverage error (Stephanie Eckman)

    • POQ and JOS have offered special issues in TSE

Some Issues in Research on Nonresponse-Response Error Interaction

  • Is there an interaction between the propensity to respond and the propensity to respond correctly?

  • Are reluctant respondents more likely to be misclassified by survey questions?

  • What can be done to obtain accurate data from reluctant respondents?

  • If so, what are some strategies for nonresponse followup that minimize total survey error?

  • What research designs allow analyst to separate nonresponse error and measurement error?

  • What models are helpful for understanding the linkage between measurement error and nonresponse error?

Example: Kristen Olsen’s “Causal” Models















Measurement Process Model

True Value Model

Common Cause Model


  • Papers from the ITSEW 2008 will be available by August 15 on the NISS website

  • Call for papers for ITSEW 2009 will be announced in September

  • ITSEW 2009 will be held June 14-17 in Sweden

  • Papers that examine the interaction between two or more error sources are encouraged.

  • Login