total survey error past present and future n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Total Survey Error: Past, Present, and Future PowerPoint Presentation
Download Presentation
Total Survey Error: Past, Present, and Future

Loading in 2 Seconds...

play fullscreen
1 / 38

Total Survey Error: Past, Present, and Future - PowerPoint PPT Presentation


  • 70 Views
  • Uploaded on

Total Survey Error: Past, Present, and Future. Robert M. Groves University of Michigan and Joint Program in Survey Methodology. Outline. Evolution of “total survey error” (TSE) Weaknesses of the TSE framework Strengths of the TSE framework Future of surveys and research agenda ideas.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Total Survey Error: Past, Present, and Future' - xerxes


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
total survey error past present and future

Total Survey Error: Past, Present, and Future

Robert M. Groves

University of Michigan and

Joint Program in Survey Methodology

outline
Outline
  • Evolution of “total survey error” (TSE)
  • Weaknesses of the TSE framework
  • Strengths of the TSE framework
  • Future of surveys and research agenda ideas
deming 1944 on errors in surveys
Deming (1944) “On Errors in Surveys”
  • American Sociological Review!
  • First listing of sources of problems, beyond sampling, facing surveys
comments on deming 1944
Comments on Deming (1944)
  • Does include nonresponse, sampling, interviewer effects, mode effects, various other measurement errors, and processing errors
  • Omits coverage errors
  • Includes nonstatistical notions (auspices)
  • Includes estimation step errors (wrong weighting)
  • “total survey error” not used as a term
sampling text treatments of total survey error
Sampling Text Treatments of Total Survey Error
  • Deming, Some Theory of Sampling, 1950
    • repeats set of comments in 1944 article
  • Hansen, Hurwitz, Madow, Sample Survey Methods and Theory, 1953
    • 9 of 638 pages on response and other nonsampling errors
    • “errors due to faulty planning,” “coverage errors,” “classification errors,” “publication errors”
sampling text treatment of total survey error
Sampling Text Treatment of Total Survey Error
  • Kish, Survey Sampling, 1965
    • 65 of 643 pages on various errors, with specified relationship among errors
    • Graphic on biases
slide8

Frame biases

“Consistent” Sampling Bias

Sampling Biases

Constant Statistical Bias

Noncoverage

Nonobservation

Nonresponse

Field: data collection

Nonsampling

Biases

Observation

Office: processing

sampling text treatment of total survey error1
Sampling Text Treatment of Total Survey Error
  • Särndal, Swensson, Wretman, Model Assisted Survey Sampling, 1992
    • Part IV, 124 pp. of 694, coverage, nonresponse, measurement error; omits processing error
  • Lohr, Sampling Design and Analysis, 1999
    • 34 of 500 pages, nonresponse, randomized response
health survey methods conferences
Health Survey Methods Conferences
  • Mid 1970’s, “data error archive” idea of Horvitz
  • 1977 key paper, “Total Survey Design: Effect of Nonresponse Bias and Procedures for Controlling Measurement Errors,” Kalsbeek and Lessler
total survey error 1979 anderson kasper frankel and associates
Total Survey Error (1979)Anderson, Kasper, Frankel, and Associates
  • Empirical studies on nonresponse, measurement, and processing errors for health survey data
  • Initial total survey error framework in more elaborated nested structure
slide12

Sampling

Variable

Error

Field

Nonsampling

Processing

Frame

Total Error

Sampling

Consistent

Noncoverage

Bias

Nonobservation

Nonresponse

Nonsampling

Field

Observation

Processing

survey errors and survey costs 1989 groves
Survey Errors and Survey Costs (1989), Groves
  • Attempts conceptual linkages between total survey error framework and
    • psychometric true score theories
    • econometric measurement error and selection bias notions
  • Ignores processing error
  • Highest conceptual break on variance vs. bias
  • Second conceptual break on errors of nonobservation vs. errors of observation
slide14

Mean Square Error

construct validity

theoretical validity

empirical validity

reliability

Variance

Errors of

Nonobservation

Observational

Errors

Coverage

Nonresponse

Sampling

Interviewer

Respondent

Instrument

Mode

criterion validity

- predictive validity

- concurrent validity

Bias

Observational

Errors

Errors of

Nonobservation

Coverage

Nonresponse

Sampling

Interviewer

Respondent

Instrument

Mode

nonsampling error in surveys 1992 lessler and kalsbeek
Nonsampling Error in Surveys (1992), Lessler and Kalsbeek
  • Evokes “total survey design” more than total survey error
  • Omits processing error
introduction to survey quality 2003 biemer and lyberg
Introduction to Survey Quality, (2003), Biemer and Lyberg
  • Major division of sampling and nonsampling error
  • Adds “specification error” (a la “construct validity”)
  • Formally discusses process quality
  • Discusses “fitness for use” as quality definition
survey methodology 2004 groves fowler couper lepkowski singer tourangeau
Survey Methodology, (2004) Groves, Fowler, Couper, Lepkowski, Singer, Tourangeau
  • Notes twin inferential processes in surveys
    • from a datum reported to the given construct of a sampled unit
    • from estimate based on respondents to the target population parameter
  • Links inferential steps to error sources
slide20

Representation

Measurement

Inferential Population

Construct

Validity

Target Population

Coverage

Error

Measurement

Sampling Frame

Measurement

Error

Sampling

Error

Response

Sample

Processing

Error

Nonresponse

Error

Edited Data

Respondents

Survey Statistic

key statistical developments in total survey error 1
Key Statistical Developments in Total Survey Error 1
  • Criteria for true values (1951)
    • uniquely defined
    • defined in a manner that purposes of survey are met
    • when possible, defined in terms of operations that can be carried through
  • Essential survey conditions, correlated response variance (1959)
    • conditioning factors for variance estimate
    • factors that affect value of variance
    • often ignore by users of
key statistical developments in total survey error 2
Key Statistical Developments in Total Survey Error 2
  • Correlated response variance (1959)
    • assume no covariance between true values and response deviations
    • assume no correlation of response deviations across interviewers
    • correlated response variance independent of workload
    • use of interpenetration
  • Simple response variance via reinterviews (1964)
    • assumes no covariance between trials
  • Relaxed assumptions of zero covariance of true values and response deviations (1964)
    • combine use of interpenetration and reinterview
  • “Error Profile” of Current Population Survey (1978)
  • Multi-method multi-trait models on survey measures (1984)
    • within-construct, among item covariance as tool for “simple response variance
    • imported from psychometrics
key statistical developments in total survey error 3
Key Statistical Developments in Total Survey Error 3
  • Measurement of imputation error variance through multiple imputation (1987)
  • Total error model for PES (1991)
  • Measurement Errors in Surveys (1992)
    • attempt to juxtapose psychometric notions with survey statistical notions of measurement error
  • Latent class model applications to survey errors (late 1990’s)
key statistical developments in total survey error 4
Key Statistical Developments in Total Survey Error 4
  • Understudied components
    • coverage error variance
    • nonresponse error variance
    • all processing errors
    • biases
slide25

Bias

Variance

summary of the evolution of total survey error
Summary of the Evolution of “Total Survey Error”
  • Roots in cautioning against sole attention to sampling error
  • Framework contains statistical and nonstatistical notions
  • Most statistical attention on variance components, most on measurement error variance
  • Late 1970’s attention to “total survey design”
  • 1980’s-1990’s attempt to import psychometric notions
  • Key omissions in research
weaknesses of the common usage of total survey error 1
Weaknesses of the Common Usage of “Total Survey Error” 1
  • Exclusions of key quality concepts
    • notably a user perspective is missing
weaknesses of the total survey error paradigm 2
Weaknesses of the Total Survey Error Paradigm 2

2. Lack of routine measurements

- error/quality profiles are useful but rare

3. Ineffective influence on professional standards

- little expansion beyond sampling error in practice

- press releases on Federal statistics rarely contain even sampling errors

4. Large burden on design of some estimators

-interpenetration, reinterviews for variance estimation not routine

5. Patently wrong assumptions

- correlation of true values with errors (drug use reports)

- intraclass correlation independent of workload (interviewer experience effects)

strengths of the total survey error framework
Strengths of the Total Survey Error Framework
  • Taxonomic decomposition of errors
    • nomenclature for different components
  • Separation of phenomena affecting statistics in different ways
    • variance vs. bias; observation vs. nonobservation; respondent/interviewer/measurement task; processing
  • Conceptual foundation of the field of survey methodology
    • subfields defined by errors
  • Tool to identifying lacunae in the research literature
    • e.g., where are the error evaluation papers on processing?
a vision of the future
A Vision of the Future

Why “total survey errorists” have permanent employment opportunities:

  • new areas of study using surveys
  • populations being studied change their behavior
  • external world offers new measurement tools
needed steps in a research agenda for total survey error 1
Needed Steps in a Research Agenda for Total Survey Error 1
  • Integrating causal models of survey errors
    • cognitive psychological mechanisms (anchoring, recall decay)
  • Research on interplay of two or more error sources jointly
    • e.g., nonresponse and measurement error
  • Research on the interplay of biases and variances
    • e.g., does simple response variance increase accompany some response bias reductions (self-administration effects)
  • Development of diagnostic tools to study model-error
    • progress is slowed with just-identified models without sensitivity analysis
needed steps in a research agenda for total survey error 2
Needed Steps in a Research Agenda for Total Survey Error 2

5. Guidance on tradeoffs between quality measurement and quality maximization

- how much should we spend on quality enhancement vs. measurement of quality (Spencer, 1985)?

6. Integrating other notions of quality into the total survey error paradigm

-if “fitness for use” predominates as a conceptual base, how can we launch research that incorporates error variation associated with different uses?

7. Exploiting a multiple-mode, multiple frame, multiple phase survey world

- how can we build models that exploit non-randomized design variations to give insight into various survey errors?

a summary pitch for the union of design and estimation
A Summary Pitch for the Union of Design and Estimation

“To err is human, to forgive divine – but

to include errors in your design is statistical.

Kish, 1977