Some type of major TSE effort

1. Some type of major TSE effort TSE for an “important” statistic Form a group to design a TSE evaluation for existing survey “exemplary TSE estimation plan” Develop group to design reliability and validity measures – US version of EU database.

2. Predictive Models Predictive models for response, other types of TSE components Interdisciplinary teams to motivate models for TSE (paradata, process data): fixed effects to reduce errors by design IV tenure Random effects for predictive value Probing behavior Model validation: gold standard vs. model fit.

3. Resource allocation Allocating resources to deal with effect of improving response rates: tradeoff b/w errors and costs. How do we understand components of error? How do we handle more than one component of error at a time? Manipulating response propensities Gold standard for estimating measurement error Errors not independent

4. Resource allocation Resource allocation is extremely important Convincing need for tradeoff: what should be the priority for sample size vs. follow-up. Non-response error is currently considered critical; measurement error usually ignored because of lack of gold standard. What could serve as GS? Intensive effort to get high response rate in small sample; try to obtain same info via multiple questions (across surveys, possibly).

5. Meta-analytic work on individual error sources NR bias vs. NR rates Comparable measures across surveys (e.g. why unemployment rate differs b/w CES and CPS – unmeasured error…) Studies to match employers w/ employees Using Census as gold standard Small area estimates for CPS, integrating in 2000 Census data Could use SIPP vs. CPS to compare, e.g, income measures, look at discrepancies using SIPP as “gold-standard” to improve Census measures of income Also matches of administrative data: crosswalk between CPS, tax returns: multi-trait, multivariate method: not assuming admin record correct Meta analysis within each TSE “box,” then conduct “meta-meta” analysis

6. Processing Error Processing error: coding, editing. Improve or introduce error? Are we spending too much (35% in Sweden, 20% in US)? How do me model? Computer science handles this? W. Winkler has compilation of lit.

7. Using TSE to Improve Meta-Analysis Bring in study characteristics. This is done, but maybe survey analysts can inform what the metrics should be. Unless you know a lot about the topic, DK what the metrics should be. Finding information is difficult. Nonresponse error. TSE informs design so that we can get info (reinterview)? Currently contact is validated, but rarely is response variance determined.

8. Simulation Nonresponse, response error, interviewer-respondent interaction.

9. What is the “quality space”? How do our measures affect what the user does with the data? SE gives sample size that is important; what about TSE? Timeliness: need it if future trader, not if historian. Similarly, short-term trends (ignore response bias) vs. long term means (need to account for response bias) Could create datasets that account for TSE vs. “real data” Need to talk to agencies, etc. to learn what is needed? Remember the ultimate goal is to use the research to improve the survey design – otherwise the managers might just view this as info they don’t want to have.