1 / 47

A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar Priming Evaluation

SIPP Basics. National panel survey ? Since 1984 with sample size between about 11,000 to 45,000 interviewed householdsThe duration of each panel varies from 2? yrs to 4 yrsThe SIPP sample is a multistage-stratified sample of the U.S. civilian noninstitutionalized populationThe survey uses a

valora
Download Presentation

A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar Priming Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. A Comparison of Survey Reports Obtained via Standard Questionnaire and Event History Calendar * Priming Evaluation * Jeff Moore, Jason Fields, Gary Benedetto, Martha Stinson, Anna Chan, and Jerry Maples U.S. Census Bureau XXV International Methodology Symposium: Longitudinal Surveys: from Design to Analysis Symposium international sur les questions de méthodologie Statistics Canada October 27-30 2009 Gatineau, Qc, Canada

    2. SIPP Basics National panel survey – Since 1984 with sample size between about 11,000 to 45,000 interviewed households The duration of each panel varies from 2˝ yrs to 4 yrs The SIPP sample is a multistage-stratified sample of the U.S. civilian noninstitutionalized population The survey uses a 4-month recall period – 3 interviews / year The sample is divided into 4 rotation groups for monthly interviewing Interviews are conducted by personal visit and by decentralized telephone

    3. The Save the SIPP Campaign New York Times Editorials Discovering What Happens Next Save our SIPP The Continuing Saga of SIPP NY Representative, Carolyn Maloney stated: “I want to thank all of the policy groups and economists who worked so hard to help reverse the Administration’s original SIPP decision. Because of their dedication and hard work, the Administration came to understand how important SIPP is to creating and implementing good public policy.”

    4. SIPP Re-Engineering Implement Improvements to SIPP - reduce costs - reduce R burden - improve processing system - modernize instrument - expand/enhance use of admin records Key Design Change: - annual interview, 12-month reference pd., event history calendar methods

    5. EHC Interviewing Human Memory - structured/organized - links and associations EHC Exploits Memory Structure - links between the occurrence and timing of events EHC Encourages Active Assistance to Rs - flexible approach to help elicit an autobiographical “story”

    6. Evaluations of EHC Methods Many EHC vs. “Q-List” Comparisons - various methods - in general: positive data quality results BUT, Important Research Gaps - data quality for need-based programs? - comparison to 4-month reference period?

    7. Field Test Goals & Design Basic Goal: Can an EHC interview collect data of comparable quality to standard SIPP? - month-level data - one 12-month ref pd interview vs. three 4-month ref pd interviews - especially for need-based programs Basic Design: EHC re-interview of SIPP sample HHs

    9. Stakeholder Concern About Test Design

    10. Design Details (1) Main Sample: SIPP 2004 Panel Cases from Waves 10-11-12 - reported on CY-2007 via SIPP “Primed” Supplemental Sample: SIPP Wave 8 Sample Cut Cases - dropped from SIPP in 2006; “Unprimed” EHC Re-Interview in 2008, about CY-2007

    11. Design Details (2) EHC Questionnaire - paper-and-pencil - 12-month, CY-2007 reference period - selected SIPP topics (“domains”) - start with landmark events - within domains, anchor on “now” - month-level (at least) detail Field Period: Mid-April thru Late June 2008 Sample of Addresses, Not People - post-interview clerical match to SIPP

    12. Control Card – 2008 Field Test Instrument

    13. Page 1 of 5 – 2008 Field Test Instrument

    14. Design Details (3) $40 Incentive, Non-Contingent and Non-FR-Discretionary Same Response Rules as SIPP - EHC interview for all adults (15+) - self-response preferred (proxy permitted) Field Staff: Census Bureau FRs - most with some interview experience - ~1/3 with SIPP experience - 3-day training on EHC methods

    15. Evaluation Plans Compare SIPP and EHC Survey Reports - Same People - Same Time Period - Same Characteristics Evaluation of “Priming” Effect Additional Evaluations: - Administrative Records Evaluation - Training Evaluation - Field Process Evaluation

    16. Design Details (4) TOTAL Initial Sample Addresses Illinois Texas Total 2007 SIPP Respondents: 487 609 1,096 SIPP Sample-Cut Cases: 427 422 849 1,945 Outcomes: Household interviews Total: 1,627 (91%) 2007 SIPP Respondents: 935 (91%) SIPP Sample-Cut Cases: 692 (91%) Individual Adult (15+) EHC interviews Total: 3,318 (99%) 2007 SIPP Respondents: 1,922 (99%) SIPP Sample-Cut Cases: 1,396 (99%) EHC Adults matched to SIPP Total: 2,756 (83%) 2007 SIPP Respondents: 1,658 (86%) SIPP Sample-Cut Cases: 1,098 (78%) FINAL ANALYSIS SAMPLE: 2007 SIPP Respondents: Total 1,620 Non-movers 1,420 Movers 200 SIPP Sample-Cut Cases: Total 1,090

    17. What were the overall research questions?

    18. What were the results? Almost All SIPP and EHC Reports Agree - all characteristics, all months - in general: 97-98% likelihood that a respondent’s SIPP and EHC reports will agree - worst case (employment): 92-94% Disagreements are RARE EVENTS Results available: http://www.census.gov/sipp/DEWS/moore-aapor2009slides-abbrev.ppt

    19. Results Summary for Reinterview Evaluation 3 Patterns: 1. EHC = SIPP All Year SSI; WIC (IL) 2. EHC < SIPP All Year Medicare; Social Security; WIC (TX); Food Stamps (IL) 3. EHC < SIPP, Early in the Year Only Food Stamps (TX); employment; school enrollment

    20. The Movers The ‘Main Sample (Wave 11)’ of EHC respondents who reported for CY-2007 has an important difference from our ‘Supplemental Sample (Sample Cut).’

    21. The Movers The two subsamples of EHC respondents show different characteristics, and one primary cause are the Wave 11 respondents who moved between June 2006 and mid-2007.

    22. The Movers

    23. Assessment of Priming (1) SIPP Wave 10-11-12 Respondents Provide CY-2007 Data Twice - first SIPP, then EHC - results presented previously Are Their EHC Reports Biased by having responded to SIPP for CY-2007 already? - e.g., more accurate EHC response - could bias field test interpretation Control Group: Wave-8 Sample Cut - last SIPP response in Jun-Sep 2006 - “Unprimed” re: CY-2007 (however still SIPP experienced)

    24. “Priming” - SSI (Supplemental Security Income)

    28. “Priming” - WIC (Women, Infants, and Children)

    31. “Priming” - Work

    34. “Priming” - Food Stamps

    37. “Priming” - School Enrollment

    40. Priming - Findings No evidence of priming with respect to having reported about CY-2007 in SIPP. - For each characteristic - Non-Movers from W11 and Sample-Cut cases are not different - Interaction b/w these groups and months are not different - Results hold for both weighted and un-weighted models

    41. Results Implications Levels of agreement from the reinterview comparison between SIPP and the EHC - are not artificially inflated due to experience as reinterview respondents re: CY-2007 - may still be biased because both the main reinterview sample and sample-cut respondents were long-time SIPP respondents.

    42. Field Test Overall Summary Successful “Proof of Concept” Overwhelming Finding: SIPP-EHC Agreement Valuable Lessons to Inform Next Test - larger, broader sample - “correct” timing of field period - automated questionnaire Specific Data Comparisons are Instructive

    43. Comments: Jason Fields – Jason.M.Fields@Census.Gov Jeff Moore – Jeffrey.C.Moore@Census.Gov

    45. Page 2 – 2008 Field Test Instrument

    46. Page 3 – 2008 Field Test Instrument

    47. Page 4 – 2008 Field Test Instrument

    48. Page 5 – 2008 Field Test Instrument

More Related