1 / 26

Mixed-Mode Designs in Social Surveys: Investigating Mode Organization and Strategies

This workshop in Rome on April 11th-12th, 2019 explores the mode organization and strategies for mixed-mode designs in social surveys. Topics include concurrent vs consecutive data collection, adaptive survey design, and optimizing the quality/cost trade-off.

guerrier
Download Presentation

Mixed-Mode Designs in Social Surveys: Investigating Mode Organization and Strategies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ROME April 11th | 12th 2019 MIMOD Mixed-Mode Designs for Social Surveys FINAL WORKSHOP WP1: Investigation of mode organisation (concurrent vs consecutive multi-mode data collection) General overview Manuela Murgia-Istat

  2. Overview of WP1: aims and results Deciding the mixed-mode strategy (M. Lo Conte -Istat) Adaptive mixed-mode survey design (B. Schouten - CBS) Conclusions and look a ahead (M. Murgia -Istat) Discussion (F. Beck - INSEE) Session overview 1 2 3 4 5 MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  3. WP1 overview W H O Investigation of mode organisation (concurrent vs consecutive multi-mode data collection) IstatCBS Manuela Murgia Barry Schouten Martina Lo Conte AnnemiekeLuiten Luciano Fanfoni Lucia Coppola Doriana Frattarola Alessandro Fratoni MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  4. WP1 overview W H Y To investigate the mode organisation of survey designs within the ESS. Mixed-mode survey designs  concurrent and sequential mixed-mode designs  adaptive designs Aims: 1) To provide an update picture of different features of survey designs within the ESS so as to provide input information to all WPs. Deliverable 1: report on results of the MIMOD survey on the ‘State of the art of mixed-mode’ 2) To provide guidance for the choice of mode strategy and for adaptive mode allocation for future survey (re)design, searching for the optimization of the trade-off quality/costs Deliverable 2: Methodological report on how to best combine data collection modes. MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  5. WP1 overview C O N T E N T Deliverable 2: Methodological report on how to best combine data collection modes Overview on mixed- mode strategy, a framework of current practises to evaluate our own choices Decision process - two checklists for: • the mode choice: selection of candidates modes and of strategies for their administration (concurrent vs sequential) • the mode allocation: different mode strategies for different sub- population (adaptive design) MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  6. Current practises on mode strategy M I X E D M O D E (*) Calculated on social surveys. (**)Iceland does not run AES and EHIS; Switzerland does not run EHIS and HETUS. MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  7. Current practises on mode strategy M O D E S T R A T E G Y Concurrent design: different data collection modes are in the field at the same time Sequential design: modes are administered in sequential time periods, one after the other Partly sequential - partly concurrent design: the first data collection mode remains on the field when the second one(s) is (are) made available to respondents. Adaptive design: different modes to different sub-populations on the base of frame data, administrative data, paradata (also different effort for timing and number of calls and visits, incentives, etc.)

  8. Current practises on mode strategy M O D E A L L O C T I O N The use of adaptive designs Adaptive designs (mode allocation) are not still rarely used using MIMOD project - Mixed-Mode Designs in Social Surveys Rome, 11-12 April 2019

  9. Current practises on mode strategy S T R A T E G I E S The use of different mode strategies Mixed-mode designs are mainly based on a concurrent approach

  10. Current practises on mode strategy Mixed-mode strategies for the main EU social surveys MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  11. Current practises on mode strategy F E A T U R E S Features of mixed-mode strategies - Modes and mode combinations - Same or different mode sequence in one survey design - Mode choice from respondents - The use incentives MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  12. Current practises on mode strategy M O D E S MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  13. Current practises on mode strategy Concurrent Designs: mode combinations Interviewer administered modes are more frequent: CATI-CAPI and CAPI-PAPI MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  14. Current practises on mode strategy Concurrent Designs: Can respondents choose the mode? P R E F E R E N C E All strategies: Yes 45.5% No 40.2% Yes, but conditionally 14.3% (*) Non-respondents to a mode MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  15. Current practises on mode strategy Mode choice : 3 different strategies in Concurrent Designs • Modes are on the field at the same time, but respondents cannot choose the mode which is assigned a priori by the NSI according to auxiliary variables • Modes are on the field at the same time and allrespondents can choose all modes available from the beginning. High organisational complexity. • 3. Modes are on the field at the same time and are assigned in advance, but respondents can choose whether to accept it or to choose an alternative one. MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  16. Current practises on mode strategy Concurrent Designs: summary • Large variety of mode combinations. Prevalence of interviewer-assisted modes: higher response rate are likely to be reached with a higher impact on costs. • The CAWI mode is less used than in other mode strategies; when CAWI is used then a ‘push to web strategy’ through invitation letter is in general applied. • It is common to allow mode choice from respondents. • When this implies a switch from CAWI to CAPI attention is needed to eventual bias in geographical distribution of interviews and to the management of interviewers workload. MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  17. Current practises on mode strategy PS-PC Designs: mode combinations Same sequence of modes to all sample units The first mode on the field is generally self-administered (CAWI) while interviewer-administered modes (CATI and CAPI) are used at a later stage MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  18. Current practises on mode strategy Sequential Designs: mode combinations Mode combinations are mainly made of two types of data collection designs Same sequence of modes to all sample units The ‘cheapest mode first’ rule generally applies (CAWI first) Differently from PS-PC, second modes are CATI or PAPI - CAPI is more rarely used MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  19. Current practises on mode strategy Same or different design/sequence of modes? EU NSIs generally apply the same data collection design to all sample units. In a few cases, the sequence of modes is different for different sample sub-groups MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  20. Current practises on mode strategy The choice of data collection mode: a respondent’s choice? Sequential designs PS-PC designs Yes No Yes, but conditionally MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  21. Current practises on mode strategy Sequential Designs: summary • Prevalence of ‘cheapest mode first’ rule (CAWI first). • Compared to PS-PC designs, the CAPI mode is less used while PAP(I) is more frequently adopted. • Strategies to reduce the printing costs are usually applied: questionnaire ‘on request’ or downloadable from website • The choice of preferred mode is in general not offered • It is seems they are mainly aimed at cost reduction and when there is the need or the will of keeping the organisational complexity low. MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  22. Current practises on mode strategy Partly sequential-partly concurrent designs: summary • Prevalence of design where CAWI is the first mode followed by CATI-CAPI. • The choice of preferred mode is in general offered to non-respondents of first mode: interviewers trained for a ‘push to web approach’; in-house fieldwork fundamental for this task. • Offering CAWI and re-offering it to non-respondents might limit costs and reduce organisational complexity, but impact of selection effects on estimates has to be evaluated. MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  23. Current practises on mode strategy The use of incentive • In general: • 14 NSIs out of 31 offer incentives to respondents (few NSIs for all surveys) • Incentives are ‘conditional’ incentives (given only to respondents); • Two main forms: monetary and non-monetary, the latter more frequent • In general the same incentive is offered to all respondents with no diversification • The use of incentives seems to be • correlated with the mode: • larger use in case of self-administered • modes • No correlation per mode strategy MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  24. Decision making process • Mixed-mode strategies  ‘plethora of designs’ • How to decide on which mixed-mode survey design to use? • Decision is based on survey errors and costs and also on survey operations • Survey modes= design element with the strongest impact on: • Errors: coverage, response, measurement • Costs: example CAWI vs CAPI • Operational complexity: • self-administered simpler than interviewer assisted mode; • pc-assisted mode simpler that paper modes. • In mixed-mode surveys: different modes are used in one fieldwork with a strong impact on operational complexity • Approaching the decision making process from the mode perspective can potentially allow to take into account all issues of a survey design MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  25. Decision making process: CHECKLISTS Checklist for the choice of mode strategy Checklist for adaptive mode allocation • 1. Identify priorities • 2. Identify major risks (incomparability in time • among groups, budget…) • 3. Define quality and cost indicators (nonresponse, • measurement error, cost indicators) • 4. Define decision rules from (different optimization • strategies) • 5. Modify the survey design and monitor the • outcomes • 6. Compute estimates • 7. Document 1. Determine and prioritize aims (quality-cost trade-off) 2. Identify risks 3. Determine the candidate modes 4. Evaluate different MM design options 5. Implement the mixed-mode design 6. Compute estimates 7. Assess 8. Document MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

  26. Thanks MIMOD - Mixed-Mode Designs in Social Surveys. Final Workshop Rome, April 11-12, 2019

More Related