Alternate Strategies for Including Cell Phone Only Households in Surveys: Comparing the Use of ABS and Cell Phone RDD

1. Alternate Strategies for Including Cell Phone Only Households in Surveys:Comparing the Use of ABS and Cell Phone RDDSamples in Massachusetts David Dutwin, PhD State Health Research Policy Interest Group Academy Health Annual Research Meeting Seattle, WA June 11, 2011

2. Overview • Comparison of alternate sampling strategies to capture cell phone households • Address-based sample (ABS) • Cell phone random digit dial sample (RDD) • Case study in Massachusetts in 2010 • Massachusetts Health Insurance Survey (MHIS) -- ABS and landline RDD • Massachusetts Health Reform Survey (MHRS) -- Landline RDD and cell phone RDD • Recommendations

3. Alternative Approaches in MA • MHIS – ABS & landline RDD • ABS: Random sample of households from the delivery sequence file used by the United States Postal Service -- Includes households regardless of phone status • Landline RDD: Random sample of landline numbers • Estimated coverage rate: 98% of households • MHRS – Landline RDD & cell phone RDD • Landline RDD: Random sample of landline numbers • Cell phone RDD: Random sample of cell phone numbers • Estimated coverage rate: 90 to 95% of households

4. Similarities in MHIS and MHRS • Both fielded every year, including 2010 • Similar questionniares focused on health insurance coverage, access and use • Both draw on SHADAC’s Coordinated State Coverage Survey (CSCS) • Similar sample sizes • Similar costs • Same organizations involved in both surveys, including same survey firm

5. Differences in MHIS and MHRS • Sponsor: state vs. foundation • Sample design • ABS & landline RDD vs. landline RDD & cell phone RDD • Size of landline RDD sample • Oversampling of selected populations • Instrument design • Data collection mode: Phone, internet & mail vs. phone • Focus: all members of household vs. random adult • Respondent selection • Survey fielding • Time period of the survey: spring vs. fall • Modes of contact: mail & some phone vs. phone & some mail • Use of incentives • Response rates & sample coverage

6. Comparison of Selected Elements

7. Advantages for Cell Phone Coverage • MHIS--ABS • Very high coverage rates for USPS in Massachusetts • Share of cell phone only households reached was close to the NHIS estimated proportion of cell phone only households in Massachusetts • Bonus coverage: Also reached people who were using voice over internet phones as well as a few non-telephone households • MHRS--Cell phone RDD • A very high percentage of all cell phone numbers are in cell phone banks that can be sampled • Based on estimates from other sources: Only about 10% of all cell phone owners in MA have non-MA area codes

8. Other Advantages • MHIS--ABS • Able to contact all sample members by mail • Creating survey weights is straightforward using traditional survey procedures since population totals for the sample are available • MHRS--Cell phone RDD • Single mode for entire sample – All CATI telephone interviews using the same core instrument • Cell phone offers potential to reach respondents who are seldom at home

9. Disadvantages • MHIS--ABS • Cannot obtain phone numbers for some households so that only method of contact was mail, leading to lower response rates for those households • Some addresses will be vacant or summer homes, etc. • More item non-response on the mail questionnaire (12% of MHIS ABS surveys from mail) • Greater operational complexity and higher costs • MHRS--Cell phone RDD • Cannot obtain addresses to mail advance letter • Cell phones expire, not in service, business, etc. • Costly to screen out respondents who do not live in Massachusetts (28% of cell phone RDD sample) • Survey weights are challenging since we don’t know the population totals for the cell phone population

10. Comparing All Adults in MHIS and MHRS • Sample is adults 19 to 64; Estimates are weighted; * p < .05

11. Comparing Cell-phone-only Adults in MHIS and MHRS • Sample is adults 19 to 64; Estimates are weighted; * p < .05

12. Summary • Despite their differences, the MHIS and MHRS yielded similar overall samples of the nonelderly adult population AND similar samples of cell-phone-only nonelderly adults in Massachusetts • Both ABS and cell phone RDD appear to be credible strategies for capturing cell-phone only households • However, important to remember: Since don’t know actual population characteristics of the cell-phone-only population we can’t assess whether missing important components of the population with both strategies

13. Recommendations • To states: ABS and cell phone RDD are credible strategies for including cell-phone households in surveys • To researchers/survey methodologists: • Explore methods/develop strategies to increase response rates to surveys across sample frames • Explore methods/develop standards for estimating consistent response rates and household coverage rates across sample frames • Explore implications of mode effects for surveys that rely on multiple modes • Explore methods/develop strategies for combining cell phone RDD and landline RDD samples

14. Thanks and Contact Information Acknowledgement: The MHIS was funded by the Massachusetts Division of Health Care Finance and Policy. The MHRS was funded by the Blue Cross Blue Shield of Massachusetts Foundation. We appreciate that they allowed us to use the survey data to conduct this analysis. Contacts: Nicole Buttermore: Research Director, SSRSDavid Dutwin: Vice President, SSRSTimothy Triplett: Survey Methodologist, Urban InstituteDoug Wissoker: Senior Research Associate, Urban InstituteSharon K. Long: Professor, University of Minnesota