1 / 39

From Dual-Frame to Triple Frame:

This study explores the coverage bias in a telephone survey design that combines Random Digit Dialing (RDD), directory-listed, and cell phone samples. It compares the effectiveness and cost of using a triple-frame design versus a traditional RDD+Cell design. The study aims to shed light on the benefits and limitations of each approach.

housel
Download Presentation

From Dual-Frame to Triple Frame:

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Presented at AAPOR 2011 Phoenix, AZ May 15, 2011 From Dual-Frame to Triple Frame: An Assessment of Coverage Bias in a Telephone Survey Design Combining RDD, Directory-Listed And Cell Phone Samples

  2. Thomas M.Guterbock University of Virginia TomG@virginia.edu Abdoulaye Diop Qatar University ADiop@qu.edu.qa James M. Ellis University of Virginia jme2ce@virginia.edu John Lee Holmes University of Virginia jlh2r@virginia.edu Trung Kien Le Qatar University kienle@qu.edu.qa Authors

  3. Overview • Why triple-frame? • The 2010 Behavioral Study of the NCR • Coverage and distribution of phone service (5 segments) • Contrast of RDD, EWP and Cell Phone frames • Calling efficiencies • Comparing substantive results: • triple frame vs. RDD+Cell • Cost comparisons • Conclusion

  4. Why triple-frame? From one frame, to two, to three

  5. From RDD to Dual-frame • “Traditional” list-assisted landline Random Digit Dialing is beset with problems • Increasing under-coverage due to Cell Phone Only (CPO) households • Lower working-number rates • Declining response rates • Dual-frame telephone surveys are now standard for many survey organizations • Combining landline RDD and cell phone RDD frames • Most often using an ‘overlap’ or ‘all cell’ design • See AAPOR Task Force 2010 for full discussion • Adding cell phones covers the CPO’s and favorably alters reachability of dual-phone households

  6. Consider: Electronic White Pages(directory listed) sample • Previous studies have compared EWP to landline RDD sample in statewide, regional, local studies. • Guterbock et al. 2003, Oldendick et al. 2004. • Guterbock, Diop & Holian 2007 explored race and other predictors of listedness in a survey of the National Capital Region • Caution: These studies pre-date the surge in CPO households

  7. EWP vs. RDD • Substantive results are similar, in general • Advantages of EWP: • More efficient • Lower cost • Greater geographic specificity • Disadvantages: • Undercoverage of minorities, lower income, renters • African Americans less likely to have listed numbers • Blacks therefore under-represented in EWP samples

  8. Who needs RDD? • There are notable similarities between characteristics of unlisted landline households and CPO households • Minorities, the young, renters, lower incomes • Could the gaps in the EWP frame (undercoverage) be filled in by inclusion of the cell phone frame? • We proposed in 2008: EWP+Cell as a dual-frame design alternative to RDD+Cell • Based on analysis of NHIS data through 2006 • Just published in Social Science Research • We presented a favorable comparison of the two designs in three county-based surveys in Virginia (2009)

  9. From Dual Frame to Triple Frame • CSR has completed ten telephone surveys that use a triple-frame design • 4 metro-area based; 6 county-based • All cell phones included—no screening for CPO • All in Virginia or DC metro area • Two reasons for triple-frame design: • To allow comparison of RDD+Cell vs. EWP vs. Cell • Most of these studies needed comparison to earlier years that used landline RDD exclusively • We are at a transitional stage in telephone sampling • Triple-frame designs are a compromise between ‘standard’ dual-frame design and our proposed alternative of EWP+Cell

  10. It’s all about. . . error cost

  11. The 2009 Behavioral Survey of theNational Capital Region

  12. 2009 Survey ofBehavioral Aspects ofSheltering and Evacuationin the National Capital Region Sponsor: VDEM Funding: U.S. DHS 12 12 12

  13. Survey Goal Collect information from residents of the National Capital Region that would predict behavior in the area in the event of an emergency. Included factorial experiment that varied features of a “dirty bomb” attack by terrorists. Ask how residents would respond to specific “shelter-in-place” scenarios What variables have the most effect on behavior? What patterns of evacuation and shadow evacuation should be expected? Where would the evacuees try to go? The resulting data are being used to inform the decisions made by administrators in the region and beyond. Details in CSR’s report, 2010. 13 13 13

  14. Features of the Survey In-depth survey: average interview length 28 minutes Fully supported Spanish language interviews as needed 2,609 interviews conducted by CSR, Sept-Dec 2009. Triple-frame sample design: 1269 Landline RDD completes 898 EWP (directory listed) completes 442 cell phone completes (no screening for CPO’s) RDD sample was backmatched to addresses Advance postcard sent to EWP and backmatched RDD cases Weighting by ownership, race, gender, geography, and type of telephone service Margin of error: +/- 2.3 percentage points After weighting 14 14 14

  15. Coverage and distribution of phone service

  16. What percentage of landline phones are unlisted? • Each respondent was asked whether their landline is listed in the directory. • Dual users reached by cell phone were asked to report on whether their landline is listed • Percent unlisted can be taken directly from the RDD frame. • 19.9% of landline RDD completes are unlisted. • As in our other studies, a small portion of those in the EWP frame report their phones to be unlisted.

  17. What is the CPO percentage? • NHIS has been used as the ‘gold standard’ for weighting by phone service. • NHIS did not provide estimates for this geography. • We used the locally based method of estimation described by Guterbock 2009. • 31.4% of cell phone respondents were CPO’s. • Final estimate: 15.4% of telephone HH are CPO. • After excluding landlines with unknown listed status

  18. 2009 NCR Telephone Universe 2 CELL + ULL 18.1% 3 ULL ONLY 1.3% 1 CELL ONLY 15.4% 4 CELL + LLL 61.0% 5 LLL ONLY 4.2%

  19. What EWP+Cell would cover 2 CELL + ULL 18.1% 3 ULL ONLY 1.3% 1 CELL ONLY 15.4% 4 CELL + LLL 61.0% 5 LLL ONLY 4.2%

  20. Little undercoverage • Households with no cell phone and an unlisted landline phone (ULL only), are but 1.3% of the region’s telephone households. • These are covered by the landline RDD frame, but not covered by the EWP frame. • EWP frame underestimates unlisted percentage. • We weight the unlisted percentage among all landlines to 19.9% (the unlisted percent in RDD).

  21. Contrasting the three frames And the matched vs. unmatched portions of landline RDD frame

  22. Comparing calling efficiency

  23. Comparing key demographics

  24. Comparing substantive results • How are survey results affected when a triple-frame sample is used? • We compare our triple-frame result with the results we would have obtained with an RDD+Cell design. • Both designs are post-weighted to the same control weights: • Ownership and race (joint distribution), gender, 8 counties, type of telephone service (CPO, LLO, dual user reached by landline, dual user reached by cell phone) and listed status

  25. Comparing substantive results

  26. Would leave scene in a dirty bomb attack Minimum hazard level Moderate Maximum Can trust most people Trust local government Has an emergency plan Has an emergency kit Has a meeting place Worry about attack 3 levels of hazard High perceived risk Property damage; injury 3 levels of hazard Agree/Strongly agree: Feel at home where I live I have a lot in common with neighbors Important for me to live in this area Let’s compare 20 variables

  27. Differences for 20 variables Margin of error +/- 2.3%

  28. Cost comparison

  29. Cost factors and assumptions • Assume that a survey of like size had been carried out with a ‘traditional’ dual frame design • Same number of cell phone completions as in our triple-frame design • 442 cell phone completes • 2167 landline RDD completes • RDD uses more sampled numbers • RDD cost per sampled number is higher • Due to extra charge for backmatching • Assume postcard sent to backmatched RDD cases • Cell phone completes get $10 incentive

  30. Triple frame costs

  31. RDD+Cell costs (projected)

  32. Triple frame vs.RDD+Cell costs

  33. Cost summary RDD+Cell would have cost: $159,413 ($61.10 per completion) Triple frame design cost: $143,849 ($55.14 per completion) Substitution of EWP for some of the landline RDD frame saved $15,564 ($5.97 per completion) or 10.8% of the Triple Frame total. Greater savings could have been realized if EWP percentage were larger relative to landline RDD 33

  34. Conclusion

  35. It’s all about. . . error cost

  36. Conclusion For representative general population results, we need to include cell phones in our telephone surveys When combined with the cell phone sample frame, EWP sample frames offer greater efficiency than landline RDD lower cost There are good reasons to retain some landline RDD sample in the mix at this transitional stage Allows direct measurement of unlisted percentage There is little or no loss of accuracy when EWP is substituted for some of the landline RDD frame and cell phones are included in the design Cost savings are considerable (over 10% in this study) 36

  37. Conclusion Three frames are better than two! RDD EWP Cell

  38. References AAPOR Cell Phone Task Force. 2010. New Considerations for Survey Researchers When Planning and Conducting RDD Telephone Surveys in the U.S. With Respondents Reached via Cell Phone Numbers. Available online at www.aapor.org. Guterbock TM, 2009. “Estimating Local Phone Service and Usage Percentages.”: How to Weight the Data from a Local, Dual Frame Sample Survey Of Cell Phone and Landline Telephone Users in the United States.” AAPOR paper. Guterbock TM, Diop A, Ellis JM, Le TK, & Holmes JLP, 2009. “Who Needs RDD–Part II: An Assessment of Coverage Bias in Dual-Frame Designs That Combine Directory-Listed And Cell Phone Samples.” AAPOR poster, Hollywood FL. Guterbock TM, Diop A, Ellis JM, Le TK, & Holmes JLP, 2011. "Who Needs RDD? Combining Directory Listings with Cell Phone Exchanges for an Alternative Telephone Sampling Frame". Social Science Research 40:3 (May): 860-872. Also presented as 2008 AAPOR paper, New Orleans. Guterbock TM, Diop A, & Holian L, 2007. “White pages, white people: Reasons for the low listed-phone rates of African-Americans.” AAPOR paper, Anaheim. Guterbock TM, Hartman DE & Hubbard RA, 2003. “RDD vs listed: An experimental study of coverage error, costs and non-response in a statewide telephone survey. AAPOR paper, Nashville. Guterbock TM, Lambert JH, Bebel RA, Ellis JM, & Kermer DA, 2010. Population Behaviors in Dirty Bomb Attack Scenarios: A Survey of the National Capital Region. Prepared for the Virginia Department of Emergency Management. University of Virginia Center for Survey Research, April. Oldendick, Robert W., et al. 2004. “Differences in an RDD and list sample: An experimental comparison.” AAPOR paper, Phoenix.

  39. Presented at AAPOR 2011 Phoenix, AZ May 15, 2011 From Dual-Frame to Triple Frame: An Assessment of Coverage Bias in a Telephone Survey Design Combining RDD, Directory-Listed And Cell Phone Samples Contact: TomG@virginia.edu

More Related