Computer Audio Recording: A Practical Technology for Managing Survey Quality M. Rita Thissen, Hyunjoo Park, Mai Nguyen

1. Computer Audio Recording: A Practical Technology for Managing Survey QualityM. Rita Thissen, Hyunjoo Park, Mai Nguyen International Total Survey Error Workshop Québec City, QC, June 2011

2. Overview • Survey process steps • Technology and methodology of CARI* • Incorporating CARI into the survey process • Addressing structural and functional errors • CARI adoption by the US Census Bureau • Overall implementation plan • American Community Survey Content Test use of CARI for behavior coding • User feedback and future possibilities *CARI: Computer Audio-Recorded Interviewing

3. Survey Process Steps • Define research objectives • *Choose methods of data collection • *Detailed planning, including quality assurance plan • *Construct and pretest questionnaire • *Collect data • *Analyze and report * All except the first step may be affected by use of audio recording methods

4. CARI Technology • Collect audio recordings during actual interview • Capture screen image and response data for comparison with recorded audio • Monitor recordings centrally after the interviewing session ends • Quantify content through coding • Review coding results and choose what action to take, if any

5. CARI System Concepts • Support for total error management • Multimode • Multi-purpose • More efficient and less burdensome than real-time monitoring

6. ISSUE Concern about mode effects between telephone (CATI) and in-person (CAPI) interviews Determine what data points might be sensitive to mode effects Difficulty of data capture in semi-structured interviews or free-response items Standardized coding, such as occupation and industry, or drug and disease codes APPROACH Audio recordings for comparison across “electronic” modes Plan to record those questions and responses, plus “buffering” by recording adjacent items Capture open-ended responses through audio recording for later coding Allow standardized coding by centrally-located experts instead of by interviewers Survey Step 2: Choose Data Collection Methods

7. ISSUE QA plan is for detection, control and remediation of error Data entry challenges: “Other, specify” questions, long lists of response options, “Code all that apply”, or free-form responses Effort and expense of live-monitoring, field observation and verification APPROACH Build CARI capabilities into QA processes for data quality and performance management Can use audio recordings for primary data entry or as a form of double data entry to confirm/correct data from CATI or CAPI May be able to reduce costs through use of recordings for performance review Survey Step 3: Quality Assurance (QA) Plan

8. ISSUE Question-level problems: Wording, ambiguity, lack of clarity Translation effects: Dialects, regionalisms or country-specific vocabulary may not be recognized until production Key data points need extra attention APPROACH Use recordings and behavior coding during pretest and production to identify issues Review of production responses by bilingual staff can confirm correctness or identify problems Select questions to record, including buffer zones as needed. Survey Step 4: Construct and Pretest the Questionnaire

9. ISSUE Managing interviewer performance & adherence to protocol Motivating interviewers when the going gets rough, especially late in a cycle Responsive design or adaptive total design; when subpopulations don’t respond APPROACH Frequent review of a sample of cases helps “keep an eye” on interviewing Positive feedback can reinforce good behavior and provide encouragement Review of recordings from targeted subpopulations may give insight into reluctance Survey Step 5: Collect Data

10. ISSUE Quantifying operational data for methodology reports Explaining difficulties or unusual situations encountered Estimating survey error APPROACH CARI coding contributes to paradata metrics Audio recordings may provide insight into or detailed examples of the challenges Coded monitoring results offer a way to standardize measurement of ‘subjective’ error, such as consistency of questionnaire delivery Survey Step 6: Analyze and Report

11. Addressing Structural Errors Translation errors Regional effects, such as vocabulary differences (Do you drink soda, pop or soft drinks? Would you drink from a water fountain?) Logic errors: complex questionnaire logic may hide errors, in question sequences rarely followed Values computed based on earlier responses and then used as fills; errors may be hard to detect during testing

12. Addressing Functional Errors Human error: Data entry mistakes, presentation or mode effects, respondent reactions Advance warning of mode effects, indications prior to analysis Additional information about item-level non-response, including both “don’t know” answers and refusals

13. CARI Adoption by the US Census Bureau CARI Interactive Data Access System Multiple-year project, 2009 - 2012 Stage 1: Develop behavior coding and quality assurance (QA) interfaces (completed) Stage 2: Field test for behavior coding, recommendations for revision (completed) Stage 3: Enhancements and revisions for BC and QA and addition of a Coaching component (in progress) Stage 4: Field test for QA and Coaching (future)

14. Behavior Coding Field Test • American Community Survey (ACS) Content Test • Data collection, 2010 • Mail: August – September • CATI: October • CAPI: November • Behavior coding immediately after data collection • Dec 2010 – March 2011 • Data analysis in progress at the Census Bureau (Contact: Joanne Pascale)

15. ACS Content Test

16. Coding Page • Compact, feature-rich interface • Playback for audio • Option to view image • Navigation bar at left (blue) for non-sequential access to recordings • Next / Prev buttons for sequential access • Dynamic code lists are structured into categories, groups and codes • Grid display summarizes codes assigned • Room for notes

17. General Feedback Features that the CARI System users found most valuable included the following: • Unobtrusive nature of CARI • Ability to select cases (or questions) for review from among those recorded • Flexibility in defining what behaviors to code • Image display offering the exact display of wording (including fills) as well as the actual data entry value • Real-time monitoring of coding • Data available for extraction at any time

18. Future Possibilities for Error Management • Greater availability of recordings as a routine part of data collection • Expanded use of recordings in performance management and training or retraining of interviewers • Expanded use of CARI paradata in survey management • Development of low-burden performance metrics for interviewers and coders • Creation of an “error profile” for a survey • And… whatever else you can imagine!

19. Acknowledgements This project would not have been possible without the support of many people at the Census Bureau and RTI. We especially thank Sherry Thorpe, CARI Project Manager, Joanne Pascale, and the ACS Office of the Census Bureau. For more information, contact: Rita Thissen – rthissen@rti.org Hyunjoo Park – mpark@rti.org Mai Nguyen – mnguyen@rti.org