1 / 24

AAPOR’s Transparency Initiative

AAPOR’s Transparency Initiative. RC-33 Eighth International Conference on Social Science Methodology Timothy P. Johnson and Paul J. Lavrakas. Problem: There is not enough transparency in the Social and Behavioral Sciences.

haggerty
Download Presentation

AAPOR’s Transparency Initiative

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AAPOR’s Transparency Initiative RC-33 Eighth International Conference on Social Science Methodology Timothy P. Johnson and Paul J. Lavrakas

  2. Problem: There is not enough transparency in the Social and Behavioral Sciences. • Premise 1: Too often, there is inadequate transparency of research methods and statistical methods when the results of social and behavioral science research are disseminated, especially by non-academic sectors. • Premise 2: This is a serious detriment to progress in these scientific domains.

  3. Openness is fundamental to scientific inquiry. • AAPOR’s Transparency Initiative (TI) is envisioned to encourage openness in survey research so the field can advance. • Openness is fundamental to credibility • TI also is envisioned to enhance the credibility of our profession by encouraging disclosure of methodological information and educating the public on how to use it.

  4. Transparency Initiative Goals • Main: • Advance the science and reputation of survey research. • Secondary: • Provide professional education on survey documentation and disclosure practice • Educate survey sponsors and the public on value of transparency

  5. A Brief Historical Overview • Transparency Initiative proposed by then AAPOR President (and Sociologist) Peter V. Miller in May 2010

  6. Membership Requirements • Membership in AAPOR’s TI is anticipated to be open to any survey organization or consultant, regardless of individual AAPOR affiliations or organizational members • Membership requires commitment to the principles and practices of the Transparency Initiative • Mainly, what will be required: • A pledgeto conform to the AAPOR Code of Professional Ethics and Practice regarding Disclosure (Section III) • Willingness to train employees regarding appropriate transparency practices • AAPOR will develop an online training tool

  7. AAPOR Code, Section III Summary Report Immediately (a) • Who sponsored, conducted & funded the research • Exact question wording • Definition of the population • Geographic location • Sample frame description • Sample design • Sample size & error • Weighting & cluster adjustments • Results based on parts of sample only • Method(s) and dates of data collection Within 30 Days (b-d) • Interviewer/respondent instructions • Relevant stimuli (show cards) • Sampling frame’s coverage • Methods of panel recruitment (for pre-recruited panels) • Sample design details (eligibility, screening, oversamples, incentives) • Sample dispositions • Weighting details • Data verification details • Response rates • All of the above for each if multiple samples or modes

  8. Anticipated Benefits of Membership in TI • TI participants will benefit in the commercial marketplace and in the marketplace of ideas • Clients will learn that publicly released reports will have enhanced credibility if they are based on transparent research methods • The news media will learn that participation in the Transparency Initiative is an important indicator of whether or not to pay attention (i.e., give credence) to a survey organization’s reports

  9. Qualification Process • Two models were considered: • Safe Harbor Model • International Organization for Standardization (aka, ISO) Model

  10. Safe Harbor Model • Run by US Dept of Commerce to provide a mechanism for US research companies to comply with EU data protection legislation • Requires organizations to develop and publish a formal privacy policy and to self-certify via letter and recertify (on an annual basis) that it complies with the standards • Pro: No significant cost burden to organizations • Con: No formal verification process, thus may be viewed as weak

  11. ISO Model • Certification only achieved after auditor visits organization, examines processes, and conducts staff interviews regarding procedure adherence • Annual surveillance audits ensure continued compliance • Pro: Credibility, is based on external certification according to independent standards • Con: Serious costs both to the certifying and the applicant organizations

  12. Current AAPOR TI Proposal Being Considered Self-certification via following steps (annually): • Organization agrees in writing to abide by disclosure requirements of AAPOR Code • Organization agrees to meet the disclosure requirements of the Code via whatever mechanism AAPOR ultimately develops • Organization agrees to train employees in disclosure policy and committing to annual refresher training

  13. Proposed Monitoring Process • Like Safe Harbor, the TI would rely on research community and the public at large for general monitoring • A “complaint driven” process • Complaints of failure to disclose would be filed much like current AAPOR Code violation complaints • Details not yet worked out, but the process would likely be a streamlined version of the current process AAPOR employs for Code violations

  14. Proposal for Handling Violations • “Three Strikes You’re Out” standard • Single violation would not cause an organization to have its TI membership revoked • AAPOR will provide constructive feedback designed to bring violators back into conformance • Three unrelated incidents within a given time period would result in recommendation to AAPOR Council that the organization be disqualified from TI membership for one year • This is all under discussion, no final decisions have been made

  15. 2011-2012 Activities • TI Steering Committee • Members • David Cantor, Leah Christian, Marjorie Connelly, Liz Hamel, Melissa Hermann, Tim Johnson, Scott Keeter, Courtney Kennedy, Paul J. Lavrakas (chair), Peter Miller, Rich Morin, Joe Murphy, Chuck Shuttles, Susan Tibbitts (ex officio) • Working Subcommittees • Subcommittee on Structure, Qualification, and Compliance (Cantor, chair) • Subcommittee on Database Process and Infrastructures (Miller, chair) • Subcommittee on Internal and External Communication and Education (Morin, chair) • Subcommittee on Costs and Staffing (Lavrakas, chair) • TI External Advisory Committee (Kennedy, chair)

  16. 2011-2012 Activities • Pilot Test 1 • Conducted February-March 2012 • Eight volunteer organizations participated • Based on what was learned, TISC proposed to AAPOR Council that: • Transition TISC into the TI Coordinating Committee (TICC) with a stable, multi-year membership • A second pilot test will be run in 2012 based on lessons learned

  17. 2011-2012 Activities • Pilot test focus group • Conducted May 1, 2012 • Included representatives from 6 of 8 organizations who participated in the first pilot test • Additional participants included: • Representatives from other TI Advisory Committee members • TISC members

  18. Key Lessons from Focus Group • There is strong verbal commitment among many AAPOR members to the concept of transparency. • Uncertainty regarding optimal scope of TI. • Should focus be on: • Education? • Certification? • Audit function? • Uncertainty as to whether TI materials should be housed centrally on AAPOR web site or de-centrally on participant web sites.

  19. Key Lessons from Focus Group • Concerns with compliance burden. • Cost of participation too much for some organizations? • Uncertainty as to what surveys should be covered by TI: • All surveys conducted by participating organizations? • Only surveys publically released by participating organizations? • Public Opinion Polls only? • What about reports, press releases, presentations, other media formats?

  20. Key Lessons from Focus Group • Uncertainty as to best strategy for implementing TI: • All-at-once? • In multiple phases? • Should there be multiple levels of TI membership? • It is clear that each of these issues needs to be further considered before we move towards another pilot test.

  21. Current Activities and Plan • Providing membership with ongoing updates of TI activities via AAPOR web page and AAPORnet • Developing TI application and monitoring protocols • Finalizing policies for certification & decertification • Submit proposed TI policies and procedures to AAPOR Council for review and approval • Second pilot test in Fall 2012 • Final recommendations to AAPOR Council Winter/Spring 2013

  22. Transparency Initiative Coordinating Committee (TICC) 2012-13 • Leah Christian • Melissa Herrmann • Tim Johnson (Chair) • Scott Keeter • Courtney Kennedy • Mary Losch • Rich Morin • Trevor Tompson • Tim Triplett

  23. Thank You

More Related