The Future of Survey Research: A Panel Discussion

The Future of Survey Research:A Panel Discussion Robert Groves Ronald Langley Burke Grandjean March 6, 2008 Survey Research CenterUniversity of California, Berkeley Berkeley, California

A Few Words about SRC, Berkeleyon its 50th Birthday

Homogeneity-Heterogeneity,Batch-Flow, Fixed-Adaptive, Centralization-Decentralization Robert M. Groves University of Michigan and Joint Program in Survey Methodology

Four Powerful Social and Economic Forces • From homogeneity to heterogeneity • From batch to flow • From fixed to adaptive • From centralized to decentralized to… a third way?

Homogeneity to Heterogeneity: Views of Society • Traditional assumption that democracies and justice systems demanded equal treatment to achieve equitable treatment • Increasing observation that modern societies are far from homogeneous • Note: this itself is a finding of large scale surveys

Inter-Nation Migration • The past few years have seen large increases in migration across national boundaries • These appear to be driven by macro-economic and demographic shifts • low fertility rates and aging of rich countries • migrants from poorer countries attracted by job markets of rich countries

Gross Inter-Nation Migration 1950-2040 United Nations Department of Economic and Social Affairs/Population Division 83 World Population Prospects: The 2004 Revision, Volume III: Analytical Report

Social Forces • This migration gives rise to changes • impact on educational systems • civic participation in “old” and “new” social organizations • new political cleavages

Heterogeneity in Marketing “Segmentation” partitions a general population with the goal of tailoring actions toward specific subgroups. It is a way of organizing customers into groups with similar traits, performance characteristics or expectations. Edward J. Hass, Ph.D. An Overview of Segmentation: Why You Should Consider It And a Thumbnail of Its Dynamics http://www.icrsurvey.com/docs/segmentation_white_paper_final_111505.doc

Heterogeneity in Survey Sample Design • Common practice in 1960-1980 to have equal probability samples in US surveys • Common practice now to have special population surveys or full population surveys with disproportionate allocation

Heterogeneity in Respondent Recruitment • Common in 1970’s-1980’s research to look for the single best argument to present to all sample units • Common now to attempt to tailor argument to characteristics and concerns of the sample unit

Heterogeneity in Survey Measurements • Common in the 1970-80’s to insist on uniform measurement for all respondents • Suchman and Jordan’s (1990) attack that standardized wording may not produce consistent meaning • Conversational interviewing (1997) • Mixed-mode data collection Note: CAI enabled this

Homogeneity to Heterogeneity: Conclusion • The more we learn, the more we see heterogeneity • The more we see heterogeneity, the more our survey tools must be suitable for the diversity we see

Batch to Flow • Much 20th century manufacturing • sequential • small steps • many “handoffs” across work groups • low skill levels, narrow focus • Increasingly • system-thinking • project design, integrated skill sets • whole product development • simultaneous work processes

Batch to Flow – Commercial enterprises • “Just in Time” supply chains • to create consistency in assembly schedule • process flow production/ “lean manufacturing” • minimal interruptions in actual processing between runs • queue time is virtually eliminated by integrating the moving of the product into the actual operations of the resource performing the work • Transaction-based processing systems • Daina Dennis; Jack Meredith (Aug., 2000), “An Empirical Analysis of Process Industry Transformation Systems,” • Management Science, Vol. 46, No. 8. pp. 1085-1099.

Batch to Flow: Survey Designs • Move from discrete, repeated cross-sections to continuous interviewing • National Health and Nutrition Examination Survey • National Survey of Family Growth • American Community Survey • UK Continuous Population Survey

Batch to Flow: Summary • Computer-assistance gains its power by unifying former disparate batch steps • Most CAI has focused on unifying question delivery, data input, editing, and data delivery • This leaves sample design, sample unit recruitment, field administration, estimation

Fixed Designs, Fixed Production • Common for manufacturing and surveys to design all the details then construct the product in a unchanging fashion until the sufficient volume is achieved

Adaptive -- Clinical Trials In a classical clinical trial, patients are allocated to one of two different treatment options with half being assigned to each therapy. At the end of the experiment a decision is made as to which treatment is more effective. In contrast, in an adaptive clinical trial, patient outcomes can be used as they become available to adjust the allocation of future patients or some other aspect of the study design. This allows researchers to improve expected patient outcomes during the experiment, while still being able to reach good statistical decisions in a timely fashion. Scott Gottlieb, 2006, 2006 Conference on Adaptive Trial Design,Washington, DC http://www.fda.gov/oc/speeches/2006/trialdesign0710.html

Adaptive -- Computer Science Adaptive designs also arise naturally in computer-related areas such as computer systems resource allocation or parameter selection for repeated computer simulations, since the computer can both collect data and run a program to decide what to do next.

Adaptive -- Commercial products Build-to-Order is the capability to quickly build standard or mass-customized products upon receipt of spontaneous orders without forecasts, inventory, or purchasing delays. These products may be shipped directly to individual customers, to stores or dealers, or as a response to assemblers’ "pull signals" (assemblers’ signals that certain parts are needed right away for assembly). David M. Anderson, (2004)o-Order & Mass Customization; the Ultimate Supply Chain Management and Lean Manufacturing Strategyfor Low-Cost On-Demand Production without Forecasts or Inventory,"

Adaptive – Responsive Survey Design • Preidentify a set of alternative features potentially affecting costs and errors of statistics • Identify a set of indicators of the cost and error properties of those feature • Monitor indicators in initial stages of data collection • Alter the active features of the survey based on cost/error tradeoff decision rules • Combine data from separate phases into a single estimator

Examples of Responsive Survey Design Features • Double samples for nonrespondents without shutdown of field operations • Dynamic adjustment of callbacks to sample units most underrepresented in the data set • Dynamic assignment of incentive values • Change of mode of data collection based on empirical triggers

From Fixed to Adaptive: Summary • Adaptive requires information that identifies heterogeneous sets • Surveys have developed over the years to • use sampling frames that offer good coverage, whether or not they have rich variables • limit data collected to substantive variables

Centralization • Hierarchies • Command and control • Downward communication • Maximize information at the top of the hierarchy

Decentralization • Flat structure • Local autonomy • Local knowledge communicated upward • Only common information needed at the top

The Growth of “Third Ways” • Outsourcing of responsibilities to experts • Centralization of production information shared by all • Separation of common good actions from local optimization actions

The False Dichotomy of Centralization and Decentralization • How can an organization be both centralized and decentralized? • Centralized information about dispersed production • Dispersed use of centralized information

Third Way: Surveys • Use of local supervision for motivation and skill-transmission • Use of centralized data bases to guide targeted deployment of resources to benefit the common good

Challenges from these Forces • The customized survey design needs new paradata to tailor designs to sample units • Software functionality must be designed with responsive design in mind • real-time analysis of paradata • integrated statistical analyses to direct question flow or recruitment step actions • This requires centralization of information resources, but dispersed use of the resources

Technological and Legal Challenges: The View from a Small Shop Ronald E. Langley Survey Research Center University of Kentucky The University of Kentucky

Language Problems? • Nearly 12% of US residents are born outside the US • 18% speak a language other than English at home • Nearly 5% do not speak any English at all Source: National Health Interview Survey/ M. Link – The Neilsen Company

Language Problems? • Solutions available with enough resources • Translation • Foreign language interviewers • Foreign Language supervisors/monitors?

Other Forces • Human Subjects Protections and Right to Privacy • Technological advances • New laws and regulatory interpretations

Social Climate • Increasing desire for privacy • Increasing thirst for data • Cell Phone more “private” than home phone?

Technology • Help us adapt • CAI • Create new challenges to which we must adapt • Privacy Managers • Answering Machines • Caller ID/Call Blocking • Cell phones

Legal Challenges • Laws protecting privacy • FERPA • HIPPA • Human Subjects Protections • Laws and regulatory interpretations in response to technological advances • Number Portability • TCPA 1991 • DNC Implementation Act 2003 • CAN SPAM Act

TCPA 1991 • Prohibited using automatic dialers or prerecorded or artificial voice to contact telephone numbers assigned to cell phones……… • Without prior express consent of the called party • Or to any service for which the called party is charged for the call • Content neutral – includes research surveys

TCPA 1991 • Solicitors can only call between 8 a.m. and 9 p.m. • Solicitors must maintain DNC list to be honored for 10 years • Originally ineffective because burden on respondent to get on each telemarketers list • Fixed by the DNC Implementation Act establishment of National DNC Registry

State Level Legal Challenges • State DNC lists • Harassment Laws • Utah • Missouri • Hawaii • Montana • Time of day restrictions • May not distinguish between solicitation and research

Cell Phones • Affect our coverage • Cell only H/H’s • Affect our Estimates? • BRFSS mixed mode results • Data Quality questions • Lack of Knowledge about the frame • Little prescreening data • More difficult to “fix” by weighting • Ethical issues • Respondent safety • Privacy re: sensitive questions • Cost to respondents

The cell phone problem: Trend in percentage ofU.S. households without landline telephones Source Source: National Health Interview Survey Source: National Health Interview Survey/ M. Link – The Neilsen Company

Cell Phone Only Households • Cell phone-only highest among: – Adults with roommates (54%) – 18-30 year olds (~27%) – Renters (26%) • Does it really make a difference? – Little change in overall estimates when cell only added in; – BUT … are 18-34 year olds in households with landlines the same as those in cell only households?

18-34 year olds: Binge drinking &HIV test by household phone access Source: 2006 BRFSS ABS mixed-mode survey (mail survey respondents only) Michael Link – The Neilsen Company

Solutions to Coverage Issue • Mixed Mode • Collecting better (sub-national) information about cell phone and cell only user population parameters • Address Based Sampling • RTI • CDC/BRFSS • Neilsen • More fully develop Web-based as access increases

The Future of Survey Research: A Panel Discussion