slide1 l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates PowerPoint Presentation
Download Presentation
The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates

Loading in 2 Seconds...

play fullscreen
1 / 51

The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates - PowerPoint PPT Presentation


  • 272 Views
  • Uploaded on

The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates Diane K. Willimack U.S. Census Bureau 10 Economic Census 11 Outline Motivation Economic Census Conceptual framework for nonresponse in establishment surveys

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates' - niveditha


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
the effects of survey design features and economic conditions on business survey response rates

The Effects of Survey Design Features and Economic Conditions on Business Survey Response Rates

Diane K. Willimack

U.S. Census Bureau

10

Economic Census

11

outline
Outline
  • Motivation
    • Economic Census
  • Conceptual framework for nonresponse in establishment surveys
  • Data analysis, results, and conclusions
    • Number of pages
    • Economic conditions
slide4

U.S. Economic Census

  • Establishment level data
  • Collected every 5 years
    • Reference year ends in 2 or 7
    • Collection year is the following year
slide5

U.S. Economic Census

continued

  • Self-administered
    • Paper mail-out/mail-back
    • 2002: Electronic via downloadable software
  • Tailored by industry
    • 2002: 520 different questionnaire versions
motivation

Thank you

for completing the 2002 Economic Census

Motivation
  • Changes in format affecting the 2002 Economic Census
    • Paper size: Legal  Letter
    • Page layout: 2 columns  1 column
  • Number of pages increased
  • Response rates decline???
the data
The Data
  • 1992, 1997, and 2002 Economic Censuses
    • Check-in rates
      • Receipts + 2nd time Undeliverables Mail Counts
    • 1st week of June, 1993, 1998, 2003
  • Unit of analysis = questionnaire version
  • n = 1459 questionnaire versions
research question
Research Question
  • Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?
slide10

A Conceptual Framework forBusiness Survey Participation

Out of Survey Organization Control

Under Survey Organization Control

External Environment

Survey Design

The Business

The Respondent

Business

Goals

Response

Burden

Survey

Participation

Decision

Source: Willimack, D.K., Nichols, E., and Sudman, S., (2002), “Understanding Unit and Item Nonresponse in Business Surveys,” in Survey Nonresponse, Groves, et al. (eds.) New York: Wiley.

defining the model
Concept

(External Environment;

Characteristics of the Business;

Characteristics of the Respondent;

Survey Design)

Measure

Economic Conditions %GDP, %Employment

Industrial Sector

----------------------

#pp for the form, availability of electronic mode

Defining the Model

Response Rate = f

the model
Concept

(External Environment;

Characteristics of the Business;

Characteristics of the Respondent;

Survey Design)

Measure

Economic Conditions %GDP, %Employment

Industrial Sector

----------------------

#pp for the form, availability of electronic mode

The Model

Response Rate = f

main effects model what is the relationship regardless of the census year
Main Effects Model:What is the relationship, regardless of the census year?

Response

Rate = a + b (#pages)

+ [effects of economic variables] + gi (Sector*) + h (Mode)

+ [error term]

* The omitted variable among the Sector dummies is Mining.

care in interpreting results
Care in Interpreting Results
  • Behavioral model
    • Explanatory, NOT predictive
    • Estimated coefficients
      • Relative magnitude
      • Direction (+ or -)
      • Significance
research question16
Research Question
  • “Does 2002 differ from previous censuses?”
  • H0: 1992 = 1997 = 2002
  • Pairwise comparisons
interaction effects model does the effect of pages differ by census year
Interaction Effects Model:Does the effect of pages differ by census year?

= a + b92(#pp) + c92(#pp)2 + d92 (#pp)3

+ b97(#pp) + c97(#pp)2 + d97 (#pp)3

+ b02(#pp) + c02(#pp)2 + d02 (#pp)3

+ [effect of economic variables]

+ gi (Sector*)

+ h (Mode)

+ [error term]

Response Rate

* The omitted variable among the Sector dummies is Mining.

slide23

Estimated Model

1992

Response Rate

Number of Pages

slide24

Estimated Model

1997

Response Rate

Number of Pages

slide25

Estimated Model

2002

Response Rate

Number of Pages

research questions
Research Questions
  • Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?
research question 1 conclusion
Research Question #1 Conclusion
  • Negative effect
    • #pages   response rates 
    • Caution: Behavioral model vs. Controlled Experiment
      • Cannot identify causality
      • Cannot isolate factors
research question 1 conclusion continued
Research Question #1 Conclusioncontinued
  • Reject H0: 1992 = 1997 = 2002
    • 2002 differed from the previous censuses (per results from pairwise comparisons)
  • Negative effect of #pages on economic census response appears to have diminished over time
slide29

A Conceptual Framework forBusiness Survey Participation

Out of Survey Organization Control

Under Survey Organization Control

External Environment

Survey Design

The Business

The Respondent

Business

Goals

Response

Burden

Survey

Participation

Decision

Source: Willimack, D.K., Nichols, E., and Sudman, S., (2002), “Understanding Unit and Item Nonresponse in Business Surveys,” in Survey Nonresponse, Groves, et al. (eds.) New York: Wiley.

slide30

Household SurveyLiterature

  • Better economic times  refusal rates increased (in a household survey)
    • Harris-Kojetin and Tucker, 1999, “Exploring the Relation of Economic and Political Conditions with Refusal Rates to a Government Survey,” JOS, 15:167-184
research questions continued
Research Questionscontinued
  • Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?
  • Did economic conditions affect economic census response rates?
the model32
Concept

(External Environment;

Characteristics of the Business;

Characteristics of the Respondent;

Survey Design)

Measure

Economic Conditions %GDP, %Employment

Industrial Sector

----------------------

#pp for the form,

availability of electronic mode

The Model

Response Rate = f

main effects model what is the relationship regardless of the census year33
Main Effects Model:What is the relationship, regardless of the census year?

Response

Rate = a + b (#pages)

+ e (%GDP)

+ f (%Employment)

+ gi (Sector*) +h (Mode)

+ [error term]

* The omitted variable among the Sector dummies is Mining.

interaction effects model do the effects of economic conditions differ by census year
Interaction Effects Model:Do the effects of economic conditions differ by census year?

= a + [effect of page variables]

+ e92 (%GDP)+ f92 (%Employment)

+ e97 (%GDP)+ f97 (%Employment)

+ e02 (%GDP)+ f02 (%Employment)

+ gi (Sector*)

+ h (Mode)

+ [error term]

Response Rate

* The omitted variable among the Sector dummies is Mining.

do the effects of economic conditions differ by census year testing h 0 1992 1997 2002
Do the effects of economic conditions differ by census year?Testing H0: 1992 = 1997 = 2002
research questions37
Research Questions
  • Was the effect of the number of pages on response rates different for the 2002 Economic Census than for previous censuses?
  • Did economic conditions affect economic census response rates?
research question38
Research Question
  • Does the effect of number of pages on response rates vary with economic conditions?
mixed effects models
Mixed Effects Models

For each census year (i = 1992, 1997, 2002):

= a

+ bi(#pp)

+ ci (#pp)2

+ di(#pp)3

+ ei (% GDP)

+ fi (% Emply)

+ g (Sector*)

+ h (Mode)

Response Rate

+ ji(#pp) * (% GDP)

+ ki (#pp)2 * (% GDP)

+ mi (#pp) * (% Emply)

+ ni (#pp)2 * (% Emply)

+ 

* The omitted variable among the Sector dummies is Mining.

summary of results under the mixed effects model
Summary of Results Under the Mixed Effects Model
  • Variables that become non-significant (with p-values)
    • (#pp)2for 1992 (0.1159)
    • (#pp)3for 2002 (0.3753)
summary of results under the mixed effects model continued
%GDP (0.0282)

(#pp) * (%GDP) (0.0103)

(#pp)2 * (%GDP) (0.0087)

%Emply (0.0322)

(#pp) * (%Emply ) (0.0117)

(#pp)2 * (%Emply ) (0.0082)

Summary of Results Under the Mixed Effects Modelcontinued

Economic variables that are significant(all are for 2002)(with p-values)

what was happening in the u s economy in 2001 2003
What was happening in the U.S. economy in 2001-2003?
  • Economic recovery – beginning to emerge from a recession.
what was happening in the u s economy in 2001 200344
What was happening in the U.S. economy in 2001-2003?
  • Economic recovery – beginning to emerge from a recession.
    •  GDP increasing sharply
    •  Employment declining, “bottoming out”
  • Unstable economy
    • Sharp opposite movements in GDP and Employment
what is the nature of the interaction between economic conditions and pages
What is the nature of the interaction between economic conditions and #pages?
  • GDP and Employment effects tend to be offsetting.
    • GDP moderates the page effect on response rates.
    • Emply exacerbates the page effect on response rates.
  • Magnitude of Employment effect is larger.
what is the nature of the interaction between economic conditions and pages continued
What is the nature of the interaction between economic conditions and #pages? continued

Employment effect

Response Rates

Number of pages

economic effects postscript economic theory of lagged expectations
Economic Effects Postscript: Economic Theory of Lagged Expectations
  • In-depth examination of effects of quarterly employment change on response rates
    • Statistically significant impact of changes 4 quarters and 3 quarters prior to beginning of economic census data collection.
interpreting the economic effect
Interpreting the Economic Effect
  • Unstable economy
    • Employment decline and GDP growth
  • Uncertainty
    • Riskier to divert resources to non-productive activities
  • Cost-benefit
    • Survey response results in a cost with no associated productive benefit.
interpreting the economic effect continued
Interpreting the Economic Effect continued
  • The bottom line
    • In a risky economic environment…
    • Employees are too valuable to devote to filling out forms.
    • Forms that appear longer (#pages) are presumed to take longer and be more costly to complete…
    • Response rates are in jeopardy!
summary number of pages
Summary:Number of Pages
  • Negative effect
    • #pages   response rates 
    • Cubic functional relationship
  • Negative effect of #pages on economic census response appears to have diminished over time
summary economic conditions
Summary:Economic Conditions
  • Evidence that economic conditions interact with form length to affect response rates
    • Mainly through changes in employment
  • Survey response = non-productive cost
    • Corroborates qualitative research findings