PPA 501 – Analytical Methods in Administration

1. PPA 501 – Analytical Methods in Administration Lecture 3d – Survey research

2. Introduction • Government administrators and elected officials love to claim that they possess a profound understanding of their public’s needs, desires, and disaffection.

3. Introduction • Unfortunately, the administrators and officials are learning that storms of controversy provide meager evidence of the workaday values of the everyday people they govern. • Surveys of the public, conducted following the basic precepts of survey design and analysis, are fast becoming the vehicle for genuine connection to the public will.

4. Introduction • Uses. • Evaluations of government services. • Changing demographics that may signal shifts in service demand. • Patterns of service utilization. • Problem identification. • Customer service.

5. Introduction • Surveys have several important qualities. • Anonymity to respondents. • Point of view, characteristics, or use patterns can be characterized with little confusion. • Good surveys provide input from a representative cross-section.

6. Begin Before the Beginning • The best surveys grow from well-conceived and well-articulated reasons for conducting them. • Resist the temptation to hit the ground running. • Be certain of the purposes of the survey.

7. Begin Before the Beginning • Identify the appropriate audiences. • Identify the political and personal will for doing the survey. • Determine whether the questionnaire to be developed is better as a one-time or periodic survey. • Think about the usefulness of comparative data.

8. Getting Started • Convene a steering committee with key stakeholders. • Enlist the help of top government officials or administrators.

9. Designing the Survey • Sampling. • Choose the appropriate sampling frame: about what population do you wish to generalize? • A sampling plan must give every respondent in the sampling universe an equal chance of ending up in the sample. • Simple random sample. • Stratified sampling. • Stratified random cluster sampling.

10. Designing the Survey • Targeting the individual in the household. • If no list exists, you may only have addresses or phone numbers. If so, use household member with most recent birthday.

11. Mail, Phone, or In-person Interviews • The best ways to conduct surveys vary by accuracy, speed, and cost. • Most common are mail and phone surveys.

12. Mail, Phone, or In-person Interviews

13. Mail, Phone, or In-person Interviews • Increasing response rates. • Multiple mailings (up to three) with stamped, return address envelope. • Press coverage. • Combination of methods often best: Mail survey with telephone and in-person followup.

14. Mail, Phone, or In-person Interviews • Selecting sample size. • The size of sample depends on desired precision of estimates. • Generally speaking, if opinions are split as much as possible, than 100 residents will have a margin of error of +/- 10% with 95 percent confidence. Four hundred residents the margin is +/- 5%. • In general, 100 is a good minimum number, especially for subgroups.

15. Questionnaire Construction • Each question should be judged against the purposes of the survey and the uses to which it will be put. • Steal widely. • National Citizen Survey from International City Management Association and National Research Center.

16. Questionnaire Construction • Major principles. • Consistency. • Clarity. • Vague wording. • Double-barreled questions. • Assumed knowledge. • Overlapping response categories. • Simplicity. • Specificity. • Brevity (30 min. Phone, 60 min. In-person, 10 page mail). • Context sensitivity.

17. Questionnaire Construction • Major principles. • Security. • Demographic at end. • General to specific. • Fairness. • Option symmetry (balanced responses). • Option wording and order. • Background info, pros and cons, opinion. • Randomize pros and cons in a complicated survey.

18. Questionnaire Construction • Open-ended versus closed-ended questions. • Commonality versus depth. • Broad categories of questions. • Factual. • Opinion. • Attitude. • Motive. • Knowledge. • Action or behavior.

19. Questionnaire Construction

20. Questionnaire Construction • Sections. • Title and identification of the survey sponsors (including human subjects information). • Instructions. • Warm-up questions (simple and direct, factual or knowledge). • Body of the questionnaire (more complex) • First third – Awareness and knowledge of factors, indicators, and causes. • Second third – attitude, opinion, motive scales. Lifestyle information (nonresponse goes up). • Last third – Focused questions, controversial, personally embarrassing. Partially completed is better than not completed. • Classification items. • Demographics. • Pretest and revise the instrument.

21. Writing Questions • Open-ended questions. • Unstructured (free to answer as they will). • Projective. • Association – React to a particular stimulus. • Construction – Create a story or self-portrait. • Completion – Finish an already started stimulus or picture. • Ordering – Arrange or select items as important or salient. • Expressive – Freely express themselves by drawing a picture or something similar. • Closed-ended questions. • Structured Answer. • Dichotomous. • Multichotomous. • Scale (see next slide).

22. Developing and Using Scales • The types of scales most commonly used in public administration are attitude scales, importance scales, rating scales, and readiness-to-act scales. • The most common are attitude scales. • Types of attitude scales. • Thurstone – 100 or more opinion statements ranked by informed judges. Assigned scale values by median ranking of judges. Time consuming and rarely used. • Likert Scales – Statements ranked on a five-point scale ranging from Strongly Agree, Agree, Neutral or Undecided, Disagree, or Strongly Disagree. Usually summed or average across multiple items. • Semantic differential – Five- or seven-point scales with opposing adjectives.

23. Conducting the Survey • The survey steering committee. • Double check questionnaire with steering committee. • Frequency of surveys. • For most multipurpose surveys, no more than once per year. • Pretest. • Test on twenty people at random. Ask questions about format and clarity.

24. Conducting the Survey • Training. • Survey assistants must be trained. All must operate uniformly, asking the questions in the same way, coding in the same way. • Consistent open-ended coding. • 10% recontact of survey respondents. • Trying hard and keeping track. • Three contacts by telephone for each number. • Warning and at least two mailings for mail surveys.

25. Reporting Results • Data analysis. • For most government surveys, percentages, average responses, simple cross-classifications. • The most complicated analysis will be to get accurate population estimates – weighting. • Report writing and presentation. • Executive summary. • Bulleted lists. • Document survey methods in appendix. • Augment tables with bar and pie charts. • Powerpoint for in person presentation.

26. Hiring a Consultant • Previous experience. • Ability to communicate findings. • Share work with in-house staff. • Intuition.