NEW PROCEDURES FOR CODING OPEN-ENDED SURVEY DATA

1. NEW PROCEDURES FOR CODING OPEN-ENDED SURVEY DATA Matthew K BERENT University of Michigan Jon A KROSNICK Stanford University Arthur LUPIA University of Michigan

2. In this webinar, you will learn about… • How and why open-ended questions are used in surveys

3. In this webinar, you will learn about… • How and why open-ended questions are used in surveys • How we recognized problems with conventional practice

4. In this webinar, you will learn about… • How and why open-ended questions are used in surveys • How we recognized problems with conventional practice • Newly developed best practices

5. In this webinar, you will learn about… • How and why open-ended questions are used in surveys • How we recognized problems with conventional practice • Newly developed best practices • An example of implementing these practices

6. In this webinar, you will learn about… • How and why open-ended questions are used in surveys • How we recognized problems with conventional practice • Newly developed best practices • An example of implementing these practices • Why being transparent about your procedure enhances your survey’s public value

7. Question Types • Numbers.

8. Question Types • Numbers. • Categorical questions with unlimited universes.

9. Question Types • Numbers. • Categorical questions with unlimited universes. • Knowledge quiz questions.

10. Question Types • Numbers. • Categorical questions with unlimited universes. • Knowledge quiz questions. • Much literature: These types of measurements are done more accurately with open-ended questions than with closed-ended questions.

11. Question Types • Numbers. • Categorical questions with unlimited universes. • Knowledge quiz questions. • Much literature: These types of measurements are done more accurately with open-ended questions than with closed-ended questions. • Categories 2 and 3 require coding.

12. ANES: Open-ENDED questions • Is there anything in particular about JOHN MCCAIN that might make you want to vote for him?

13. ANES: Open-ENDED questions • Is there anything in particular about JOHN MCCAIN that might make you want to vote for him? • Why do you think Barack Obama won the Presidential election?

14. ANES: Open-ENDED questions • Is there anything in particular about JOHN MCCAIN that might make you want to vote for him? • Why do you think Barack Obama won the Presidential election? • What do you think is the most important political problem facing the United States today?

15. Typical practice • The interviewer transcribes responses

16. Typical practice • The interviewer transcribes responses • The coder later:

17. Typical practice • The interviewer transcribes responses • The coder later: • Is given a list of coding categories but little or no instructions.

18. Typical practice • The interviewer transcribes responses • The coder later: • Is given a list of coding categories but little or no instructions. • Reads transcribed answers.

19. Typical practice • The interviewer transcribes responses • The coder later: • Is given a list of coding categories but little or no instructions. • Reads transcribed answers. • Assignseach answer to a category.

20. Typical result Most Important Problem – CBS/NYT June, 2013

21. problems with ANES coding

22. Luskin (2002) “The verdict is stunningly, depressingly clear:

23. Luskin (2002) “The verdict is stunningly, depressingly clear: most Americans know very little about politics…”

24. Conclusion based on open questions • ANES: • “What job or political office does William Rehnquist hold?”

25. Conclusion based on open questions • ANES: • “What job or political office does William Rehnquist hold?” • Correct answers: 12% of the respondents.

26. Public polls

27. Implications for politics • Elected representatives should ignore public opinion?

28. Implications for politics • Elected representatives should ignore public opinion? • Why bother spending money on surveys measuring the public’s opinions about policy issues?

29. Should we believe this?

30. Scholars could not check the validity of this result • ANES didn’t release to the public:

31. Scholars could not check the validity of this result • ANES didn’t release to the public: • the coding instructions

32. Scholars could not check the validity of this result • ANES didn’t release to the public: • the coding instructions or • the answer transcriptions.

33. ANES instructions to The coder • “We are strict regarding acceptable answers. • We will accept ONLY ‘Chief Justice.’ • ‘Justice’ alone is definitely not acceptable. • The court must be ‘the Supreme Court’. • “Chief Justice of the Court” won’t do.

34. In fact … • In addition to the 12% who said “Chief Justice” and “Supreme Court”, …

35. In fact … • In addition to the 12% who said “Chief Justice” and “Supreme Court”, … • 30% identified Rehnquist as a Supreme Court justice. • 26% of respondents: • said that Rehnquist was a judge or • said that he was on the Supreme Court

36. The impact of instructions No wonder we characterize most Americans as knowing little or nothing about politics.

37. One lesson learned already Because ANES was not transparent about its coding instructions, those surveys may have misled scholars about the extent of Americans’ knowledge about politics.

38. Another indication that standard coding may not be optimal Differences Between Organizations

39. Pew Research Center - Most Important Problem 01 Economy (unsp) 02 Unemployment/lack of jobs/ Job cuts 03 Risky bank loans/sub prime loans/people taking on too much debt 04 Inflation/difference between wages/costs 05 Drop in retirement accounts (401K)/stock market 06 National debt/budget/deficit/balanced budget 07 The bailout 8 The stimulus 10 Taxes (unsp) 11 Financial crisis/credit crunch/banking situation 12 Housing market/foreclosures 13 Wallstreet/corporate America 14 Education/schools/affording education 15 A/O economic issues mentions 16 Finances 18 Crime/Violence/gangs/justice system (unsp) 20 Drugs/alcohol 25 Abortion 26 Social Security 27 Issues relating to the elderly 28 Welfare abuse 29 Race relations/Racism/Racial profiling 31 Morality/religion/family values (unsp) 33 Homelessness 34 Poverty/Hunger/Starvation 37 Healthcare (costs/accessibility) 43 Dissatisfaction with government/politics/Scandal/corruption in govt 46 Partisanship/the parties 47 Immigration/immigration situation/foreigners 50 Environment/pollution/Global warming 52 A/O social/domestic issues mentions 56 Too much foreign aid/spend money at home 57 Defense issues/national and homeland security/military & defense spending 58 Trade/Jobs moving overseas 59 US foreign policy/International relations Americas image overseas 60 Iraq / War in Iraq 61 China 62 Terrorism 67 A/O international/foreign issues mentions 70 Recession/Slowing of the economy 76 Energy costs/Rising gas/heating prices 77 Middle East situation 78 Many things/Everything 79 Uneven distribution of wealth 80 Peace in the world/Peace 86 More regulation of markets 89 Oil dependence/Energy policy and alternatives 90 Obama 91 Govt control/overreach - socialism

40. Gallup Organization - Most Important Problem 37 Lack of a Military/Defense 38 Fuel/Oil Prices 39 Dishonesty/Lack of Integrity 40 Judicial System/Courts/Laws 41 National Security 42 Gap Between the Rich and Poor 43 Care for the Elderly 44 Wage Issues 45 Fear of War 46 Breakdown of the Family 47 Cancer/Diseases 48 Child Abuse 49 Advancement of computers/technology 50 Lack of energy Sources 51 Gun control/Guns 52 Costs associated with health insurance 53 Overpopulation 54 Way children are raised 55 Presidential choices/election year 56 Lack of respect for each other 57 Poor leadership/corrupt 58 Abuse of power 59 Lack of money 60 Education 61 The media 62 Election/Election Reform 63 Unifying the Country 64 Situation with China 65 Energy crisis 66 Terrorism 67 Feeling of Fear in this Country 68 Corporate Corruption 69 Space Shuttle Disaster 70 Iraq 71 Gay marriage/Homosexual rights 72 Licenses for the Undocumented 73 Natural disaster response 75 War-Conflict in the Middle East 6 Economy (General) 7 Unemployment-Jobs 8 Federal Budget Deficit-Debt 9 Taxes 10 Foreign Trade-Trade Deficit 11 Cost of Living-Inflation 12 Recession 13 Other Economic 14 Crime-Violence 15 Health Care-Hospitals 16 Drugs 17 Poverty-Hunger-Homelessness 18 Ethical-Moral-Religious Decline 19 Access to Education 20 AIDS 21 Medicare Increases-Senior Citizen Insurance 22 International Problems-Foreign Affairs 23 Government-President Clinton-Congress-Politicians 24 Foreign Aid-Focus Overseas 25 Race Relations 26 Immigration-Illegal Aliens 27 Welfare 28 Environment-Pollution 29 Iraq-Saddam Hussein 30 Kosovo-Serbia-Yugoslavia-Milosevic 31 School Shootings 32 Gun Laws too Weak-Availability of Guns 33 Gun Control-Gun Laws Too Strong 34 Abortion Issues 35 Social Security Issues 36 Children’s Needs

41. Most important problem: economy

42. Most important problem: economy

43. Most important problem: economy

44. Most important problem: economy

45. Another indication that survey practice may not be optimal What Psychologists Do

46. Psychology • Multiple independent coders

47. Psychology • Multiple independent coders • Elaborate written instructions

48. Psychology • Multiple independent coders • Elaborate written instructions • Assess levels of agreement

49. Psychology • Multiple independent coders • Elaborate written instructions • Assess levels of agreement • Negotiate disagreements

50. Anes and every other major survey organization we talked to • One coder