Surveys and Questionnaires

Surveys and Questionnaires Research Methods & Data College of Advancing Studies Brendan Rapple

Surveys Provide Important Knowledge • Economists, psychologists, health professionals, political scientists, and sociologists conduct surveys to study such topics as: • Income and expenditure patterns among households; • Voting behavior; • Effects on family life of women working outside the home, etc. • Auto manufacturers use surveys to find out how satisfied people are with their cars. • And a multitude of other topics This presentation owes much to the American Statistical Association brochure series on survey research: http://www.amstat.org/sections/srms/whatsurvey.html

Specific Purpose Essential • Objectives of a survey should be as • Specific • Clear-cut • Unambiguous as possible

"Men's Health Practices" is a very nebulous topic. • Better: How often do African-American males aged 40-49 visit the dentist? Or A survey of 50-60 year old male professors at BC about their weekly exercise habits

Steps in Conducting A Survey • Define precise purpose • Specify population • Specify appropriate sample • How to administer survey? • Draft of survey instrument • Pretest it • Revise it • Administer survey • Analyze, write it up, and communicate the results • Use results meaningfully

Decide on Mode of Data Collection • Mailed Questionnaire • Telephone • In Person Interview • Computer Questionnaire

Advantages of In-Person Interview • Interviewer can clear up any ambiguity in the questions. • The interviewer can gather more complex and “sophisticated” answers/input. • The interviewee does not have to be literate. Of course, in-person interviews can be very costly as well as time-consuming.

Use Mailed/Computer Questionnaires or Not? • Perhaps “better” responses by phone -- but phone interviewing is very time-consuming. • On other hand, people tend to be more truthful with anonymous questionnaires. • Generally cheaper than one-on-one interviews. • Perhaps a mix of questionnaire and interviewing?

For All Surveys it is Essential to • explain purpose of survey very clearly and precisely. • explain any potential use the results will have for the respondents themselves. • stress voluntary nature - respondents are doing a favor to the researcher. • explicitly promise confidentiality. • mention a clear expression of thanks.

Whatever Format Used . . . . . . • Important to specify a deadline for response. • Follow-up letter (e-mail) often advisable for a mailed/computer questionnaire • Always essential for PILOT STUDY or PRE-TEST Main problems revolve about: • Question content, e.g. confusion with overall meaning of question as well as misinterpretation of individual terms or concepts

Population → Sample • Usually the population to be surveyed is too large. • Accordingly, one must select a smaller, representative sample. • “This sample is usually just a fraction of the population being studied.”

Samples • The quality of the sample – whether it is up-to-date and complete – is probably the dominant feature for ensuring adequate coverage of the desired population to be surveyed. • Must be representative of population. • Are the distributions of attributes, opinions, and beliefs in the sample the same as in the population? • You want to be able to make inferences about the population as a whole based on what you find to be true of the sample.

Variability • Variability is large, then sample often should be large • Converse also true

2 Barrels of Apples • Barrel A (low variability) -- all apples about 5 ins. in diameter (range 5.1 to 4.8 ins.) • Barrel B (high variability) -- apples range from 2 to 6 ins. in diameter • Picking 3 apples from Barrel B might give result well below (above) average.

Still, Size of Sample Isn't Everything • Large numbers do not, in and of themselves, increase the representativeness of a sample. • Most professional survey conductors hold that a moderate sample size is enough statistically and operationally.

Representative Sample • Survey:Success of unwed teenage mothers in a specific community in raising children? • To be representative, sample must contain sameproportion of unwed teenage mothers at --each age level --each educational level --each socio-economic statusin the community

Population -- Sample • It is essential that you select sample in such a way that every name on the population list has an equal chance of being included in the sample.

Random Sample Example: 500 part-time students in Advancing Studies • Sample of 20% is required • Assign each student a number from 1 to 500 • Randomly select 100 numbers

Systematic Random Sampling Example 1. • 2,000 in population and you want a sample of 200, then you might select every 10th name Example 2. • 500 part-time students in Advancing Studies • Sample of 20% is required --Randomly Select a Number from 1 to 5 --Select Every 5th Person --002, 007, 012. 017, 022, and up to 497.

Possible Problem • Staff in govt. agency may be listed unit by unit • Each unit has 9 line-level workers and 1 supervisor. • The supervisor is the 10th person on the list. • It’s a survey of 20% -- every 5th person is selected. • If first no. selected is 1, 2, 3, or 4 then no supervisor will be selected, though they comprise 10% of population. • If first number selected is 5, then supervisors will be greatly overrepresented.Thus, possibility of bias due to periodicity or patterns.

Stratified Sampling Population: 2,000 (800 females; 1,200 males)Sample required: 200If gender is an important variable in your survey, then both females and males should be included in appropriate numbers, that is, in proportions that correspond to their presence in the population.

Strategy: Treat both sexes as separate populations and take 10% sample from each. ORMake sure that all females are listed first and then take every tenth name.Either way you will end up with 80 females and 120 males

Convenience Sampling • Could ruin an otherwise well-conceived survey. • It’s simple and cheap to select a sample of names from a phone-directory to find out which candidate people intend to vote for. • However, this sampling procedure could give incorrect results since persons without telephones or with unlisted numbers would have no chance to be reflected in the sample. • Also, more and more people have got rid of their landlines. • Their voting preferences might be quite different from persons who have listed telephones.

Confidentiality • Confidentiality of data supplied by respondents is of prime concern to all reputable survey organizations. • Important that individual respondents are not identified in reporting survey findings. • All of the survey’s results should be presented in totally anonymous summaries, such as statistical tables and charts.

Problems with Volunteers Example: TV programs asking viewers to vote. • people call who are most committed to issue. • “stuffing of ballots” by multiple calls. • Time of day is important – who’s available?

Margin of Error • Error margin of 1,000 randomly chosen individuals is generally said to be about 3.1%. • Thus, if a random sample of 1,000 indicates that 59% will vote for Obama, the actual number could range from 55.9% to 62.1%.

Questions in a Questionnaire Should Be • Woven together • Flow smoothly • Avoid confusion A good questionnaire forms an integrated whole.

Different Understandings • Everyone should see/understand the exact same question – no ambiguity. • But people from different backgrounds, with diverse frames of reference, may have different perceptions of the same question.

KISS PrincipleKeep It Simple, Statistician • Questions should be, as far as possible: • Simple • Clear, • Easy to answer • Personally relevant to them • Often recommended that questionnaires be written at the 5th grade reading level

Do You Need Permission for a Survey? • Administering a survey may be a sensitive issue. • Perhaps you require permission from the organization where you work? • Perhaps you require approval from your college’s/university/s Institutional Review Board (IRB)?

Remember these Four Points: It’s essential • to ask the correct question • that the respondents understand the question • that the question is one that the respondents know • that the respondents are both willing and capable of answering the question

Some Examples of Questions

Avoid Ambiguity • Do you favor governmental involvement in health care? • “What is your income?” __________________ do you mean: weekly monthly annual pretax after tax from salary or from all sources

Don’t be Vague Bad Better Check off which major kitchen appliances you purchased for your home during the past 12 months: -- Washing machine -- Dishwasher -- Refrigerator -- Kitchen oven • What major kitchen appliances did you purchase for your home during the past 12 months? List them.

Another Example: (A blouse manufacturer wants to ascertain what type of sleeves teenage females prefer in their blouses)1. Do you like short sleeve blouses? YES__ NO__2. Do you like long sleeve blouses? YES__ NO__3. Do you like sleeveless blouses? YES__ NO__

Another Example • “Do you jog regularly?”Problem, of course, lies with REGULARLY

Avoid jargon, slang, abbreviations • Plumbers talk about “snakes” • Psychologists about “oedipus complex” • Lawyers about “mens rea”

“NATO” usually means North Atlantic Treaty Organization But some respondents might take it to mean:--National Auto Tourist Organization --Native Alaskan Trade Orbit --North African Tea Office

Types of Questions • Open-Ended Questions: • What is your age? • What is the total turnover in your company? • Which of the four seasons do you prefer? • How would you spend a a $1,000,000 lottery win?

Disadvantages of Open-ended Questions: • Variation in answers make coding/scoring difficult

Advantages of Open-ended Questions • Do not impose researcher's opinion on respondent • Can lead to a very precise answer

Forced-Choice Questions • How many books do you read each year? Please check as appropriate below: ___ none ___ 1 to 5 ___ 6 to 10 ___ 11 to 20 ___ more than 20

What do you recall about the frequency of snowfall in Boston during the winter of 2009? ___ It snowed almost every day ___ It snowed about once a week ___ It snowed about once every two weeks ___ It snowed about once a month

Importance of Wording • Take a very simple question: How many drinks do you have each day? (Check one of the following) ____ 5 or more ____ 4 ____ 3 ____ 2 ____ 1 ____ none

Better to Ask • Are your daily drinking habits reasonably consistent --i.e. do you take about the same number of alcoholic drinks each day? _____ YES _____ NO (if you mark "NO," skip the following question).

Checking Respondents' Consistency Q. 4. Check one of the following: _x_ I believe that manufacturing should be increasingly computerized, even if layoffs ensue. ___ Preserving the jobs of workers is more important than computerizing manufacturing.Later on, the Questionnaire might ask: Q. 30. Check one of the following:___ I support G.M.'s increasing use of robotics, despite the resulting massive layoffs. _x_ The government should force G.M. to curtail computerization to ensure a reduction in layoffs.

Avoid Emotional Language • “What do you think about a policy to pay murderous terrorists who threaten to steal the freedoms of peace-loving people?”Problematic Words:murderous freedoms steal peace

Avoid Loaded Questions: • "Should the mayor spend even more tax money trying to keep the streets in top shape?" • "Should the mayor fix the pot-holed and dangerous streets in our city?"

Arrangement Is Important What is your present marital status? 1 never married 2 married 3 divorced 4 separated 5 widowed

A Better Arrangement Would Be: • What is your present marital status? (circle number) 1. never married 2. married 3. divorced 4. separated 5. widowed

Surveys and Questionnaires