Questionnaire & Interview Item-writing Part 1

Questionnaire & Interview Item-writing Part 1 Dr Desmond Thomas, MA TESOL University of Essex

Useful References • Denscombe, M., 1998, The Good Research Guide for Small-scale Social Research Projects, Open U. Press • Foddy, W. 1993, Constructing Questions for Interviews and Questionnaires - Theory and Practice in Social Research, Cambridge: Cambridge University Press • Oppenheim, A.N., 1992, Questionnaire Design, Interviewing and Attitude Measurement, London: Pinter

Questionnaire or interview? What’s the difference? • Not a great deal, when questionnaire items are read out and responses filled in by the researcher • But a world of difference between an anonymous structured survey questionnaire and an unstructured 1-to-1 interview • QUESTIONNAIRE OR INTERVIEW: WHICH SUITS YOUR RESEARCH? • And what kind of interview?

A research questionnaire should … • Be designed to collect information which can be used as data for analysis: this can be quantitative, qualitative or a mix • Gather information by asking people about issues directly related to a research project (demonstrating validity) • Consist of a written list of questions with each respondent reading an identical set and following identical procedures (reliability)

Questionnaires are useful when … • Used with large numbers of respondents in multiple locations • Information required is straightforward • Standardized data from identical questions is required

Questionnaire issues: Sampling • Is a sample representative of a total population really necessary? • Or can the sample in a qualitative survey represent itself? • If a representative sample is needed are there ready-made sampling frames? • Will sampling be random or in clusters? • How important is sampling size?

Quantitative Survey Issues: Sampling 1 • If the target population is : secondary school teachers of English in Bulgaria • How large a sample? 30 or more people? 5-10%? • Problem of self-selection in survey responses. How to solve this?

Quantitative Survey Issues: Sampling 2 • Sample A: teachers belonging to five different schools (cluster sampling)? • Sample B: randomly selected? • Sample C: randomly selected within large clusters (eg the five schools)? • Sample D: Every 10th teacher on a national register (or other sampling frame)?

Questionnaire issues: Piloting • What should be piloted? Everything! • Questionnaire layout, length, question types, question wording, order of questions, rubrics all need to be tested • It is impossible to get things right at a first attempt; second attempts, in turn, will need to be piloted

Advantages of questionnaires • Supply a large quantity of data for a relatively low cost: not labour-intensive • Standardized pre-coded answers can enable speedy data collection, management and even analysis • Eliminate ‘interviewer bias’ – at least to a certain extent • Face validity

Questionnaire problem areas • Low response rates • Frustration for the respondent: box-ticking can deter respondents when no answer seems appropriate • Frustration for the researcher: no scope for clarification of answers • One chance only to ‘get it right’ – especially in terms of item wording

Creating web-based questionnaires • Go to: http://www.surveymonkey.com/ • Design and edit your survey • Send the link by email to potential respondents • Collect and analyse the data • Note: limitations of the free vs the paid version of this software

Issues for all types of questionnaire • What makes a good questionnaire item? • Why is it so difficult to formulate the precise questions that we need to ask in order to obtain the required answers? • Why is it sometimes difficult to interpret the answers that we obtain?

Fundamental Problem 1 • “It is almost impossible to ask a question without suggesting answers. The very fact that a question is asked implies that the researcher thinks the topic is of interest. Moreover, the way a question is asked inevitably reflects the researcher’s preconceptions. Unstated presuppositions always underlie a question. “ (Foddy 1993: 53-4)

Fundamental Problem 2 Q: Which soft drink do you usually buy? (What is a ‘soft drink’ and what isn’t? How often is ‘usually’? What is understood by ‘buy’ and by ‘you’? What if you buy more than one?) A: Probably the first brand that comes to mind The question may not be understood as it is intended and results may therefore be invalid. (Starting point: what information does it seek to elicit?)

Question questions such as … • How many journeys have you made on London Underground over the past month? • Do you enjoy going to coffee bars and restaurants in the evening? • Do you regularly do your shopping in large supermarkets? • The library facilities at Essex University are adequate for my needs. Strongly agree ( ) Agree ( ) Disagree ( ) Strongly disagree ( )

And these as well … • Complete the following sentence. I enjoy studying at Essex because … 2. Do you think that the library should do more to improve its facilities for Masters students? 3. How important is exercise as part of your daily routine? Very important ( ) Important ( ) Not sure ( ) Unimportant ( )

Framework for questionnaire design

Design principles • Choose the right instrument! • Target information required first → afterwards the right questions to ask. Required information depends on aims. • Rigorously monitor all items • Pilot and re-pilot the questionnaire • Provide clear instructions and standard procedures (for reliability)

Open vs closed questions OPEN: Answers have more depth but data processing is more complex. CLOSED: Easy to process but many useful insights are lost and respondents can become irritated by being put ‘in boxes’. (Oppenheim 1992, p.115)

Some pitfalls to avoid • Respondents often answer questions when they don’t really know the answer. • Respondents can deliberately not answer or answer incorrectly (ie they lie) • Respondents misinterpret questions • Small changes in wording can produce major changes in distribution of responses. • Attitudes and opinions can be unstable; circumstances can change • Memory can be unreliable. • Relationship between what respondents say they do and what they actually do is not strong. • Cultural context has an impact on responses (eg attitude scale grades) • The format itself can affect responses • The order of questions and answers to earlier questions can affect responses.

NEXT WEEK !!! Questionnaire Item-Writing Workshop: Coursebook Feedback Survey

Types of 1-to-1 interviews • Structured: with tight control over question format and possible answers. Like a face-to-face questionnaire • Semi-structured: with a clearly-defined question schedule but some flexibility and more open-ended answers • Unstructured: a narrative prompted by one general question, perhaps

1-to-1 Interviews are useful when … • Detailed information is needed from respondents • A smaller number of respondents is acceptable • Attitudes or feelings are investigated • Sensitive issues are explored (?) • ‘Key players’ are targeted • Ideas for a questionnaire need to be explored or ‘fine-tuned’

Advantages of 1-to-1 interviews • Depth of information • Insights eg gained from key informants • Flexibility of formats • Validity: direct contact means that data can be checked for accuracy and for relevance • High response rate • Opportunity for targeted individuals to make their voice heard – element of advocacy

Interview problem areas • The ‘interviewer effect’ and power relationships: face-to-face contact can directly influence answers • The ‘instant position’ effect: respondents feel obliged to supply answers of some sort • Investment of time and resources • Complexity of data analysis • Reliability issues for multiple interviews

Reliability issues • How can we know if respondents consistently understand the question? Or if they have misinterpreted it? • How can we know if respondents are consistently telling the truth? Or if they are misleading the interviewer deliberately or at a subconscious level?

Features of interview design • How many questions? • In what order? • How long/short? • Format? (open qqs, statements etc.) • Follow-up probes and/or prompts? • Wording of questions? • Mode of recording/analysing answers?

Conducting an interview • Establishing a relationship/trust • Explaining aims & procedures • Low key presentation of self and involvement • Active listening • Clarifications, probes and prompts • Recording, transcribing and analysing

Focus groups • Consist of a small group of people who are brought together by a trained ‘moderator’ (usually the researcher) to explore attitudes, feelings and ideas about a particular topic or set of issues

Main characteristics of FGs • A question schedule is prepared by the moderator to focus the discussion • Place value on interaction within the group as a means of eliciting information – a social experience • No requirement to reach consensus • Less pressure to hold a fixed point of view – in fact opinions can develop

FGs are useful when … • Topics or issues need to be explored in depth, or attitudes revealed • Typical groups of key informants are to be targeted eg language teachers

Advantages of FGs • A less directive interview format • ‘Interviewer bias’ can be greatly reduced • Provide a more natural social forum for the exchange of ideas. Interviewees are able to take the initiative – not just respond • The right to silence

FG problem areas • Less directive means less predictable • Huge amounts of data can be collected, much of which may be unusable • Perception that FGs are a market research tool (Tony Blair & New Labour)

Questionnaire & Interview Item-writing Part 1