The following lecture has been approved for University Undergraduate Students

1. The following lecture has been approved for University Undergraduate Students This lecture may contain information, ideas, concepts and discursive anecdotes that may be thought provoking and challenging It is not intended for the content or delivery to cause offence Any issues raised in the lecture may require the viewer to engage in further thought, insight, reflection or critical evaluation

2. Survey Research Samples and Populations Response rates Data Types Honesty Delivery Validity Dr. Craig Jackson Senior Lecturer in Health Psychology Faculty of Health UCE Birmingham

3. Brief Research History Fatigue in short-haul budget airline pilots 2002-2004(Web-based Survey) Mental Health of UK Farmers using OP Pesticides (X2) -- 1997-2000(Epidemiological Surveys) Neurobehavioural Performance of desert-based Oil Drillers -- 1998-2000(Clinical assessment) Temporary Hearing Loss in Student Bar Staff – 2000-2002 (Epidemiological Survey) Benefits of Occupational Health Advice in Primary Care Settings -- 2001-2004(Randomised Controlled Trial) Smaller-Scale projects – (Tri-Services, NHS Personnel, NHS Patients)(Cross-sectional Surveys, Clinical Trials) Multiple roles of psychologist, statistician, and methodology designer

4. PatientsStaffHealthy Quantitative Research Designs Laboratory Experimental RCT Approach Case - control Epidemiology Cohort study Observational Survey Postal questionnaire

5. Are Postal Surveys Dead Yet? IT predicted death of postal surveys Use of IT at home and work increases survey methods Comparison of surveys using WWW or EMAIL or POSTAL Subjects – UK university staff 200 email questionnaires 200 emails with www url 100 postal questionnaires Asked 3 questions: teeth cleaning fruit walking Jones, R. and Pitt, N. BMJ1999

6. Are Postal Surveys Dead Yet? Results Days after sending email www post n=200 n=200 n=100 numbers responding 3 25 31 0 5 59 34 16 9 61 35 53 10 63 35 60 17 68 37 72 17 day response rate 34% 19% 72% cost per reply Actual cost 35p 41p 92p With 100% response 19p 7.5p 72p

7. The Postal Survey is Alive and Well . . . . Communication age? Postal surveys still much better than e-surveys Novelty value of email is dead “Junk mail” perceptions Email filters are improving – fewer emails get through Postal letters demonstrate & emphasize their importance www surveys & email allow immediate data processing (software) Email & www have potential for low cost regular user surveys Intranet use of e-surveys seems ok

8. Survey Research Questionnaire is a fundamental component of most surveys Most MSc / MPhil /PhD projects use survey methods Can be very efficient Weaknesses weak / dubious questionnaires non-valid questionnaires biased samples biased responses poor response rate

9. Wide range of applications Psychometrics -- personality, attitudes, stress, symptoms Physical measurements -- working, environment, symptoms Exposure assessments -- hazards, risks, ppm3, duration, chemicals Necessary Requirements Take multiple measurements (and take the mean) Under same conditions, but if not…. Statistical remedies to adjust e.g. age, time of day etc. Reliable & Validated tool Defined and Regular variables e.g. problem with stress research Well defined standard of reference e.g. how far back are you “surveying”?

10. Validity Does the survey measure what it says it is measuring Reliability Does the survey yield stable data over time

11. Quantitative Research Requires . . . . Numerical / Quantifiable data Probability-based Nomothetic (group data not individual data) Sufficiently large sample size (to detect statistically significant effects) Randomised sampling of a population (cannot guarantee a random sample) Statistical analyses of data

12. Population Samples Achieving a high response rate to a questionnaire is vital as helps ensures a normal distribution of responses? Postal questionnaires rarely get a response rate > 40% Unless respondents have a vested interest in the outcome Bias? Most efficient (best) response rates usually happen when respondents have to do very little to take part in the study Multiple phase projects see a depletion in numbers at every stage Quick “in and out” one-stop approach is best

13. RANDOM sampling OPPORTUNISTIC sampling CONSCRIPTIVE sampling QUOTA sampling A “Normal” Sample N of population 5’6” 5’7” 5’8” 5’9” 5’10” 5’11” 6’ 6’1” 6’2” 6’3” 6’4” Height

14. Being Practical – how many is a sample? Student Pop N indepth I.D Forces yachting training schools 300 E.M Companies using stress counselling 150 S.M Divers and ear barotrauma 142 N.O Solvent exposure in Myanmar 80 V.W Routine flu vaccinations 900 A.F Dermatitis in hairdressers 102 S.M O.H needs of NHS staff 23 yes T.R NIHL in student employees 14 yes I.C Blood tests in British Army pilots 408 O.Y Upstream oil company deaths 161 A.A Renal colic in flight deck crew 254 A.C Hepatitis B in army regulars and territorials 476

15. The Importance of Sample Size • Forgotten in many studies • Appropriate size needed to confirm / refute hypotheses • Small samples can’t detect anything but the grossest difference • Too large a sample – unnecessary waste of resources • Ethical considerations – waste of patient time, inconvenience, discomfort • Essential to make assessment of optimal sample size before starting investigation

17. Non-Responders just as Important Postal surveys may accrue poor response rates (e.g. 20%) from pop. May need to re-write to pop. to re-recruit bigger sample Inefficient to write to all pop. again Need to re-write to non-responders and NOT responders Impossible in anonymous studies with no linkage Can be done with confidential studies

18. Diminishing returns of multi-stage recruitment Researcher Potential Sample 1000 people 540 consents 540 questionnaires Under-powered studyn = 210 Response rate of 21% 210 questionnaires

19. Encryption devices Steganography Secret communication of a message by hiding it’s existence Steganos, meaning covered. Gk Graphein, meaning to write, Gk If message is discovered it is easily read because of no encryption Cryptography Secret communication of a message by hiding it’s meaning Kryptos, meaning hidden. Gk Message established using a known protocol, to be decrypted by the receiver Steganography & Cryptography can be combined together if needed Steganogrpahy arouses less suspicion in questionnaire respondents

20. Encryption devices Steganography “This example of steganogrpahy may not work very well when projected onto a large screen, but it works very well on paper, such as questionnaires. Hopefully many of you will not notice the method of steganography used in this piece of text. Gosh how clever I am……” Cryptography drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui The above text (containing an encrypted name and address) looks suspicious and may be obliterated by the respondent drbjh kbdltpo, jotujuvuf pg pddvqbujpobm ifbmui

21. Selection Bias Sampling properly is Crucial Samples may be askew Specialist publications attract a specialist response group Exists a self-selection bias of those with special interests Controversial topics, or litigious areas Bird Flu Call Centres Gulf War Syndrome THIS IS AN INHERENT PROBLEM WITH HEALTH RESEARCH COMBAT IT WITH LARGE SAMPLES AND CLEVER METHODOLOGY Depleted Uranium Weaponry Organophosphate Pesticides Stress Telecomms

22. Structure of Surveys Identifying Items Title Preamble Instructions Research Items does not need to be honest some deception is necessary shorter surveys better than long surveys

23. Identifying items & Demographics Preliminary questions Collecting info necessary for screening: recording keeping tracking tracing data manipulation Ask only for relevant info – unethical to “harvest data” Fewer items minimizes chance of alienating respondent Subject’s need for privacy & anonymity The need for rich info to improve the study

24. Preamble An important introduction Frame of reference Without a FOR a respondent may base their answer on a wrong context “We would like to know if you have had any medical complaints and how your health has been in general, over the past few weeks. Please answer ALL the questions on the following pages by ticking the answer which best applies to you. Remember that we want to know about present and recent complaints, not those that you had in the past. It is important that you try to answer all questions”

25. Title Not to be underestimated – try to be user-friendly Sets the tone for the respondent The General Health Questionnaire vs The GHQ 28 Beware of abbreviated titles - may alienate respondents e.g. ASS - Agricultural Satisfaction Scale LEI – Life Events Inventory BDI – Beck Depression Inventory MMPI – Minnesota Multiphasic Personality Inventory HAD – Hospital Anxiety Depression scale “Simple language for simple people”

27. Respondent “Outrage” increases with item intimacy Are Respondents Telling the Truth? Use a sprinkling of LIE DETECTOR items to assess reliability e.g “Have you ever taken anything without asking permission?” “Did you ever lie to your parents as a child?” “Have you ever visited a pornographic web-site?” Decide what to do with any respondent who “fails” the lie detector Keep them in? Exclude immediately? Retain but “flag” them?

28. Response-Set Bias Enables collection of detailed data easily Build a detailed data set from a simple binary item Randomise the layout / order of responses “true / false” e.g I eat ice cream True False I like ice cream True False I enjoy ice cream True False I love ice cream True False I hate ice cream True False Respondents ticking the same items Swap the “True & False” responses sequence

29. happy sad X option 1 (measure from start) happy sad option 2 (measure from median) Likert scales A visual linear scale for rating purposes Swap around the words to avoid response-set bias Avoid using numbers on the scale can be (mis)leading

30. X X X X X X X X X X X X X X X X X X X X X happy sad -.75 2 1 0 -1 -2 Likert scales

31. happy sad Likert scales Bend your data and avoid fence-sitters X X X X X X X X X X X X

32. Design Pointers for Surveys Avoid colloquialisms or abbreviations Beware of local expressions Avoid words with double meanings e.g.”Fair....Dip....Lie....Well” Set a definition of specific terms e.g. “OK.....Average” Avoid long questions Specify exact time, place and context e.g “At school, did you ever...” Avoid impossible denial e.g “When did you first...” Avoid numbers on your responses / scales Have items seemingly related to the research topic

33. Design Pointers for Surveys Clearly phrase items Make items unambiguous Avoid leading items e.g. “Many people think...” Ask only what a respondent is qualified to answer Avoid socially loaded items e.g. “beggars” or “junkies” Do not use socially / religiously biased items Avoid phrases, clichés or sooth-sayings Ask for ages as a true number / D.O.B - - NOT in age groups Easy Fonts Pictures (not smiley faces!)

34. Some Common Errors in Surveys “In your organization, do women have the same responsibilities as men, and should women have more?” “Preventing accidents in the workplace is vital, and more money should be spent on prevention.” “Training in risk assessment is something I would like to do, and I would like to see my colleagues do it to.” Only ask one question per item

35. Are life preservers and flares essential on-deck equipment ? agree disagree Some Common Errors in Surveys What is your age ? (please tick) 16 - 26 26 - 36 36 - 46 46 - 56 What is your marital status ? (please tick) Married Single How many blood splashes have you had ? (please tick) 1 - 5 6 - 10 11 - 15 15 - 20 20 plus

36. Factorial approach The General Health Questionnaire (GHQ 28) A self-completion questionnaire assessing mental health How? 28 items 7 about Anxiety Anxiety score 3 7 about Severe depression Depression score 4 7 about Dysfunction Dysfunction score 5 7 about Somatic symptoms Somatic score 2 Global score 14 By summing the 4 factors there is a Global Mental Health score Statistics are performed on the factor scores and the global score NOT on each individual item

37. Increasing Response Rates Incentives Appearance Delivery Origin Contact Content Communication Edwards et al. 2002

38. Incentives Money Vouchers Prize draw Ethical aspects Bias O.R Monetary incentive vs. None 2.02 Incentive with Q. vs. Incentive on return 1.71 Non-monetary incentive vs. No incentive 1.19 Edwards et al. 2002

39. Appearance O.R Shorter format vs. Longer format 1.86 Brown envelope vs. White 1.52 Coloured ink vs. Black 1.39 Folder / Booklet vs. Stapled pages 1.17 Personalised vs. Not personalised 1.16 ID feature on return vs. No ID 1.08 Coloured Q vs. White Q 1.06 Edwards et al. 2002

40. Delivery methods O.R Recorded delivery vs. Standard 2.21 Stamped return envelope vs. Business reply / franked 1.26 Q sent to work vs. Q sent to home 1.16 1st class outward mail vs. Other class 1.12 Pre-paid return envelope vs. Not pre-paid 1.09 Stamped outward envelope vs. Franked 0.95 Commemorative stamp vs. Ordinary stamp 0.92 Edwards et al. 2002

41. Origin & Contact Origin O.R University vs. Other organisation 1.31 Sent by senior persons vs. Juniors 1.13 Ethnically ambiguous name vs. Non-white name 1.11 Contact O.R Pre-contact vs. No contact 1.54 Follow up vs. No follow up 1.44 Postal follow up with Q vs. Without Q 1.41 Mentioning follow up vs. None 1.04 Pre-contact by telephone vs. Postal pre-contact 0.90 Edwards et al. 2002

42. Content O.R More interesting vs. Less interesting 2.44 User-friendly vs. Standard 1.46 Factual items only vs. Factual & attitude items 1.34 Relevant items first vs. Other items 1.23 Demographic items first vs. Other items 1.04 “Don’t know” boxes vs. no “Don’t know” boxes 1.03 Sensitive items vs. No sensitive items 0.92 General items first vs. Last 0.80 Edwards et al. 2002

43. Communication O.R Explain drop-out required vs. Not 1.32 Stresses benefit to respondent vs. Others 1.06 Stresses benefit to sponsors vs. Others 1.01 Stresses benefit to society vs. Others 1.00 Response deadline given vs. No deadline 1.00 Instructions given vs. No instructions 0.89 Choice to opt out given vs. No opt out 0.76 Edwards et al. 2002

44. Human Nature Then . . . . . Cash incentives work Want cash upfront not contingent Like to feel important Little time to spare Not motivated by benefit to any other people Lazy

45. Survey Summary • Surveys widely used in data collection • Mark pre-pay or addressed envelopes • Steganography better than cryptography • Perform analyses on factors not individual items • Identifying items / demographics usually useful in later analyses • Think about scoring before rushing ahead with survey • Store questionnaires securely until passing any viva voce / publication • Collect as much info at source, parse it down later at discretion

46. Survey Summary • Perfect for epidemiological studies and health research • Non-response to postal questionnaires reduces sample & introduces bias • Identification of effective ways to increase postal response rates • Use existing metrics - pilot items if making new questionnaire • Use factor analysis • Keep it all as brief as possible • Don’t alienate respondents • Alternate types of items

47. Final Points – Bias does not ruin everything Bias Avoiding bias is a good aim to have Existence of some bias in a sample does not ruin a project entirely Mostly leads to underestimation rather than overestimation of main effects Spector et al, (2000) shows the “inflating effect” of self-report bias may not be so prominent Spector PE, Chen PY, O’Connell BJ. A longitudinal study of relations between job stressors and job strains while controlling for prior negative affectivity and strains. Journal of Applied Psychology 2000; 85: 211-218.

48. Final Points – Generalizability is not everything Basic principle: Internal validity is always more important than its generalizability Never appropriate to generalise an invalid finding Mant et al. (1996) Mant J, Dawes M, Graham-Jones S. Internal validity of trials is more important than generalizability. British Medical Journal 1996; 312: 779.