Questionnaire Design, Pretesting and Quality Control in Clinical Research

1. Questionnaire Design, Pretesting and Quality Control in Clinical Research Celia P. Kaplan Division of General Internal Medicine August 27th 2008

2. Good Clinical Practice (GCP) “A standard for the design, conduct, performance, monitoring, auditing, recording, analysis, and reporting of clinical trials that provides assurances that the data and reported results are credible and accurate, and that the rights, integrity, and confidentiality of trial subjects are protected” WHO good clinical practice standards

3. Quality Control

4. Sources of Data Errors • Data acquisition • instrument problems • lost data • falsified data (made up data or changed) • Data recording • Data entry • Data management

5. Quality Control Stages • Pre-implementation/Developmental Stage • Design high-quality questionnaires and forms • Pretest all aspects of the project • Develop an Operations Manual • Design a secure data-management system • Conduct staff training and certification • Implementation • Make periodic study- and data-monitoring reports • Conduct periodic project meetings • Complete interim staff training and performance reviews

6. Quality Control Stages • Pre-implementation/Developmental Stage • Design high-quality questionnaires and forms • Pretest all aspects of the project • Develop an Operations Manual • Design a secure data-management system • Conduct staff training and certification • Implementation • Make periodic study- and data-monitoring reports • Conduct periodic project meetings • Complete interim staff training and performance reviews

7. Questionnaires • Much of the data in clinical and epidemiological research is gathered using questionnaires and interviews

8. Questionnaire Measures • Desired qualities • Reliability – the extent to which the measures give consistent results • Validity – the extent to which the measures reflect the “truth” • Accommodate participants’ age, literacy, cultural, linguistic, and SES characteristics • Yield good variability in answers • Result in low percent of missing data

9. Questionnaire Development: Pitfalls • Double-barreled questions – Avoid “and/or” statements in questions • e.g., “Does your department have a special recruitment policy for ethnic minorities or women?” • Long questions and/or questionnaires • Use of “other” as a response category

10. Questionnaire Development: Order • First question should be a fact • Beginning questions should stimulate interest • Questions should be in logical order and groupings • If ask attitude questions first and then behavior questions, may inflate reports of behavior • Sensitive questions and open-ended questions should be at the end of the survey • e.g., sexual behavior, religion, income

11. Questionnaire Development: Language • Use simple, clear, common, and natural language • Avoid jargon, complex terms, clichés, colloquialisms, e.g., “down in the dumps” • Check number of syllables in words • Check reading level: aim for < 8th grade • Define ambiguous terms, e.g., “family” or “neighborhood” • Keep sentences short

12. Questionnaire Development: Recall Solutions: • Narrow the reference period • Average the response: ask about a typical day or in general (e.g., hours of sleep) • Use landmark events or milestones to anchor time frame: “Since you moved to the US,” or use calendar with major holidays to establish major life events • Use cues to generate associations to jog memory: describe procedure, setting

13. Questionnaire Design 7 Finalize the Questionnaire Draft survey Interview methodology Review existing measures Conduct exploratory work Finalized instrument Pre-test the survey Translation/ back-translation

14. Questionnaire design • Selection of interview methodology • Face-to-face or in-person interview • Telephone • Self-administered, paper • Self-administered, computer/web/e-mail

15. Questionnaire design Interview methodology: Face-to-Face • Advantages • Enables the interviewer to establish rapport with respdenont • Permits more complex questions • Permits the use of visual aids • Longer interviews are possible • Disadvantages • Cost per interview • Quality of answers too personal (sensitive questions)

16. Questionnaire design Interview methodology: Telephone • Advantages • Faster contact of participants • Better to elicit personal/sensitive information • Results are available right away if using computer-assisted interviewing (CAI) • Can do random selection of telephone numbers • Disadvantages • Certain sectors of the population are switching to cell phones only • More expensive than mail surveys • Difficulties in reaching participants during the day

17. Questionnaire design Interview methodology: Self-administered and mail surveys (Paper) • Advantages • One of the least expensive methodologies • Needs addresses of potential participants • Allows participants to answer at their leisure • Less intrusive • Elimination of interviewer bias • Disadvantages • Response rate is low • Difficulties to develop clear skip patterns • It may be answered by a family member • Very difficult to implement in low literacy populations

18. Questionnaire design Example: Self-administered & Mail Surveys

19. Questionnaire design Interview methodology: Self-administered, web-based surveys (Computer) • Advantages • Elimination of data entry and editing costs • Get better answers to sensitive questions • Elimination of interviewer bias • Ensure that skip patterns are accurately followed • Allows participants to answer at their leisure • Disadvantages • Interviewees must have access to a computer • Must possess (or purchase) a list of email addresses • Lower response rate • People may quit in the middle of a questionnaire

20. Questionnaire design Example: Web-based survey

21. Questionnaire design Review Existing Questionnaires & Measures • Reference databases • Medline, Pubmed, Psychinfo, others • Compendia of measures • Books that compile various measures and review their characteristics • Request questionnaires from other investigators

22. Questionnaire design Review Existing Questionnaires & Measures • Is there evidence the instrument works in your target population? • Examine validity and reliability of measures when used in populations similar to your study population • When gold standard does not exist, for construct validity, look at relationship of measure to other measures to see if in hypothesized direction

23. Questionnaire design Conduct Qualitative Exploratory Work • Focus groups or semi-structured interviews • Explores a topic before constructing formal questions • Rationale: • Determine what is important • Discover how respondents think about the topic • Assess better language/words to use • Identify response categories

24. Questionnaire design Create a Draft Survey Instrument • Write initial draft • Process of revision and refinement through cycles of translation/back-translation, reviews and pre-tests (Steps 5 and 6)

25. Questionnaire design Translation/Back-translation Procedures • Both versions should have the same meaning • Translation and back translation • Review by a group of native speakers • Challenging when more than one language involved in the study

26. Quality Control Stages • Pre-implementation/Developmental Stage • Design high-quality questionnaires and forms • Pretest all aspects of the project • Develop an Operations Manual • Design a secure data-management system • Conduct staff training and certification • Implementation • Make periodic study- and data-monitoring reports • Conduct periodic project meetings • Complete interim staff training and performance reviews

27. Quality Control Pre-testing the Questionnaire: Overview • Refines • Instruments and questionnaires • Evaluates • Recruitment methods • Interventions • Data entry and management system • Protocols • Assists with training of personnel

28. Quality Control Pre-testing the Questionnaire • Good to test: • Sensitive questions • Complex or poorly defined topics • Questions with terms respondents may not understand • Translations • Layout and instructions

29. Quality Control Pre-testing the Questionnaire: Field Testing Techniques • Expert reviews • Cognitive interviews • Full pretest

30. Quality Control Pre-testing the Questionnaire: Expert Reviews • Formal and systematic examination of a questionnaire by experts in the field • Can cover question wording, layout, and skip patterns • Fast method prior to formal testing • May not be a complete review

31. Quality Control Pre-testing the Questionnaire: Cognitive Interview: Overview • Diagnostic tool for pretesting survey questions derived from social and cognitive psychology • Interview about an “interview” • Explores the processes by which respondents reach answers • Based on structured questionnaire and protocol

32. Quality Control Pre-testing the Questionnaire: Cognitive Interview: Techniques • Concurrent proving • Ask probes immediately after respondent has given answer to survey item • Advantage: information is fresh in respondent’s mind • Disadvantage: Disrupts interview flow and relationship between questions • Retrospective probing • Ask probes after entire interview or block of questions • Advantage: able to assess standard administration of items • Disadvantage: Participant may not remember thought process

33. Quality Control Pre-testing the Questionnaire: Cognitive Interview: Techniques • Probing questions • Think aloud interviews • Interviewer asks respondent to think aloud as they answer question • Paraphrasing • Respondents rephrase the question in their own words

34. Quality Control Pre-testing the Questionnaire: Cognitive Interview: Key Features • Comprehension of the question • Do participants understand words and phrases as intended by the researchers (meaning) • “How did you arrive at your estimate of risk?” • “What did you understand by the word ‘risk’?” • Retrieve of information • Identify process respondents use to answer questions • Examine strategies used to access memory • Edit responses: Decide what to report • Are items unacceptable? Is answer embarrassing, socially undesirable?

35. Quality Control Pre-Test • Pretest with a sample similar to the population to be studied • Simultaneously pretest recruitment procedures and questionnaire implementation

36. Quality Control Pre-Test: Behavior Coding • Code interaction between interviewer and respondent • Examples • R answers before hearing whole question • R looks puzzled • R asks for clarification • I repeats question

37. Quality Control Pre-Test: Respondent Debriefing • Structured follow-up questions at the end of the interview to assess: • Whether questions were clear • Ease of completing the questionnaires • If two versions are tested, which is preferable • Whether an important question was missed

38. Quality Control Pre-Test: Interviewer Debriefing • Interviewers implement pretest with participants • Interviewers detect problems with questions, response categories, and skip patterns • Interviewers report issues to investigators

39. Quality Control Pre-Test Questionnaire: Summary • Testing takes time and resources • Some procedures take place late in the developmental cycle, limiting the amount of change possible • Danger of making changes that are not further tested • Not testing leads to questionnaires that are potentially unreliable or will give you invalid results

40. Quality Control Stages • Pre-implementation/Developmental Stage • Design high-quality questionnaires and forms • Pretest all aspects of the project • Develop an Operations Manual • Design a secure data-management system • Conduct staff training and certification • Implementation • Make periodic study- and data-monitoring reports • Conduct periodic project meetings • Complete interim staff training and performance reviews

41. Quality Control Operations Manual • Aim is to ensure high quality data • Clearly written, detailed written instructions to obtain uniformity across sites, staff, and procedures • Standardizes procedures for all aspects research project. Defines how to… • Recruit participants • Consent participants • Measure variables • Transport clinical samples • Enter data • Monitor progress • Conduct data analysis • Requires updates, clarifications, and recording of decisions made

42. Quality Control Data Management System • Desirable features • Range and field type checks • Ease of screen set up and use • Double data entry (if possible) • Security features • protection of human subjects’ rights (privacy) • For Web-based systems • Virus protection to monitor and eliminate security threats • Database server behind firewall • All data backed up regularly

43. Quality Control Staff Training • Training and certification of the research team members • Rules to conduct interviews • Rules for recruiting participants, including how to obtain consent • Methods • “Train the trainer” model • Audio-visual techniques • Certification/recertification to maintain skills

44. Quality Control Stages • Pre-implementation/Developmental Stage • Design high-quality questionnaires and forms • Pretest all aspects of the project • Develop an Operations Manual • Design a secure data-management system • Conduct staff training and certification • Implementation • Make periodic study- and data-monitoring reports • Conduct periodic project meetings • Complete interim staff training and performance reviews

45. Quality Control Study Reports • Recruitment reports (e.g., actual vs. expected accrual) • Follow-up reports (e.g., overdue visits) • Timeliness of data collection activities • Data entry error rates

46. Quality Control Data Monitoring Reports • Standard procedures for • Identifying missing, incomplete data • Identifying incorrect skip patterns • Identifying inconsistent and erroneous data • Verifying correction

47. Quality Control Interim Staff Training and Performance Reviews • Provide continuous supervision • Hold frequent meetings of the study team • Review all aspects related to implementation of the study, including • Interviewers activities • Listen to interviews • Compare measures between interviewers • Compare refusal rates between interviewers

48. Good Clinical Practice • Good Instruments • Complete and clear protocol manuals • Extensive pretesting • Ongoing monitoring procedures