1 / 50

Q-BANK: Development of a Tested- Question Database

Q-BANK: Development of a Tested- Question Database. Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics. Cognitive Interviewing at NCHS. A Brief Overview. Questionnaire design problems.

hunter
Download Presentation

Q-BANK: Development of a Tested- Question Database

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Q-BANK: Development of a Tested- Question Database Questionnaire Design Research Lab Office of Research and Methodology National Center for Health Statistics

  2. Cognitive Interviewing at NCHS A Brief Overview

  3. Questionnaire design problems • Some possible threats to accuracy derived from questionnaires: • Questions not understood as intended • Don’t adequately capture respondent experience • Pose an overly challenging response task • Don’t provide a legitimate option for an answer

  4. Questionnaire design problems (2) • Errors can be systematic (bias) or random (noise; high variance) • Problems may not be visible in the actual data, inviting delusions about the quality of self-reports that have been collected

  5. “In the last year, have you been bothered by pain in the abdomen?” • Seems to be straightforward

  6. “In the last year, have you been bothered by pain in the abdomen?” • Seems to be straightforward • But suppose we ask: • What do people think of as their “abdomen”?

  7. What, to you, is your abdomen?

  8. “In the last year, have you been bothered by pain in the abdomen?” • Seems to be straightforward • But suppose we ask: • What do people think of as their “abdomen”? • What sort of threshold do people use for answering “yes”?

  9. “In the last year, have you been bothered by pain in the abdomen?” • Seems to be straightforward • But suppose we ask: • What do people think of as their “abdomen”? • What sort of threshold do people use for answering “yes”? • What period of time are people thinking about while answering?

  10. “In the last year, have you been bothered by pain in the abdomen?” • Possible revisions: • Show shaded picture of abdomen • Drop “bothered” • Use “In the past 12 months” • “In the past 12 months, have you had pain in the abdomen? By abdomen, we mean the shaded area in this picture.” • Clear alternatives address these problems, with no apparent drawbacks

  11. Shaded picture of the abdomen

  12. What is cognitive interviewing? 1) An administration of a draft questionnaire… 2) With additional verbal information collected about the responses… 3) And used to either: • Evaluate the quality of responses • Or help to determine whether the question is generating the sort of information its author intends. • Within these general parameters, there are a number of variants:

  13. Practices of cognitive interviewing • The “purist” version– encourage participants to think out loud while answering questions; interviewer “prompts” participants without much other activity.

  14. Practices of cognitive interviewing (2) • Newer versions rely much more heavily on direct probing. But even so, there can be much variation: • Probes scripted or based on interview/participant particulars (more “active” role for the interviewer) • Probes that are “cognitive” or more narrative in nature • A relatively realistic administration of the questionnaire (with debriefing at the end) or intensive discussion throughout

  15. Some general practices • We generally opt for: • Designing a semi-scripted interviewing protocol. It asks some probes in anticipation of potential problems, but allows improvisation. • Selecting a group of interviewees who would be typical recipients of the questionnaire and collectively cover a range of situations. Get a demographic/educational mix as much as possible. • Considering the interviewers to be investigators and not simply data collectors (methodologists with subject matter knowledge).

  16. What sorts of probes should you ask? Consider this simple model of the response process as a starting point: • Comprehension: Respondent interprets the question • Retrieval: Respondent searches memory for relevant information • Estimation/Judgment: Respondent evaluates and/or estimates response • Response: Respondent provides information in the format requested

  17. Probes in cognitive interviews Traditionally the majority of attention focuses on comprehension • Direct comprehension of terms: What does 'dental sealant' mean to you?" • But we also find that the most efficient way to learn about frame of reference and interpretation is to ask for narrative about the basis for the respondent’s answer • Why did you answer that way? • You said ___; tell me more about that. • Or, ask for paraphrase of the question.

  18. Probes in cognitive interviews (2) To get at other aspects of the response process: • Recall: How did you figure your answer to that? How do you remember that? • Confidence: How certain are you about that? (can be used to identify judgment issues/uncertainty) • Response categories: How did you pick an answer to that? Was it difficult to find a response category? • You can’t go wrong with: • “Tell me more about that?” • “How so?”

  19. The end result • Qualitative analysis follows one or more rounds of interviews • Project reports include rich data on • Participant experiences and how these line up with question wording • Participant interpretations of the questions • Problems participants had answering • Potential ambiguities, trade-offs involved in asking question a certain way • Reports provided to sponsor, including recommendations for changes when appropriate • Then what?

  20. Q-BANK Origination • 2001 QUEST conference • Losing knowledge • Repeat testing

  21. Lack of “place” • Implications: • Personal • Administrative • Survey Research

  22. Impact on Question Evaluation • Inability to compare studies • Hinders methodological conclusions • Prevents insight • Stagnant method • Limited use to survey research

  23. Database Structure • National Center for Health Statistics original developers • Access to past evaluation projects • Methodological research in question design

  24. Q-BANK houses: • Individual questions • Test reports • Test findings • Methods used • Question characteristics

  25. Question Subject Matter • Numerous, open categories • Accessed by drop-down list • Examples: Demographic: Income, Race, Gender, Nationality Health: Insurance, Chronic conditions, Smoking Employment: Status, Occupation, Job change, Salary, Benefits

  26. Question Information Type • Events, Actions and Behaviors • Objective Characteristics • Subjective Characteristics • Speculations • Attitudes/Opinions • Knowledge Tests • Other

  27. Response Category Type • Yes/no • Select one textual • Select one numeric • Mark all that apply • Allocation • Open-ended textual: Delineated & Non-delineated • Open-ended textual: Delineated & Non-delineated • Other

  28. Response Error Indicator • Interviewer difficulty • Visual design problem • Missed instructions • Unknown terms • Ambiguous concepts • Overly complex • Assumption • Double barrel • Questionnaire effects • Recall & estimation error • Inadequate response options • Problematic answer space • None as tested

  29. Value of Q-BANK Survey Research Tool for: • Questionnaire Designers • Subject Analysts • Survey Methodologists

  30. Interagency Effort • Co-Sponsors: • Office of Behavioral and Social Science Research, • Bureau of the Census, • National Science Foundation, • National Cancer Institute, • Bureau of Labor Statistics • NCHS provides internet access

  31. Current Status • On-line to steering committee members • Testing and refinement • Broader access possibly by end of year • Looking for sponsors and partners

  32. Challenges Past • Integrating interagency needs • Additional fields for establishment surveys • Additional fields for self-administered surveys • Reworking existing categories: response error typologies, question type typologies, question topic categories • Establishing procedures • Methodological requirements of evaluation reports

  33. Challenges Current • Coding consistency • Populating the database • Staffing limitations • Funding limitations

  34. Challenges Future • Maintaining a methodological research agenda • Continual eye toward development

  35. Searching in Q-BANK • Agency Name • Survey Title • Response Error • Advanced Searches

  36. Q-BANK Search Fields

  37. Agency Name Search

  38. Agency Name Search

  39. Agency Name Search

  40. Q-BANK Reports

  41. Searching on Survey Title

  42. Searching on Response Error

  43. Advanced Searches

  44. Advanced Searches

  45. Cognitive Testing Final Reports • Don’t get lost in the weeds • Important information about a tested question is lost without referencing the final report • Conclusions about specific questions can be distorted without the final report

  46. Contents of a Typical Report • Introduction • Method • Procedures • Summary of findings • Question-by-question analysis

  47. Why refer to the final report? • Search categories may not be straightforward • Example: “At this time, is (CHILD) covered by health insurance that is provided through an employer or union or obtained directly from an insurance company?” Type of response error = double barreled question

  48. Why refer to the final report? • Search categories may not be straightforward • Example: “At this time, is (CHILD) covered by health insurance that is provided through an employer or union or obtained directly from an insurance company?” Type of response error = double barreled question • The interpretive process and underlying construct can’t be gleaned from the search category • Example: “Have you ever taken any herbal supplements for yourself?” Type of response error = ambiguous concept

  49. Final Reports • Q-Bank is a good tool for quickly finding relevant reports of different cognitive testing projects • Searches not meant to replace final reports

  50. Questions? Kristen Miller KSMiller@CDC.GOV

More Related