Using Cognitive Interviews to Improve Survey Instruments. Presented at the Association for Institutional Research Forum June 2-6, 2012.
Presented at the Association for Institutional Research Forum
June 2-6, 2012
Heather Haeger, Indiana UniversityAmber Lambert, Indiana University-BloomingtonJillian Kinzie, Indiana University-BloomingtonJames Gieser, Indiana University
Conclusion and Discussion
Cognitive Interviews (CI) part of NSSE survey design from outset
Focus in 2005 to test survey among historically under-represented students
Planned NSSE update for 2013 provided occasion for multiple rounds of CIs
3 stages and related sub-stages that respondents faced during cognitive interviews:
1) Understanding the survey question and response options
a) Comprehending the survey question
b) Comprehending the response options
2) Performing the primary survey tasks
a) Retrieving information
b) Deduction; making conclusions about information
c) Mental arithmetic computation
3) Formatting responses
a) Mapping data yielded by primary task processes to an explicit response option
b) Response option is not available/offered
Types of Problems
Coding within these stages can address any of the following problems:
a) Political views,
b) Economic and social background,
c) Religious beliefs or philosophy of life,
d) Race, ethnic background, or country of
e) Sexual orientation
Question: In your experience at your institution during the current school year, about how often have you had serious conversations with people who differ from you in the following ways? (Never, Sometimes, Often, Very Often)
Example 1 (cont.)
Question: Indicate the quality of your interactions with the following people at your institution: a) Student Affairs Professional
Example 2 (cont.)
Participated in a community-based project as part of a regular course (i.e., service-learning)
Question: During the current school year, in about how many of your courses did you do the following?
Participate in a formal program where groups of students take two or more classes together (sometimes called a learning community)
Question: Which of the following have you done or do you plan to do before you graduate from your institution?
Example 4 (cont.)
Question: In a typical week this year, about how many total pages have you read for all of your courses?
How might these methods help triangulate NSSE results on your campus?
What questions are you most concerned about in terms of what your students mean by their responses?
Are there item terms that may have less face validity with your student populations?
How might it help to know more about students’ interpretations of response option in terms of what to do with findings?
How might you initiate this activity on your campus? Who might be interested in this work? Who should conduct the interviews?