Surveys as Student Learning MeasuresCLAS Unit Head MeetingSeptember 27, 2006Maria Cimitile, CLAS Associate DeanPriscilla Carreras, CLAS Graduate AssistantJulie Guevara, Accreditation and Assessment Officer
What are surveys? • Systematic efforts to collect information about people by asking them to respond to specific questions. • Helpful in gaining backgrounds, experiences, plans for the future, opinions, and attitudes. • Can use paper, phone, or internet format. • Provide insight into student attitudes about programs, assignments, courses, exams etc.
Survey Planning and Conceptualization • Should you use a survey? • What do you want to learn? • Who is the sample or audience? • How will you distribute the survey? • Sampling techniques – interviews vs. individual survey.
Survey Planning and Conceptualization • Obtain representative sample. • Determine time and people power. • Evaluate survey process when done. • Timeline – when it’ll be created, returned. • IRB Institutional Review Board. • Divide items into two categories: “have to know” and “nice to know”, then omit “nice to know”.
Designing the Survey QuestionsClose-ended Questions • Closed ended questions are those questions, which can be answered finitely. Quick and require little time investment. Can also provide incomplete responses. By definition, these questions can be restrictive. • Examples: Multiple choice Rankings Rating Scale Ecosystem Rating Scale Likert Scale
Multiple Choice • Quickly answered and tallied. No room for unexpected answers. Chemistry Example: - Goal: Graduate students who are well equipped to enter the job market. - Objective: Technical emphasis majors will enter the workforce and apply their skills and knowledge. - Measure: develop and send employer surveys to companies that hired our graduates 6-12 months after they’ve been hired.
Rankings • Tedious. • Assume respondents feel differently about each response. • Do not provide size of feeling between ranked items. • AVOID THEM!!! Example: on a scale where “10” means you have a great interest in a subject and “1” means you have none at all, how would you rank your interest in each of the following subjects?
Rating Scale • Another type of multiple choice question. Example: States conclusions based on data collection. • Excellent • Good • Fair • Poor
Eco-system Rating Scale • Ask for two ratings. The second gives information on the environment in which the first was made • Observes difference between growth obtained and growth desired. • Measures attainment of goals and objectives. Example: Ability to conduct a psychosocial assessment Importance: Not Critical __1 __2 __3 __4 __ 5 Vital Preparation: Poor __1 __2 __3 __4 __ 5 Excellent
Likert Scale(Agreement Scale) • Efficient • Permits comparison • Can be used in place of ranking Example: Employees exhibit requisite safety skills in the lab. Strongly Agree Agree Disagree Strongly Disagree NA
Numeric Open-ended Questions Example: In the past two years (1/04 – 1/06) how many of our graduates have you hired?
Text Open-ended Questions • Participants compose their own answers. • Provide infinite response. • Develop trust, are perceived as less threatening, allow an unrestrained or free response, and may be more useful with articulate users. • Not popular in surveys because: • Lengthen time required to complete survey. • Require more mental energy. Examples: How can we achieve the objectives of this course?
Writing Good Survey QuestionsKeep It Short and Simple (KISS) • Keep questions short. • Each item should ask only one question. • Use simple vocabulary – NO JARGON! • Make definitions, assumptions, and qualifiers clear. • Do not make significant memory demands.
Writing Good Survey Questions • Do not ask for broad generalizations of attitudes “Do you consider yourself a good student?” • Do not ask questions which are uncomfortable to answer honestly. • When possible, give the option of not knowing; Not applicable. • Avoid asking leading questions • Ex. Why are GVSU students your best employees? • Define terms, qualifiers, assumptions, etc.
Maximizing Participation Rate • Professional appearance of survey is important. • People are more likely to complete and return a survey if it looks like someone cares about it. • Be considerate of your participants. • Include self-addressed & stamped envelope. • Give adequate time for response. • Remember that they are doing you a favor.
Telephone Survey • Survey should be no more than 10 minutes. • Use simple format, with easy to follow directions. • Avoid open-ended questions. • Pre-test survey by reading it to people without looking at them. • Document attempts to reach participants.
Employer Surveys • Before completing employer survey, inform alumni. • Skills • Importance of skills in particular field/work place. • Skills employee attained at university or in specific program.
References • Creative Research Systems (2006). The Survey System. http://surveysystem.com/sdesign.htm • Introduction to Developing Surveys. Georgia Tech Office for Assessment. http://www.assessment.gatech.edu/eReports/slide_shows/Short_Guide_to_Survey_Research.pdf#search=%22developing%20surveys%22 • Palomba, C.A. and Banta, T.W. (1999). Assessment essentials. San Francisco, CA : Jossey-Bass. • Richardson, Dr. John V. (2002). Open versus closed ended questions in the reference environment. http://polaris.gseis.ucla.edu/jrichardson/dis220/openclosed.htm • Survey How To (2006). University of Texas at Austin Division of Instructional Innovation and Assessment. http://www.utexas.edu/academic/diia/assessment/iar/how_to/methods/survey.php • Suskie, L. (2004). Assessing student learning: A common sense guide. Bolton, MA : Anker Publishing.
USE SPARINGLY • When there are more than 6 possible answers. • When asking for occupation. • When structured item may bias responses by steering participants in one direction. • When asking a high interest question, such as “How would you change/improve/etc.”
Student Surveys • Surveys of graduating seniors. • Longitudinal Surveys of students attending university, enrolled in specific program.
Alumni Surveys • Employment related surveys. • Programmatic. • Departmental mission • Skill attainment
Who to Survey? • Students • Alumni • Employers