1 / 39

Lessons in Implementing a Post-Graduation and Learning-Gains Questionnaire

Lessons in Implementing a Post-Graduation and Learning-Gains Questionnaire. Sara Lazenby sllazenby@wisc.edu University of Wisconsin - Madison May 21, 2013 AIR Annual Forum. Why implement an institutional exit questionnaire?.

cady
Download Presentation

Lessons in Implementing a Post-Graduation and Learning-Gains Questionnaire

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Lessons in Implementing a Post-Graduation and Learning-Gains Questionnaire Sara Lazenby sllazenby@wisc.edu University of Wisconsin - Madison May 21, 2013 AIR Annual Forum

  2. Why implement an institutional exit questionnaire? • Prior to 2008, UW-Madison had no institutional exit questionnaire for Bachelor’s degree recipients • The need to implement a questionnaire arose due to: • Voluntary System of Accountability (VSA) question on future plans • Higher Education Opportunity Act of 2008 • Essential Learning Outcomes adopted in 2008 • Priority of campus leadership

  3. What is the Voluntary System of Accountability? • A voluntary initiative developed by the higher education community to meet the following objectives: • Demonstrate accountability and stewardship to the public • Support institutions in the measurement of educational outcomes and facilitate the implementation of effective practices as part of institutional improvement efforts. • Assemble and disseminate information that is transparent, comparable, and understandable • Provide a useful too for students during the college search process • Sponsored by the Association of Public and Land-grant Universities and the American Association of State Colleges and Universities • More information available at: http://www.voluntarysystem.org • Viewable on the College Portrait: http://www.collegeportraits.org/

  4. Developing Criteria for an Exit Questionnaire – Phase 1 • Include the required VSA question • Create a cost-effective and sustainable questionnaire • Integrated into the student record • Census approach has potential for high response rates • Reporting considerations

  5. Importance of the questionnaire design • Developer coded to be part of our Student Portal • As part of the Student Record, student responses are protected by FERPA • Placement in ISIS/PeopleSoft means the questionnaire is easily connected to other student information • No need to ask additional questions for demographic/academic data

  6. Questionnaire Timeline • Module (annoyingly) stays live in student’s MyUW portal until student responds • During the questionnaire period, the data is extracted into an InfoAccess data warehouse table and refreshed nightly. This is available to anyone who also has access to the relevant tables -11 Days: E-mail invitation goes out End of Term (December and May) -12 Days +21 Days Questionnaire goes live Questionnaire deactivated -4 Days: Reminder e-mail sent

  7. ELOs and Post-Grad Plans Many contributions to this project’s success: Questionnaire Development Voluntary System of Accountability AAU Data Exchange Senior Exit Survey Team NSSE University of Illinois UW-Madison Convergence Team AACU’s LEAP UW-Madison Assessment Fund John Stevenson, UW Survey Center Jennifer Dykema, UW Survey Center Jeremy Beach, UW Survey Center Student participants in focus groups Technical Development and Implementation Connie Chapman, Registrar’s Office Michael Fay, DoIT James Helwig, DoIT Rob Lauer, DoIT Wanjiru Pontes, DoIT Stephan Wahe, DoIT Bob Mayville, DoIT Betty McIlwee, DoIT Jim Steele, Registrar’s Office Kay Steiner, InfoAccess, DoIT

  8. Example of Portal Appearance

  9. Examples of the Questionnaire

  10. Examples of the Questionnaire – Required VSA Question

  11. Examples of the Questionnaire

  12. Early Lessons Learned • Quick resolutions needed for technical issues • Varying response rates by question • Some questions did not produce usable data • Salary • Workplace/Next Institution

  13. Developing Criteria for an Exit Questionnaire – Phase 2 • Need for indirect measures of the Essential Learning Outcomes • Questionnaire fatigue considerations • Removal of questions that weren’t useful

  14. http://www.provost.wisc.edu/content/WI_Exp_ELOs.pdf

  15. Question Development • Indirect measures of learning, student perceptions • ELOs, UI-UC, NSSE were all influences • UW Survey Center did review and pre- testing, including layout and syntax • Added questions to the post-graduation plans questionnaire in December 2010 and into the future

  16. What are our response rates?

  17. Response Rates – Effects of Reminder E-mails December 2012 – Number of responses by day. May 2012 – Number of responses by day.

  18. Communicating Findings • Time lag before degrees are posted • Challenge: Displaying the results in a way that campus can easily understand. • What do the results mean?

  19. Analyzing and Disseminating data • Campus Wide • Schools/Colleges • Individual Majors for Program Review • Disciplinary “Divisions” • Arts and Humanities • Biological Sciences • Physical Sciences • Social Studies

  20. Post Graduation Plans – ALL GRADUATES Fall 2011 and Spring 2012 Bachelor’s Degree Recipients What is most likely to be your principal activity/activities after graduation? (Unduplicated) If employment selected, which best describes your plans at this time? Have accepted a position Work Full-Time Considering one or more offers Work Part-Time Graduate School Full-Time Currently searching for a position Other Will begin searching for a position after graduation Other includes: graduate school part-time, additional coursework, military service, volunteering, and unspecified activities. Based on 3083 responses (54% response rate) Do you have a valid passport? Yes, from the United States Do you plan to work in Wisconsin after graduation? Yes, from another country Yes Undecided No No Based on 2,689 responses (47% response rate) Based on 1853responses (87% response rate among those with work as their primary activity) SL, Academic Planning and Institutional Research, 10/12Source: Post Graduation Plans Questionnaire Responses)

  21. Post Graduation Plans – Campus Summary Trend in VSA question responses over time What is most likely to be your principal activity/activities after graduation? (Unduplicated)

  22. Post Graduation Plans – Essential Learning Outcomes • Eighteen questions that provide a self-assessment at graduation, that maps to the essential learning outcomes at UW-Madison. • Respondents asked to assess their ability upon entering UW-Madison and at graduation. • Results are tabulated as the non-null average for each question

  23. All this data, now how do we communicate findings? • Display Challenges • Need something visual to help people interpret the data • Understanding differences • A lower learning gain on a particular metric wasn’t always “negative” • Putting learning gains in the context of our general education requirements • Differences in learning outcomes match our expectations given major content.

  24. Post Graduation Plans – ALL GRADUATES “How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 2,828 to 2,886 responses, depending on question.

  25. Post Graduation Plans – Biological Sciences Division 11-12“How able were/are you to …” SL, Academic Planning and Analysis, 8/11 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 423 to 431 responses, depending on question.

  26. Post Graduation Plans – A Biological Science Major“How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 104 to 107 responses, depending on question.

  27. Post Graduation Plans – Arts and Humanities Division 11-12“How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 398 to 407 responses, depending on question.

  28. Post Graduation Plans – An Arts and Humanities Major“How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 63 to 66 responses, depending on question.

  29. Post Graduation Plans – Physical Sciences Division 2011-12“How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 366 to 375 responses, depending on question.

  30. Post Graduation Plans – A Physical Science Major “How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 71 to 73 responses, depending on question.

  31. Post Graduation Plans – Social Studies Division 11-12“How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 883 to 903 responses, depending on question.

  32. Post Graduation Plans – A Social Studies Major “How able were/are you to …” SL, Academic Planning and Institutional Research, 10/12 Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –Extremely Responses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 112 to 115 responses, depending on question.

  33. Majors with at least 50 responses to question about plans after graduation • Biochemistry • Biology • Business: Accounting • Business: Finance, Investment, and Banking • Business: Management and Human Resources • Business: Marketing • Business: Risk Management and Insurance • Civil Engineering • Communication Arts • Computer Sciences • Economics • English • Environmental Studies • History • International Studies • Journalism • Mathematics • Mechanical Engineering • Nursing • Political Science • Psychology • Sociology • Spanish • Zoology

  34. Lessons Learned • Have worked with campus partners to try and increase response rates • Reminder e-mails! • Overlap with “exit surveys” within School/College and Departments • Low cost, not a lot of opportunity for changes • Stability of questionnaire has advantages • Popularity of handheld/tablet devices impact response rates

  35. Opportunities and Challenges • Going forward … • Opportunities to make part of the graduation process • Changes in regulations may necessitate a different approach • Continued collaboration with other units, trying to help them understand the value of the questionnaire

  36. So far, it’s working! • Involved a lot of creative people with a wide range of skills and experience. • Inspiration from our peers/competitors. • Made use of our existing enterprise systems • ISIS/PeopleSoft • MyUW student portal • InfoAccess (data warehouse) • Census approach has potential for high response rate. • Cost-effective, sustainable collection and reporting system.

  37. Questions and Discussionsllazenby@wisc.edu https://apir.wisc.edu/pgp.htm

More Related