Using an On-line Survey Tool to Streamline Outcomes Assessment

1. Using an On-line Survey Tool to Streamline Outcomes Assessment Phillip R. Rosenkrantz, Ed.D., P.E.Industrial & Manufacturing Engineering DepartmentCalifornia State University, Pomona

2. Topics • Introduction • Examples of Survey Use • Design & Deployment Features • Analysis & Sharing Features • Questions/Comments

3. Introduction • SurveyMonkey on-line surveys are being used for a variety of applications • Motivated by past experiences with data collection and analysis for ABET visits • SurveyMonkey used for a variety of assessment jobs • Other companies offer similar products (e.g. Zoomerang)

4. Examples • College Level Assessment • Program Level Assessment • Course Level Management & Assessment • First Year Experience • Capstone Course Sequence

5. College Level Assessment • Cal Poly Pomona Project Symposium Feedback • Industry evaluators for senior project presentations surveyed for overall assessment of outcomes vs. importance • Gap analysis was easily created using summarized survey data • College of Engineering Assessment Interest Survey • College faculty and administrators involved with preparing the self study surveyed to identify assessment committee priorities

6. Program Level Assessment • IME Department 2005 Alumni Survey –Every three years to: • Assess progress on objectives • Feedback on changing curriculum needs. • Student Advising Survey • Comprehensive survey to explore student attitudes and practices • Revised advising program based on actual student feedback.

7. Program Level Assessment • IME Basic Knowledge Survey – • Assesses student retention of knowledge and concepts from five selected lower division courses. • Valuable in finding out where to put place efforts to increase learning. • Validity and Reliability were considered • ME Summer Class Needs Survey • A quick survey of ME students regarding summer course offerings • resulted in higher-than-expected summer course attendance

8. Course Level Management & Assessment • IME Outcomes Assessment Survey • Primary outcomes assessment tool for a-k outcomes in key courses • Looks for improvement & weaknesses to target for special attention. • IME 312 Statistics Knowledge Survey • Pre and Post knowledge survey to assess student learning • Uses Bloom’s Taxonomy. • EGR 403 On-line Introductory Survey • Find out student interests, strengths, and weaknesses for forming project teams • EGR 403 Team Member Evaluations • Student assessment of team members • Used midway through and at the end of the project • IME 301 Project Proposal • Proposal form for student data collection and analysis projects. • Faster and more efficient than Blackboard survey tool

9. First Year Experience • Engineering First Year Experience Survey • Assess the results of the EGR 100 first year experience course • IME 112 Team Evaluation Survey • Introductory course team evaluation survey. • IME 112 Time Use & Management Survey • Input time management assignment results. • Summarized feedback about student habits and practices. • IME 112 Outcomes Survey • Assessment of specific course features and outcomes

10. Capstone Course Related Assessments • Sample Senior Project Report Assessment • Students use to practice scoring written reports as part of a Senior Project preparation course. • Sample Senior Project Presentation Assessment • Students use to practice scoring video tapes of oral report • Senior Project Report Assessment • Faculty input assessment of written senior project reports • Reflective Piece Results • Evaluator asked to input assessment of a reflective piece • Written by students after completion of the senior project • Covers three a-k soft skill outcomes scored by a rubric. • Industry Supervisor Senior Project Assessment • Industry feedback on professionalism of Senior Project students. • The survey link is sent in an email message.

11. Design and Deployment Features • Relatively inexpensive for the features provided • SurveyMonkeyand Zoomerangmonthly “Pro” packages for about $19/month • Annual packages available for $200/year that will handle amount needed • Ease of creating a new survey or starting with a previous survey • User friendly • Old surveys can be imported as templates for new surveys • Color schemes and customized page design • Many color schemes and formats to choose from • Customized designs can be created with a little more effort

12. Design and Deployment Features • Paging and end of survey options • Can be one page long or broken into separate pages • Responders can be sent to various “thank you” or other pages • Can embed survey links in email & web pages and send reminders • Links created for embedding in email or web pages • Can send to email list for tracking and reminder messages • Reminder feature does produce good results • Surveys can be opened or closed for data collection as necessary

13. Analysis and Sharing Features • Sharing survey results • Unfiltered survey results can be shared with others • Various levels of sharing interaction allowed • Can be password protected • Ability to filter the results by response alternative (e.g., demographic or specific question choice • Surveys can be reused • Results can be filtered to a specific demographic combination • Avoids need to create a new survey for each application • Can name and save filters • Ability to collect surveys with options required for IRB approval • Can create various levels of anonymity and confidentiality • IP address, email address, and other info can be excluded

14. Analysis and Sharing Features • Can create crosstabs for up to five choices for any question • Powerful, but limited option • Data downloadable in various forms depending how needed • Data can be downloaded to a spreadsheet as raw data • Can be summarized various ways • Virtual database can be created by carefully choosing questions • Demographic questions can be chosen to create a data base allowing for analyzing data over time • Useful feature for outcomes assessment

15. For Additional Information • Phillip R. Rosenkrantz, Ed.D., P.E.rosenkrantz@csupomona.edu • IME Outcomes, Objectives & Assessment http://www.csupomona.edu/~ime/objectives.htm