Assessment of Online Graduate Degree Programs

1. Assessment of Online Graduate Degree Programs Phil O’Leary Chair, Department of Engineering Professional Development University of Wisconsin- Madison Wayne Pferdehirt Director, Graduate Distance Degree Programs University of Wisconsin- Madison

2. Defining Quality:What Matters to Students? • Curriculum that Fits their Needs and Goals • Responsive Faculty • Culture that Fosters Supportive Community • Assignments that are Authentic and Meaningful • Elimination of Unnecessary Frustrations and Hurdles • Clear Instructions • Reliable Technology and Support • Removal/Mediation of Institutional BS • Knowing Their Feedback Matters

3. Defining Quality:What Matters to Faculty? • Capable, Motivated, Responsible Students • Support that is Proactive, Responsive, and Timely • Knowing their Efforts are Valued and Appreciated • Fair, Appropriate Compensation • New Opportunities • Knowing their Input Matters

4. Defining Quality: What Matters to Students’ Employers? • Seeing Impacts on Students’ Performance and Their Projects • Engagement with Students in Selection/Focus of Team Projects • Minimizing Conflicts with Work Schedule and Travel • Reputation of Institution • Reputation/Accreditation/Track Record of Program • Reputation and Experience of Faculty

5. Key Quality Areas:Where Does Quality Matter? • Courses • Program Curriculum • Technology Support • Student Services • Admissions • Staff Work Processes and Communications • Employer Relations • Alumni Relations • Institutional Support

6. MEPP Track Record • 96% Graduation Rate • 94% graduate on time (within 2 years) • University Continuing Education Association • “Outstanding Program Award” (2002) • Distance Learning “Program of Excellence Award” (2002) • U.S. Distance Learning Association • Award for “Excellence in Higher Education Distance Learning Programming” (2003) • Sloan Consortium • “Most Outstanding Online Teaching and Learning Program” (2004) • Program’s success profiled by: • National Society of Professional Engineers • American Society of Engineering Education • Institute of Industrial Engineers • Christian Science Monitor • Society of Women Engineers

7. Building Quality as Part of MEPP’s Program Planning and Launch • Curriculum built in response to market • Format designed for targeted students • Learned from other programs on and beyond campus • All courses tested via full pilot • Instructors were paid for course development • Cohort design enabled organic strength of student body • Faculty selectively recruited • Team approach to course design, evaluation and improvement • Curriculum includes initial course to orient, equip and focus students • Dedicated Director of Student Services

8. Elements of MEPP Evaluation each semester Course Evaluations at graduation Program Evaluation 9-12 months after graduation Program Impact Survey 5 years after graduation Five-Year Follow-up Survey

9. Course Evaluations • Learning Objectives Met? • Instructor(s)? • Course Elements • Workload • Most Valuable Lesson • Least Valuable Lesson • Suggestions

10. Course Review and Planning at End of Semester • Who? • Instructor(s), program director, instructional designer • Resources • Summary of student course evaluations • Course evaluation by instructor(s) • Trend chart for course and program-wide stats • Outcomes • Affirming what is working well • Plans for improving course content, format, logistics, … • Issues warranting further consideration • Plan and schedule for revisions

11. Trend Analysis Helps Interpret Current Semester Feedback

12. Program Completion Evaluation(conducted at graduation) • Addresses Program-wide Quality • Most Important Changes to Student’s Abilities • Program Strengths and Weaknesses • Salary/Promotion • Value of Each Course as Part of Program Curriculum • Recommended Changes to Program

13. Findings from Survey of Graduating Class: Graduates’ Ratings of Importance of MEPP Courses 1=should not be part of MEPP curriculum 2=not particularly important 3=neutral 4=important but not essential 5=course is essential part of MEPP

14. Program Impact Evaluation(conducted 1 to 2 years after graduation) • Graduate • Most significant changes in abilities • Changes in key skill areas • Key accomplishments since graduation • Salary/promotion impacts • Impacts on career/life • Suggested improvements • Workplace associates • Observed changes in key skill areas • Observed changes and accomplishments • Overall impact • Family • Observed changes re. work, education, family, confidence level, and career goals • Greatest sacrifice and greatest reward to family • Suggestions for improvement

15. Principles of Building Quality • Build culture that values, affirms and is committed to quality • People pay attention to what gets measured • Need to be able to the difficult discussions • Evaluation needs to be safe, supportive, team-based • Sometimes you have to change who’s on the bus • The value in evaluations is what you do with what you learn • Enable linkages between the right nerves and the right muscles • Find ways to model and affirm what you expect from others