Enhancing & Evaluating Student Success: Integrating LMS and Assessment

1. Enhancing & Evaluating Student Success: Integrating LMS and Assessment Delia Heck Director, Assessment and Institutional Research Assistant Professor of Environmental Science Christine Stinson Chief Information Officer Ferrum College, Ferrum, Virginia Presentation at AIR Forum, June 6, 2007 www.ferrum.edu/dheck

2. Motivation • Many educational institutions use Learning Management Systems (LMS) to deliver and enhance student learning. • These same institutions are under increasing pressure to establish effective processes for evaluating (and improving) student learning outcomes.

3. Motivation continued • How do you use your LMS to get assessment data? • Can LMS aggregate and export information useful in evaluating student success?

4. Overview • Purpose of today’s presentation: • Review how we are integrating Assessment Process and LMS. • Share what we are doing, get your feedback and suggestions. • Background on Ferrum College • Developing a Culture of Assessment at Ferrum College. • Process of Choosing Assessment Software • Ferrum’s experience with LMS: 3 systems in last 5 years. • What are we doing now: Assessment, ANGEL 7.1, and ePortfolio. • What will we be doing: Assessment, ANGEL 7.2, and ePortfolio • Your suggestions, questions, comments, and discussion.

5. Ferrum College • Liberal arts college affiliated with the Methodist Church • Mainly residential undergraduate population. • 1000-1100 students, 60% male, 40% female • SW Virginia (eastern Appalachian region). • Member of 36-school Appalachian College Association. • Accredited by Southern Association of Colleges and Schools (SACS).

6. Overview and review of Assessment.

7. Assessment • What is assessment? • Systematic process to gather, analyze & interpret information for continuous improvement of student learning • What are learning outcomes? • Know, Do, & Be Like • What do we do with assessment results? • Improve student learning • Improve pedagogy • Collect learning artifacts

8. Assessment Process • Assessment should: • Set standards for programs & courses • Evaluate progress & provide feedback to students and program coordinators • Motivate & improve student learning • Help the institution enhance its effectiveness

9. Assessment Systems • Assessment Systems provide valuable feedback • How students are being shaped and transformed • Faculty learn how they can improve their own pedagogy • Coordination of LMS and Assessment Systems is potentially valuable, but… • faculty can be reluctant to embrace new programs that require large investments of time.  • Need to design assessment systems that meet the simultaneous needs of • Individual faculty • Programs or departments • Divisions or units • Overall institution

10. Evaluation of Assessment software alternatives.

11. Ferrum College’s Process • Determined our selection criteria • Critical criterion: Ease of use for faculty and staff. • Critical criterion: Interfaces with LMS. • Critical criterion: Curricular and co-curricular use.

12. Ferrum College’s Process • Key issues: • How to capture learning artifacts • How to tie learning artifacts to student learning outcomes • How to aggregate data about student learning outcomes across courses, programs & institution • Any redundancy of work between LMS and assessment system will render process “dead-in-the-water”

13. Assessment software products we considered • Evaluated various systems through on-line demos with Assessment Taskforce and Director of Instructional Technology • Nuventive • TaskStream • True Outcomes • WEAVE Online • Powersight (WebCT/Blackboard) • LiveText

14. Ferrum College’s Process • Two on-site, open demonstrations for campus community • TaskStream • True Outcomes • Participants at demonstrations completed evaluation sheets for each product.

15. Sample Evaluation Form • Learning Outcomes • Sample Learning Outcomes • National Standards • Co-curricular • On-line instructions • ePortfolio • Analyze & Track individual outcomes • Summarize across programs • Track data across outcomes • Interface with ERP • Technical Support • Time requirements • Costs – initial, direct, indirect

16. Ferrum College’s Process • All assessment software products were either too limited or too complex. • If the assessment process isn’t simple, faculty and staff members won’t do it well or happily.

17. Overview and review of Learning Management Systems.

18. Ferrum College and LMS • We had been searching for LMS that combined flexibility, ease of use, and allowed us to achieve institutional goals. • 2003-2004, 2004-2005 • used a Moodle-based system. • 2005-2006 • WebCT Vista (with Appalachian College Association) • Not pleased with cost/performance combination. • Needed LMS that integrated with our Assessment Process.

19. Evaluation of LMS Alternatives

20. Ferrum College and LMS • We have been searching for LMS that combined flexibility, ease of use, and allowed us to achieve institutional goals. • 2003-2004, 2004-2005 • used a “free” Moodle-based system. • 2005-2006 • WebCT Vista (with Appalachian College Association) • Not pleased with cost/performance combination. • Needed LMS that integrated with our Assessment Process. • 2006-2007, 2007-2008 • Happly home with ANGEL 7.1 (and, soon, 7.2).

21. Outcome and progress at Ferrum College.

22. Ferrum College and ANGEL Faculty Perspective… • Extensive training available for faculty. • Strong (voluntary) adoption in first year. • 60% of sections used ANGEL. • High faculty (and student) satisfaction.

23. How are we integrating Assessment and ANGEL? • Developing faculty culture of assessment. • Encouraging and supporting use of ANGEL 7.1 to deliver course content. • Encouraging and supporting use of ePortfolio to store artifacts (evidence of Learning Outcomes).

24. Faculty Culture of Assessment • Faculty owned and managed the process of developing assessment goals and methods. • Used pilot projects (course-level and program-level) to gradually introduce process. • Identified key players, both on early adopters-side as well as the objectors and worked to get their buy-in and find out their concerns.

25. Assessment Timeline

26. Example of using ANGEL to undertake Course Assessment • Professor used ANGEL Gradebook to record grades for subsets of major exams. • Each subset corresponded to course Learning Outcome.

27. ACC 202 A - Learning Outcomes • Be able to record economic events as accounting journal entries. • Be able to aggregate journal entries into a new Income Statement, Balance Sheet, and Statement of Cash Flows. • Be able to compute and interpret simple financial accounting ratios.

28. ACC 202 A - Results • 19 out of 20 students who took the Final Exam scored 70% or higher on Learning Outcome 1. • Only 3 of those 20 students scored 70% or higher on Learning Outcome 2. • 14 of 20 students scored 70% or higher on Learning Outcome 3 (5 of the 6 who scored poorly ran out of time on the exam).

29. ePortfolio and Assessment • Used by independently-accredited program now (Teacher Education). • Used by Director of Assessment now to collect evidence for College-level Learning Outcomes.

30. Creative Inquiry and Critical Thinking

31. Professional Capability

32. ePortfolio and Assessment • Used by independently-accredited programs now. • Used by Director of Assessment now to collect evidence for College-level Learning Outcomes. • Next year: use by Program Coordinators to collect evidence of Program Learning Outcomes. • Next year: use by students for personal portfolios with Academic Advisor & Student Affairs.

33. Environmental Science Program

34. What are our ANGEL 7.2 plans? • Import College-level and Program-level Learning Outcomes. • Train faculty to use these for their assessment activities. • Develop scheduled export of results to Program Coordinators, Associate Deans, Dean, and Director of Assessment. • Some reports built into system, others built by query

35. What potential problems exist? • Differences in vocabulary between our internal culture and ANGEL 7.2 (e.g., “Standards”). • Affects faculty perception of difficulty of adoption. • Not all faculty use LMS. • Not all assessments are conducive to on-line environment, but grades & feedback can be entered.

36. Summary • ANGEL 7.1 & 7.2 and ePortfolio are already supporting assessment of college learning outcomes. • Implemented solutions are institution-specific. • Process is key for our faculty and staff. • Software has not yet caught up with our needs, but we are getting there slowly.

37. Thank you! Questions? Suggestions? Comments?

38. E-mail Addresses • cstinson@ferrum.edu • dheck@ferrum.edu

39. Notes • Blackboard: http://www.fool.com/news/commentary/2006/commentary06081803.htm • eCollege: http://www.fool.com/news/mft/2006/mft06081123.htm • 2004 review of LMS: http://www.obhe.ac.uk/cgi-bin/keyresource.pl?resid=24 • UK Market share: http://mfeldstein.com/index.php/weblog/comments/359/ • combined market share of Blackboard and WebCT declined from 56.8% in 2001 to 43.7% in 2005. • share for Open Source went from 0% to 11%, with 8% of the gains happening in the last 2 years. • number of homegrown VLE’s jumped from 7% in 2001 to 30% in 2005. • Desire2Learn: http://www.desire2learn.com/ • Canada's 18th fastest growing technology company (2006 Deloitte Technology Fast 50 program). • Sakai: http://www.sakaiproject.org/ • ANGEL’s openness: http://findarticles.com/p/articles/mi_qa4057/is_200504/ai_n13642051/pg_1 • WebCT/Blackboard merger: http://www.infotoday.com/newsbreaks/nb051017-2.shtml • WebCT/Blackboard antitrust concerns: http://www.antitrustlawblog.com/article-on-line-learning-market-may-be-monopolized.html • Moodle: As of July 15, 2006 there were some 13,526 Moodle sites involving nearly 45,000 courses, five million users, over seven million enrollments, and over 150 countries. http://www.isu.edu/itrc/resources/moodle-info.shtml