Using Analytics to Intervene with Underperforming College Students

1. Using Analytics to Intervene with Underperforming College Students Kimberly Arnold (Purdue University)John Fritz (University of Maryland, Baltimore County)Eric Kunnen (Grand Rapids Community College) January 20, 2010

2. OVERVIEW • Analytics 101 • Five Minutes of Fame • Purdue University’s “Signals” • UMBC’s “Check My Activity” (CMA) • GRCC’s & Seneca College’s Project ASTRO • More Demos (time permitting) • Q&A

3. ANALYTICS 101

4. WHAT IF . . . • Can performance and/or backgrounds of past students predict success of future students? • How would we know? • If so, how would we communicate (intervene?) with students? With teachers? • How would this change teaching & learning?

5. CLASSROOM WALLS THAT TALK • Course or Learning Management Systems are NOT just content delivery or interactive learning environments. • The record or “residue” of online learning is a potentially rich data source that needs to be studied further. • How are schools thinking about this?

6. Five Stages of Analytics on Campus 1. Extraction and reporting of transaction-level data 2. Analysis and monitoring of operational performance 3. What-if decision support (such as scenario building) 4. Predictive modeling and simulation 5. Automatic triggers of business processes (such as alerts) ECAR Study: Academic Analytics: The Uses of Management Information and Technology in Higher Education – (2005).

7. FIVE MINUTES OF FAME

8. PURDUE’S “SIGNALS”

10. SIS data (historic) CMS and other technologies (real time) Other data Prediction Specific and customizable interventions

11. INTERVENTIONS • Customizable • Real-time • Specific • Actionable

12. 5 MINUTE OF FAME RESULTS • More Bs and Cs • Less Ds and Fs • Students are getting more help, earlier and more frequently • Faculty like that they can give feedback to large courses (150-1,200) • Students • Direct contact with faculty • Motivation • 60% say they got a better grade

13. UMBC’S “CHECK MY ACTIVITY” (CMA)

17. CURRENT VERSION

19. FUTURE VERSION

20. UMBC BLACKBOARD ACTIVITY BY GRADE DISTRIBUTION (2007-2009)

21. WHAT DOES THIS LOOK LIKE?

22. FA2008 SCI100 Findings • How would you describe the CMA’s view of your Bb activity compared to your peers? • 28% “I was surprised by what it showed me” • 12% “It confirmed what I already knew” • 42% “I’d have to use it more to determine its usefulness” • 16% “I haven’t used it.” • 2% did not respond to this question

23. FA2008 SCI100 Findings • If your instructor published a GDR for past assignments, would you be more or less inclined to use the CMA before future assignments are due? • 54% “More inclined” • 10% “Less inclined • 36% “Not sure”

24. FA2009 STUDENT USE

25. GRADE DISTRIBUTION Part of our public Blackboard Reports, run after the last day of classes every semester. Final GDR run after final grades are submitted. Faculty “opt in” by including a final letter grade in their Bb grade book with the column heading “GRADE.” www.umbc.edu/blackboard/reports.

26. "Project ASTRO" Blackboard Greenhouse Grant Eric Kunnen Coordinator of Instructional Technologies Grand Rapids Community College ekunnen@grcc.edu Santo Nucifora Manager of Systems Development and Innovation Seneca College santo.nucifora@senecac.on.ca

27. Overview • Evaluating Blackboard Use on Your Campus A review of the "Project ASTRO" Greenhouse Grant - Session Description: • Collecting and reporting on system activity information from Blackboard is often a challenge. Come and learn how to easily access reports on how Blackboard is being used by faculty, staff, and students which will help you: inform stakeholders, improve the engagement of end users, increase adoption, and to encourage deeper use of Blackboard by faculty, staff, and students. • Key Functions of Project ASTRO: • Tracking Automatic tracking (via Building Block) of courses, organizations, users, and tools. • Reporting Easy point and click access to advanced reports. • Discovery Ability to measure trends and analyze usage. • Sharing Inform key stakeholders of usage levels. • Acting Identify, target, and engage users using reports.

28. The Potential of Reporting • Usage Statistics – Ability to monitor and better determine which parts of the system are the most frequently used and less frequently used, thereby providing targeted promotion and training of specific tools or features that would benefit both faculty and students. (Student/Faculty Engagement, Adoption, Retention, and Satisfaction) • Accountability – Ability to provide metrics and trend data that can be obtained from the system to determine accurate statistics on usage for stakeholders requiring data-driven decisions and measures. (Department Action Plans, Continuous Quality Improvement) • Planning - Data obtained will enable institutions to better prepare and optimize system configuration, change management, manage upgrades, monitor performance, rolling out new tools, communication of issues, and investigate infrastructure hardware purchases for future growth. (Upgrade Management and System Growth) • Return on Investment - Data gathered can help determine and allow for the maximization of Blackboard use by faculty, staff, and students. (ROI and Accountability)

29. Dashboard • At a Glance Views • Active vs Inactive Courses including Departments • Top Courses and Organizations for Week • Top Tools Used by Instructors and Students • Top Portal Modules Used

30. Active Courses • Department vs Sub Department • Courses, Percentage, Instructors, Students • Semester Trends

31. Activity in Active Courses • Page Views (Hits) by Instructor and Student • Courses with Page Views vs Total Page Views • Break Down by Day, Week, Month, All

32. Top Week Tool Page Views • Student vs Instructor

33. Course Tool Items • Building Block Tool Tracking • Drill Down by Courses and Instructor Using Tools • Trends

34. Activity By Course

35. Activity by User

36. GRCC – STARFISH EARLY ALERT • Identify & Detect • Manual Flags • Automatic Flags • Attendance • Intervene & Track • Instructor • Advisor • Groups of Courses and Students • Improve & Retain • Student Communication and 360 Close Loop More info: http://www.starfishsolutions.com

37. GRCC - Starfish Example Instructor Manually Raises Flag The instructor can select one or more students from the student list and manually raise a flag on the student. When raising a flag, the instructor writes a description of the flagged issue. This information is forwarded to someone who can help the student, as determined by the flag rule setup by the administrator.

38. GRCC - STARFISH EXAMPLE INSTRUCTOR RESPONDS TO A “FLAG SURVEY” EMAIL Administrators can email survey requests to instructors. Clicking on the request takes the instructor to a flag survey where they are prompted to flag their students if they are experiencing any specified problems.

39. GRCC - STARFISH EXAMPLE AUTOMATIC FLAGS BASED ON BLACKBOARD GRADEBOOK/COURSE ACCESS Administrators can set up flags to be raised that are auto-generated. Flags can be raised by the system by grades and average scores and specific gradebook columns in Blackboard. Flags can also be raised based on students’ access to their courses in Blackboard. Additional customization is available through API’s.

40. MORE INFORMATION • Purdue University Signals Project Sitehttp://www.itap.purdue.edu/tlt/signals/ • UMBC’s Blackboard Reports & CMAhttp://www.umbc.edu/blackboard/reports • GRCC & Seneca College - Project ASTROhttp://projects.oscelot.org/gf/project/astro/ • GRCC Starfish Early Alert Project Sitehttp://www.grcc.edu/starfish

41. THANK YOUQuestions?