Early Warning Grades: Heeding the Warning

1. Early Warning Grades: Heeding the Warning Early Warning Grades:Heeding the Warning William Polley, Ph.D. Angela Lynn, Ph.D. Fall 2012 IACRAO Conference Session 25.25

2. Institutional Profile • Public, Four-Year University in West Central IL • Residential campus in Macomb, IL • Non-residential campus in Quad Cities/Moline, IL • Founded as a Teacher’s College in 1899 • Offer 66 baccalaureate degrees, 36 master’s degrees and 1 doctoral degree • Total enrollment Fall 2012: 12,205 students (Undergraduate: 10,263, Graduate: 1,942)

3. Early Warning Background • Originated in Fall 1982 • Initial target population • Freshmen, sophomores, transfers, academic probation • Notices to students earning less than C grades • Current population • Undergraduates in regularly scheduled on-campus courses • Continued purpose • Timely communication • Improve student performance • Increase retention

4. Purpose of Study • Ad hoc committee of Faculty Senate established • Membership • Voting: 6 faculty & 2 students • Ex-officio: 1 academic advisor & registrar • Charge of committee • Identify ways to improve faculty participation • Determine best timing of early warning grades • Consider conversion to midterm grades • Address any other appropriate issues

5. Plan for the Study • Consider timing issue • Review early warning and grade data from Fall 2011 and retention from fall to spring • Survey faculty and students for attitude toward early warning system • Consider midterm grades in light of grade data and surveys • Any obvious ways to bring system up to date?

6. Timing of Early Warning Grades • Easiest issue to address • Institutional processing vs. individual notices • Early enough to affect change • Late enough to have graded work • Early warning notices primarily dictated by last day to withdraw • Online screen open to faculty in 6th week of semester • Two weeks before notices sent to students • Notices sent to students at start of 8th week of semester • Two weeks before withdrawal date

7. Early Warning System • Specific grades are reported only for C- and below. • C or better is reported as “*”. • If the instructor does not submit, it is reported as “N/A”. • Student and adviser receive e-mail alert if any C- or below. • No e-mail alert if all “*” or “N/A”. • Student can see their report on STARS even if all are “*” or “N/A”.

8. Early Warning and Retention • The wrong question to ask: • What is the retention rate among students receiving an early warning (C- or below)? • Too easy. You already know the answer.

9. Early Warning and Retention • Better questions More informative answers • What is the effect of receiving an early warning report by e-mail? • The “Hidden N/A” problem • Controlling for academic factors (GPA, whether regularly admitted, whether they maintained a C or better in any class), does a “hidden N/A” matter?

10. Early Warning and Retention • A “hidden N/A” does have a statistically significant impact on fall to spring retention. • Our statistical model (probit regression) predicts: • Students with a 2.0 GPA and who had one “C or better” (“*”) on their report but had a “hidden N/A” had a fall to spring retention rate of 83% compared to 87% for students with no “hidden N/As.”

11. Early Warning and Retention • Statistically significant, but not extremely large • About 19 students for a school our size • Much larger effect from one of our control variables • Did they maintain at least one C or better?

12. Early Warning and Retention • Students who maintain at least one C or better had a fall to spring retention rate of 90%. • Students who did not maintain a C from midterm to final had fall to spring retention rates much lower—around 56%. • These are overall percentages, independent of GPA (which is the most statistically significant predictor of retention included in our model).

13. Faculty and Student Attitudes • Surveyed students and faculty • Focused on faculty who taught freshmen • About 70% of responding faculty (N=190) • Most questions on a 5 point Likert scale from 1=Strongly Disagree to 5=Strongly Agree

14. Faculty Attitudes • Among faculty teaching freshmen… • 43% can determine midterm grades with a high degree of precision. • 44% can distinguish students in serious difficulty, but not specific letter grades at midterm. • Full professors and faculty not teaching freshmen were more likely be in the 2nd group (can’t determine specific letter grade).

15. Faculty Attitudes • Question A: The nature of my course is such that midterm grades or early warning grades are unnecessary in my opinion. • About 19% of faculty respondents either agree or strongly agree.

16. Faculty Attitudes • Question B: Midterm grades or early warning grades are a good idea in general, but my students don’t need them because they should know where they stand. • About 39% of faculty respondents either agree or strongly agree.

17. Faculty Attitudes • Question C: I am aware of students of mine who have dramatically improved their performance after receiving a low early warning grade. • About 41% of faculty respondents either agree or strongly agree.

18. Faculty Attitudes • Question D: Informing students of their performance throughout the semester, including giving early warning grades, is something that I consider to be a priority. • About 66% of faculty respondents either agree or strongly agree.

19. Faculty Attitudes • Question E: If no students are earning less than a C at the time when early warning grades are due, then I am less likely to log into the system and submit the early warning grade roster. • About 26% of faculty respondents either agree or strongly agree.

20. Faculty Attitudes • Question F: The culture of my department/college encourages the submission of early warning grades. • About 59% of faculty respondents either agree or strongly agree.

21. Faculty Attitudes: Major Findings • 63% (12 of 19) of faculty who did NOT submit early warning grades responded with agreement that they are unnecessary (question A). • 50% (9 of 18) of faculty who did NOT submit early warning grades responded with agreement that midterm grades are a good idea in general but that their students should know where they stand (question B).

22. Faculty Attitudes: Major Findings • All 11 of faculty who strongly agree with the statement that they are aware of students who have improved as a result of early warning grades did submit early warning grades. • All 41 of faculty who strongly agree with the statement that communication is a priority submitted early warning grades. (94% combined agree and strongly agree).

23. Faculty Attitudes: Major Findings • Only 54% (7 of 13) of those who strongly disagree that communication is a priority submitted early warning grades. • Submission rates decreased monotonically with increasing agreement with question E.

24. Student Attitudes • The following slides report the survey results for freshmen only. • Retention focus • Number of responses from students by class were 79, 81, 138, and 148 (Fr., So., Jr., Sr.) • Upperclassmen responses more reflective of their experience • Summary of differences between freshmen and upperclassmen at the end of this section

25. Student Attitudes • Question A: Receiving early warning grades is important to me.

26. Student Attitudes • Question B: If I received an early warning grade of C- or below, I would contact the instructor of that course as soon as possible.

27. Student Attitudes • Question C: I am frequently surprised by my early warning grades.

28. Student Attitudes • Question D: If I do not receive an early warning grade in a course, I have other ways of determining my grade (e.g. Western Online).

29. Student Attitudes • Question E: I would like to receive a specific letter grade in each course at midterm.

30. Student Attitudes • Question F: If I received an early warning grade of C- or below, I would be likely to drop the course.

31. Student Attitudes • Question G: If I received an early warning grade of C- or below, I would consider dropping the course, but am more likely to remain in the course.

32. Student Attitudes • Question H: If I received an early warning grade of C- or below, I would likely be contacted by my academic adviser or other University staff member regarding my grades.

33. Student Attitudes • Question I: If I received an early warning grade of C- or below, I would seek additional help from tutoring labs, the Office of Academic Services, my academic adviser, and other resources as necessary.

34. Student Attitudes • Question J: If I received an early warning grade of F, I would be likely to drop the course.

35. Student Attitudes: Major Findings • A. Freshmen are more likely to say that “Receiving early warning grades is important to me.” • E. Freshmen are more likely to say, “I would like to receive a specific letter grade in each course at midterm.”

36. Student Attitudes: Major Findings • H. Freshmen are more likely to say, “If I received an early warning grade of C- or below, I would likely be contacted by my academic adviser or other University staff member regarding my grades.”

37. Student Attitudes: Major Findings • I. Freshmen are more likely to say, “If I received an early warning grade of C- or below, I would seek additional help from tutoring labs, the Office of Academic Services, my academic adviser, and other resources as necessary.” • J. Freshmen are less likely to say, “If I received an early warning grade of F, I would be likely to drop the course.”

38. The Secret of Our Success: Participation • First, do no harm! • Voluntary system • Participation by faculty teaching freshmen is reasonably good • Lower participation in classes where early warnings are probably not critically needed (physical education activities, music ensembles, and the like)

39. Midterm Grades? • Changing to a midterm grade system reporting specific grades (A, B, C, D, F with +/-) would probably not improve retention. • Simply maintaining a C or better is what we find to be critical to retention. • Participation in a voluntary system of midterm grades would almost surely be lower than early warning system. • Mandatory midterm grades seem redundant in an era of online gradebooks.

40. Low Hanging Fruit? • Is there any easy change to the system that would have the potential to extend the reach, encourage more participation, etc.? • Yes! Online courses • Online courses had been excluded from the population of courses for which early warnings were submitted. • Why? Possibly because when online courses first appeared, they were often asynchronous, not to mention experimental. Not so today.

41. Committee Recommendations • Maintain the existing timetable for submission of early warning grades • Maintain the system of early warning grades, as opposed to specific midterm grades • Expand the early warning grade population to include online courses • Recommend to Provost that resources be allocated for the purchase of GradesFirst for the Office of Academic Services • Study the effectiveness of the inclusion of online courses and the use of GradesFirst after their implementation

42. Questions??? Fall 2012 IACRAO Conference Session 25.25