Cecilia Chan, IR Planning Analyst Paul Freebairn, Ed.D , Director of Assessment

1. Instructor Characteristics Correlated Positively with Overall Instructor Ratings at a Small, Ethnically-Diverse University Cecilia Chan, IR Planning Analyst Paul Freebairn, Ed.D, Director of Assessment Ronald M. Miller, Ph.D, Associate Professor 2009 AIR Forum, Atlanta June 2, 2009

2. BYU-Hawaii • 2400 students • Mission to serve Asia and the Pacific • 50% are international students • Students represent 70 different countries • At least two-thirds of our students speak two or more languages • Average class size: 25 • Student/faculty ratio: 17 to 1

3. Expanded Study • Sample size (859 to 2118) • Year (2007 only; now 2007 to April 2009) • Trends (similar) • Effect size and data impact • Majority of the findings remain similar to those of 2007

4. Overview • Introduction • General Trends • Best Predictors • Findings by Faculty Demographic Characteristics • Findings by Course Characteristics • Automated Neural Network Analysis • Conclusions

5. Previous Studies onStudent evaluations of teaching (SET) • Extensively researched since 1970’s; no general consensus. • Students’ perceptions of their grades appear to be related to the evaluations they give both for the class and the instructor. Both a leniency and a reciprocity effect have been found (Clayson, 2009) • Relatively valid against a variety of indicators hypothesized as potential biases, such as class size, grading leniency, workload and prior student interest (Wachtel, 1998) • The same instructor gets higher ratings when giving higher grades or teaching smaller classes (Greenwald and Gillmore, 1997)

6. Previous Studies onStudent evaluations of teaching (SET) • Negatively related to age and years of teaching experience (Feldman, 1983) • Provides valid information on instructor effectiveness; little evidence of bias, but typically shows substantial correlations with student achievement as measured by examination performance (d’Apollonia & Abrami, 1997) • No one has given a widely accepted definition of what “good” teaching is; universally agreeable criterion of teaching effectiveness has not been established (Kulik, 2001)

7.  Objective To investigate the relationships between various characteristics of faculty members, and their course and faculty ratings for possible course improvement

8. Overview of Instructors • 2118 individual classes evaluated from Jan ‘07 to April ‘09 • Majority of classes taught by teachers who were • from colleges of Business, Computing & Government (26%) or Language, Culture and Arts (26%) • non-tenured faculty status (41%) • employed for fewer than 5 years (44%) • male (68%)

9. Overview of Courses • Courses were mainly • 200 level (26%) • had fewer than 20 students evaluate the course (73%) • major courses (52%)

10. Primary Dependent Variables • Overall Course *Rating • Overall Faculty *Rating *7-point Scale (very poor, poor, fair, good, very good, excellent, exceptional)

11. Independent Variables • Faculty demographics • College, department, faculty status, years teaching at BYUH, type and gender • Course characteristics • Class level, number SETs, course department and class type • Evaluation Form includes • Two overall questions • 13 questions about the course • 18 questions about the instructor

12. Sample Questions • Course • “Course objectives are clear.” • “Assigned workload is appropriate for credit hours.” • Instructor • “Gives clear examples and explanations.” • “Motivates me by his/her example to want to learn about the subject.”

13. Findings

14. General Findings • The overall faculty rating is slightly higher than the overall course rating • High Correlation between the overall course rating and overall faculty rating (r=0.92, p<.00)

15. Regression Analyses • Conducted two multiple regression analyses • Examined the relationship between the overall ratings and specific questions for course and instructor

16. Regression AnalysisOverall Faculty • Eight of the “instructor” questions were significant in predicting overall instructor rating (listed in descending order of significance) • The regression model explains 87% of the variance (Adjusted R²= .87, p<.0000)

17. Most Significant PredictorsOverall Faculty

18. Non-Significant PredictorsOverall Faculty • 3.2 Is enthusiastic about the subject. • 3.8 Responds respectfully to student questions and viewpoints. • 3.10 Is available to students during regular and reasonable office hours. • 3.14 Seldom misses class. • 6c Help students prepare to live effectively in society.

19. Regression AnalysisOverall Course • Nine of the “course” questions were significant in predicting the overall course rating (listed in descending order of significance): • The regression model explains 85% of the variance (Adjusted R²= .85, p<.0000)

20. Most Significant PredictorsOverall Course

21. Non-Significant PredictorsOverall Course • 2.3 Student responsibilities are clearly defined. • 2.5 Assigned workload is appropriate for credit hours. • 2.11 Grading procedure is fair and impartial • 2.12 Assignments are appropriately distributed throughout the semester.

22. Findings by Demographics

23. Analyses of Variance (ANOVA) • Analyses of Variance compares specific faculty and course characteristics • Significant differences were found among faculty: • gender, status, position, years of teaching at BYUH, department, and college • Significant differences were found among course: • level, type, size and department

24. Findings by Demographics:Faculty

25. Overall Ratings by Faculty Gender

26. Overall Ratings by Faculty Status

27. Overall Ratings by Years Teaching at BYUH

28. Overall Ratings by Faculty Type

29. Overall Ratings by Faculty College

30. Overall Ratings by Faculty Department(Highest)

31. Overall Ratings by Faculty Department(Lowest)

32. Findings by Demographics: Course

33. Overall Ratingsby Course Department(Highest) • Overall Faculty • Courses with a smaller number of evaluations or that are more discussion-based tend to receive higher overall faculty ratings • Language classes also tend to be rated higher • Overall Course • Beginner level courses tend to have lower overall course ratings • Language classes do not have high overall course ratings

34. Overall Faculty by Course Department(Lowest)

35. Overall Ratingsby Course Department • Course type (like major vs GE) and number of student evaluating the class were found to have no confounding effect

36. Overall Ratingsby Course Type (Math Only)

37. Overall Ratingsby No. of Evaluation (Fine Arts Only)

38. Overall Ratings by Class Level

39. Overall Ratings by Class Type

40. Overall Ratingsby Number of Evaluation

41. Predicting Overall Ratings • A model with predictive validity of 94.3% was generated • Below are the best predictors in this model: • 3.11 Motivates me by his/her example to want to learn about the subject. • 3.12 Has produced new knowledge, skills and awareness in me. • 3.7 Clearly explains difficult concepts, ideas, or theories. • 3.5 Gives clear examples and explanations.

42. Predicting Overall Ratings • Below are the least predictive variables in this model • 2.10 Exams are good measures of my knowledge, understanding and ability to perform. • 2.12 Assignments are appropriately distributed throughout the semester. • 2.5 Assigned workload is appropriate for credit hours. • 2.9 Exams are clearly worded. • 3.10 Is available to students during regular and reasonable office hours.

43. Conclusions • Significant predictors • Questions emphasize learning-centered activities • Non-significant predictors • Questions are more faculty-interaction and housekeeping questions • Findings suggest that level of course difficulty has no effect on the overall ratings in the regressions

44. Conclusions • Male instructors tend to have higher overall ratings than female instructors • Instructors with non-continuing status are more likely to have higher ratings • There is a non-linear relationship between the overall ratings and years of teaching at BYUH

45. Conclusions • Faculty Type trends are consistent with faculty status and years of teaching at BYU-Hawaii trends • College of Languages, Cultures and Arts instructors have the highest overall rating than the other three colleges • Instructors of a more discussion-based department tend to have higher overall ratings

46. Conclusions • Instructors of a more science-based department tend to have lower overall ratings • Courses with a smaller number of evaluation or that are more discussion-based tend to receive higher overall faculty ratings • Language learning classes also tend to be rated higher.

47. Contact Information Paul H. Freebairn freebaip@byuh.edu Cecilia Yiu Chan ceci1027@gmail.com Paper Available online

48. References • Ambady, N., & Rosenthal, R. (1993). Half a minute: Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness. Journal of Personality and Social Psychology, 64(3), 431-441. • Arreola, R.A. (2000). Developing a comprehensive faculty evaluation system (2nd ed.). Bolton, MA: Anker Publishing Company, Inc. • Cashin, W.E. (1988). Student ratings of teaching: A summary of the research (IDEA Paper No. 20). Manhattan: Kansas State University, Center for Faculty Evaluation and Development. • Cashin, W. E., & Downey, R. G. (1992). Using global student rating items for summative evaluation. Journal of Educational Psychology, 84(4), 563-572. • Cashin, W.E. (1995). Student ratings of teaching: The research revisited (IDEA Paper No. 32). Manhattan: Kansas State University, Center for Faculty Evaluation and Development.

49. References • Clayson, D.E. (2009). Student evaluations of teaching: Are they related to what students learn? A meta- analysis and review of the literature. Journal of Marketing Education, 31(1), 16-30. • Costin, F., Greenough, W. T., & Menges, R. J. (1971). Student ratings of college teaching: reliability, validity, and usefulness. Review of Educational Research, 41(5), 511-535. • Crader, K.W., & Butler, J.K. Jr. (1996). Validity of students’ teaching evaluation scores: The Wimberly-Faulkner-Moxley Questionnaire. Educational and Psychological Measurement, 56(2), 304-314. • Cranton, P. A., & Smith, R. A. (1986). A New look at the effect of course characteristics on student ratings of instruction. American Educational Research Journal, 23(1), 117-128. • d’Apollonia, S., & Abrami, P.C. (1997). Navigating student ratings of instruction. American Psychologist, 52, 1198-1208.

50. References • Darling-Hammond, L., Wise, A. E., & Pease, S. R. (1983). Teacher evaluation in the organizational context: A review of the literature. Review of Educational Research, 53(3), 285-328. • Feldman, K. A. (1978). Course characteristics and college students' ratings of their teachers -What we know and what we don't. Research in Higher Education, 9, 199-242. • Feldman, K.A. (1983). The seniority and instructional experience of college teachers as related to the evaluations they receive from their students. Research in Higher Education, 18, 3-124. • Fernandez, J., Mateo, M.A., & Muniz, J. (1998). Is there a relationship between class size and student ratings of teaching quality? Educational and Psychological Measurement, 58(4), 596-604. • Gilbert, L. A. (1985). Dimensions of same-gender student-faculty role-model relationships. Sex Roles, 12(1-2), 111-123.