Evaluating a Student Rating of Teaching Form

1. Evaluating a Student Rating of Teaching Form John Ogier Survey & Testing Unit (STU) University Centre for Teaching & Learning (UCTL) john.ogier@canterbury.ac.nz Ph 64-3-364-2850

2. A case of quality not being assured! “No single student rating item, nor set of related items will be useful for all purposes” Cashin (1995) • “Student ratings should be used to make only crude judgements of instructional effectiveness (exceptional, adequate, and unacceptable).” • D’Apollonia & Abrami (1997)

3. Course Organisation a well organised course Stimulation helped to stimulate my interest in the course area Workload The overall workload in this course was reasonable Difficulty The level of difficulty of this course was reasonable Overall Overall, this was a good quality course Teaching Organisation classes were well organised Communication able to communicate ideas and information clearly Interest stimulated my interest in the subject Attitude attitude towards students has been good Overall Overall, the lecturer is an effective teacher UOC Student Surveys

5. How appropriate are these 5 Likert scale questions? Do they define a “good teacher”? Is that measure reliable? 1-5 scale appropriate? Some cultures are use to “1” being a good score Is 3 really “neutral”? Students circle 2 no’s! Students circle a column!! Is it correct to use the mean of a discontinuous scale? Median rather than mean? Effective?

6. Other survey “quality” issues • Was the class told about it in advance? • What extra questions were asked? • When in the semester was the survey held? • What time was the survey held? • How many other surveys were given out? • Who administered the survey? • How long were they given to complete it? • Is it really anonymous? • Ad hominem comments!

7. Course Organisation Stimulation Workload Difficulty Overall Teaching Organisation Communication Interest Attitude Overall Possible factors

8. Correlation of Q5 (Overall) mean against the means of Q1, Q2, Q3 & Q4weighted by responses (capped at 50) The lecturer is able to communicate ideas and information clearly Overall, the lecturer is an effective teacher

9. Scatterplot & Regression Line- Overall v Communication

10. Where to now? • Difficulties faced by tertiary NESB students in English language lectures • Cosmopolitan UOC academics – both NESB & ESB students must cope! • Finegan, T.A., & Siegfried, J.J. (2000). Are student ratings of teaching effectiveness influenced by instructors’ English language proficiency? • they found in an introductory Economics course that“the student ratings of ESL instructors are, on average, about 0.4 points lower, on a scale of 1 to 5, than the student ratings of native English speaking instructors.”The American Economist, 44, 17-29.

11. NESB by Subjective Classification • Surveys (UGrad, 2 or more responses) - 7072 from ESB lecturers and 524 from NESB lecturers

12. A similar difference, but… • What about across • Faculties? • Levels? • Departments?

13. Number of Cases within grouping variables (and no. of lecturers surveyed) Arts Commerce Engineering Science Law Level Y ESL N Y ESL N Y ESL N Y ESL N Y ESL N 1 (Lect) 14 (8) 476 (153) 12 (5) 98 (31) 0 38 (15) 42 (12) 585 (167) 0 29 (10) 2 (Lect) 40 (16) 678 (196) 24 (9) 213 (62) 22 (6) 245 (81) 89 (23) 713 (206) 0 88 (18) 3 (Lect) 31 (13) 517 (151) 46 (14) 267 (83) 49 (15) 387 (96) 98 (24) 820 (218) 0 193 (29) 4 (Lect) 0 0 0 0 57 (17) 537 (148) 0 0 0 0 Breakdown of surveys

14. Faculty Means

15. Why the faculty effect? • Cashin (1990) mentions a hypothesis that • students’ quantitative skills are less well developed than their verbal skills so that quantitative based courses are more difficult for students and are more difficult for lecturers to teach.

16. By Level

17. Some relevant student comments • The following student comments from relevant courses put ‘quantitative skills’ into perspective: • ~ “He did the best he could but his accent was still very strong, and made a complex subject even harder to understand.” • Ratings: ‘Comms’ = 2, ‘Overall’ = 3 • ~ “.. we should have loud, clearly spoken English from the lecturer as otherwise you have to concentrate too hard on just understanding the lecturer.” • Ratings: ‘Comms’ = 1, ‘Overall’ = 1 • Compare a NESB student rating an ESB lecturer ~ a strong accent, • ~ “It’ll be very interesting if the lecturer can speak in Thai.” • Ratings: ‘Comms’ = 2, ‘Overall’ = 2.

18. Implications • For a “Transmission model” of teaching, do we need anything more than a Communication question!!! ? • Need questions that cover the important aspects of teaching (or courses) with an appropriate weighting. • NESB lecturers may have a tougher time gaining promotion and tenure. • Assistance with language? • Alternative communication methods? • Assisting ESB & NESB students to understand and cope?

19. The NESB student’s view • Econ104 ~ 33% NESB Students • Online survey – 353 responses ESB, 165 NESB • Overall – ESB 4.58 ± 0.07, NESB 4.28 ± 0.10 • “... And think about the language he use in the test paper some words I even do not know and if he want to use it then use it in his lecture. ECON test is not a word game its aim should be test people how well they learn for ECON!” Ratings: ‘Comms’ = 3, ‘Overall’ = 4

20. Questions? John Ogier john.ogier@canterbury.ac.nz Ph 64-3-364-2850