Anders Jonsson, Malmo University, Sweden Liesbeth Baartman, Utrecht University, The Netherlands

1. Estimating the Quality of New Modes of Assessment: The Case of an “Interactive Examination” for Teacher Competence Anders Jonsson, Malmo University, Sweden Liesbeth Baartman, Utrecht University, The Netherlands

2. Competence assessment Assesses the ability to integrate knowledge, skills and attitudes in order to master relevant (job) situations. Consists of multiple modes of assessments: e.g. performance assessments, conventional testing, self- and peer assessment etc.

3. Task specification Problematic areas: Marker consistency

4. Task specification The tasks should demand the knowledge required for mastery of the specified domain – and only that knowledge. The principle of equity demands that irrelevant variables are not contaminating the results.

5. Consistency of marking Marker consistency is problematic because interpretations about assessment performance can vary – both within a single assessor and between different assessors. The principle of equity demands that the results are credible and trustworthy.

6. Why use performance assessment in higher education? A main reason: Traditional testing can only measure a limited part of students’ competence… …and other modes of assessment are needed in order to ”capture” the rest.

7. An assessment based on only a limited part of the educational objectives is not fair, and the grade/degree will not represent the competence at stake.

8. Everything else Reliability What is quality in assessment?

9. What is quality in assessment? Reliability Everything else

10. What is quality in assessment? Reliability Everything else

11. Dual purpose:1. Summative – assessment of learning, and 2. Formative – assessment for learning.

12. ”The Wheel of Competency Assessment” From Baartman et al. (2006)

13. Self- assessment Analysis of situations Comparison with expert Feedback Assessment Evaluation 1From Mattheos et al. (2004) The Interactive Examination1

14. Self-assessment Analysis of situations Expert comparison Feedback Assessment Evaluation The Interactive Examination 1. Quantitative self-assessment16 questions graded from1 (poor) to 6 (excellent)

15. Self-assessment Analysis of situations Expert comparison Feedback Assessment Evaluation The Interactive Examination Analysis of simulated classroom situationsThree fields:ObservationAnalysisTaking action

16. Självvärdering Analys av situationer Jämförelse med expert Feedback Bedömning Utvärdering The Interactive Examination Comparison with “expert”: Qualitative self-assessment a) Identify differences between your own and the ”expert’s” analysis of the situation.b) State reasons for the differences.c) Identify shortcomings in your competency as a teacher and, on the basis of these, suggest own educational needs. Anders Jönsson, Malmö Högskola

17. Self-assessment Analysis of situations Expert comparison Feedback Assessment Evaluation The Interactive Examination EvaluationStudents’ perceptions of the methodologyMostlyLikert-scale questions1 (Do not agree) – 9 (Agree)

18. Self-assessment Analysis of situations Expert comparison Feedback Assessment Evaluation The Interactive Examination Assessmentwith a scoring rubric

19. Part of Scoring Rubric

20. Claims about rubrics (1) Makes criteria explicit and thus specifies what is to be assessed. Irrelevant variables are excluded.

21. Claims about rubrics (2) Makes criteria explicit and thus specifies what is to be assessed. Assessments become more consistent and reliable.

22. Claims about rubrics (3) Makes criteria explicit and thus specifies what is to be assessed. Promote learning and self-assessment.

23. Self-assessment Analysis of situations Expert comparison Feedback Assessment Evaluation The Interactive Examination Feedbackfor each criteria in the scoring rubric

24. Quality estimation: Method Sample:a cohort of student teachers during their first semester at the School of Teacher Education,Malmo University n 157; 60% female & 40% male

25. ”The Wheel of Competency Assessment” From Baartman et al. (2006)

26. Quality estimation: Data Summativepurpose Students’ backgrounds Students’ examination results(from 2 independent assessors)

27. Quality estimation: Data & Analyses Summativepurpose Students’ backgrounds Regression analysis Students’ examination results(from 2 independent assessors)

28. Students’ examination results(from 2 independent assessors) Cronbach’s alpha Generalizability theory Interrater reliability Quality estimation: Data & Analyses Summativepurpose Students’ backgrounds

29. Quality estimation: Data Formativepurpose Students’ self-assessments Students’ examination results(from 2 independent assessors) Evaluation questionnaire with(mostly) Likert questions and free text fields

30. Quality estimation: Results Summativepurpose Comparability Both raters showed an acceptable level of consistency in their scoring = .793 and .833

31. Quality estimation: Results Summativepurpose Reproducibility of decisions The three movies required for each student was enough to get generalizable results  = .929 and .822

32. Quality estimation: Results Summativepurpose Fairness The assessment seems to be reflecting the knowledge at stake % var. = 68.4 and 60.7

33. Quality estimation: Results Summativepurpose Fairness No effects of gender, ethnical backgrounds or computer skills. Language was not a problem

34. Quality estimation: Results Formativepurpose The students perceive that: - the examinationisan acceptable, authentic and meaningful form of assessment,well aligned with course instruction,- they have to use analytical skills, as opposed to remembrance, in order to succeed.

35. Quality estimation: Results Formativepurpose Transparency The students have felt some difficulty in interpreting the criteria in the scoring rubric and also in knowing what is expected of them to get a passing grade Median = 6 (of 9)

36. Quality estimation: Results Formativepurpose Transparency The students have felt some difficulty in interpreting the criteria in the scoring rubric and also in knowing what is expected of them to get a passing grade However: Those using the rubric had significantly higher scores… ES = 1.3, p<.001

37. Main points: Performance assessments CAN be used for summative purposes…

38. Main points: Performance assessments CAN be used for summative purposes …but they should be supported by scoring rubrics.

39. Main points: Performance assessments CAN be used for summative purposes… The quality of summative and formative parts CAN be estimated separately and by (partly) different criteria…

40. Main points: Performance assessments CAN be used for summative purposes …and a comprehensive framework of quality criteria might facilitate this process. The quality of summative and formative partsCAN be estimated separately and by (partly) different criteria…

41. Main points: Performance assessments CAN be used for summative purposes …where a comprehensive framework of quality criteria CAN facilitate this process Also: For formative parts the student perspective is thought to be imperative! The quality of summative and formative parts CAN be estimated separately and by (partly) different criteria…

42. Thank you for your attention! E-mail: anders.jonsson@mah.se