The use of CAA for formative and summative assessment: student views and outcomes.

1. The use of CAA for formative and summative assessment: student views and outcomes. • Sandra Pattinson • Animal Production and Science Group • Harper Adams University College • Newport • Shropshire • TF10 8NB. • e-mail: sepattinson@harper-adams.ac.uk

2. E.g. HND Agrifood Marketing and Business Studies + HND Countryside Management. Initial failure rate 2001 - 02 25 % 2002 - 03 28 % 1.0 Introduction 1.1 Farm Animal Production Compulsory core module (Level 1). Non-agricultural students - high failure rate.

3. 1.0 Introduction 1.2 Aim of study To examine the effects of the introduction of formative CAA on the results for the module. The module is taught over two semesters and results are presented for the first semester only.

4. 2.0 Methods 26 students on Farm Animal Production module. 2.1 Odd semester: Subjects taught: beef, sheep, poultry. Methods: lectures, tutorials, farm classes. Optional: self-assessment tutorials using QuestionMark Perception.

5. 2.0 Methods 2.2 Self-assessment tests Each test consisted of 10 multiple-choice questions selected at random from database. Size of database Beef production 67 questions Sheep production 58 questions Feedback provided for each question.

6. 2.0 Methods 2.3 Summative assessment 29 sheep production questions 29 beef production questions 40 poultry production questions 2 general questions Questions not randomly selected. N.B. Negative marking used in tutorials and in summative examination

7. 2.0 Methods 2.4 Questionnaire Presented to students at the start of the subsequent semester (prior to publication of exam marks). Aim - to assess the students’ views on the use of the self-assessment tutorials.

8. 3.0 Results 3.1 Examination results 2003 - 04 (tutorials available) mean mark 41.5 % pass rate 46 % 2002 - 03 (tutorials not available) mean mark 34.8 % pass rate 25 %

9. 3.0 Results 3.1 Examination results Use of tutorials Number of Mean students mark 0 times 10 (38%) 32.3 1 - 9 times 9 (35%) 37.6 10+ times 7 (27%) 59.6

10. 3.0 Results 3.1 Examination results Correlations Use of tutorials v. exam mark Sheep rs = 0.59 P < 0.01 Beef rs = 0.69 P < 0.01 Mark v. poultry mark Beef r= 0.52 P < 0.01 Sheep r = 0.36 P < 0.1

11. 3.0 Results 3.2 Questionnaire 3.2.1 How often did you use the self-assessment tests? Very often, often, occasionally, rarely, never.

12. 3.2.2 As a method of learning I enjoyed the tests

13. 3.2.3 The wording of the questions was unclear

14. 3.2.4 The assessment questions were closely related to the content of the module

15. 3.2.5 I used my lecture notes whilst completing the self-assessment tests

16. 3.2.6 The self-assessment tests helped me to understand the lecture notes

17. 3.2.7 The marks helped me to assess how I was doing on the module

18. 3.2.8 The feedback given if you get the question wrong is useful

19. 3.2.9 Doing the tests made it easier to study for the exams

20. 3.0 Results 3.2 Questionnaire 3.2.10 Free response A) Why did you use/not use the self-assessment tests Not used Forgot Used To see how much they knew To practice for exams

21. 3.2.10 Free response B) What were the three best things about the self-assessment tests? Easy to use Detailed answers (feedback) Similar format to exam C) Give three suggested improvements. Remove negative marking Remote access Provide the correct answer in the feedback More questions and more feedback

22. 3.2.10 Free response D) Any other comments They were a good way of learning and very useful. It was very helpful. Thank you.

23. 4.0 Conclusions Students who sat optional practice tests had higher marks in subsequent summative tests. This is in agreement with other authors e.g. Sly and Rennie (1999), Charman (1999) Students who scored highest on the beef and sheep sections of the summative assessment also had the highest marks in the poultry section.

24. 4.0 Conclusions Questionnaire responses indicate that for most students the tutorials provided a positive learning experience. Some students requested that the correct answer should be provided if the question was answered incorrectly Deep v. surface approach to learning

25. 5.0 Future direction 5.1 Current semester Pig and dairy production self-assessment tutorials Revision test at start of lecture 5.2 Questionnaire Questions relating to Feedback Test structure

26. Question: The feedback should state which was the correct answer.

27. 5.0 Future direction 5.3 Question analysis - to identify problem areas of the module or poorly written questions 5.4 Remote access