Virtual presentation . Session: e-Learning, Training, Evaluation and Assessment

1. Virtual presentation. Session: e-Learning, Training, Evaluation andAssessment TEACHERS’ CORRECTION ACTIONS OF FREE WORDED EXERCISE SOLUTIONS FOR MODELING A TEACHER ASSISTING FEEDBACK TOOL Dr. Sonja Ruda Chemnitz University of Technology, Germany Communication on Technology

2. 1. INTRODUCTION • Exercises • 1) preset solution possibilities, e.g. multiple choices • 2) solution text drafted by the students themselves Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

3. 1. INTRODUCTION • Exercises • 1) preset solution possibilities, e.g. multiple choices •  student can only guess the answer • 2) solution text drafted by the students themselves  students entirely have to resort to their own knowledge Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

4. Marking self worded answers solutions requiresof the teachers • special knowledge, • much time and • energy. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

5. Marking self worded answers solutions requiresof the teachers • special knowledge, • much time and • energy. • Automatic correction: • Teaching and learning systems can identify errors simply as formal deviations. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

6. Some problems 1) Ambiguous word combinations: A false solution is though assessed as “correct”. System: What is a ‚turn-taking‘? Student: The previous hearer gives the word to a previous speaker. System: Correct! The previous speaker gives the word to a previous hearer. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

7. Some problems 2) The teacher does not consider correct alternatives, right solutions can nevertheless be marked as “wrong”. System: What is a ‚turn-taking ‘? Student: Change of speakers. System: Unfortunately wrong. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

8. Some problems 3) Student’s answer: “not completely correct”, the corresponding text passages cannot be marked exactly. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

9. Some problems 4) Sample solutions: The teacher has no control whether the student compared the solutions correctly. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

10. Some problems 5) Feedback: “Your answer is unfortunately wrong”. “Congratulations, you get 3 points out of 3!” Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

11. All this demonstrates • technical shortcomings • enormous implementation expenditure of a differentiated feedback • costs Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

12. Teacher assisting feedback tool • for marking free worded exercise solutions • Combination: advantages of free worded exercise solutions, advantages processed by the computer • Analysis: • Teachers’ correction actions and their knowledge of free worded exercise solutions Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

13. Some questions • Which links are there between tasks and • solution text? • How is the comparison between the lecturer’s and student’s solution text made? • How are mistakes recognized, analyzed, corrected and assessed? • Which actions are performed by all lecturers? • Which actions could be performed by the computer? Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

14. 2. METHODOLOGY What do teachers, when correcting free worded exercise solutions? Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

15. Text corpus: • offline university examination questions • self worded answers with at least one word, • communicative-pragmatic approach • (Brinker, 2005, p. 18) • ’think aloud method’ • (van Someren, Barnard & Sandberg, 1994) • ’concurrent record analysis’ and ’focused interview’ (Karbach & Linster, 1990, pp. 84 and 87) Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

16. 14 lecturers produced 15 verbal records • analytical transcripts • each revised between three and seven papers • 71 papers from 13 special fields • with 488 corrected exercises • including subtasks • The tape recordings: from about nine minutes up to almost two hours. The average recording amounted to one hour. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

17. Pragmalinguistic analysis ‘Speech Act Theory’ Austin, J. L. (1962).How to Do Things with Words. Cambridge, Massachusetts: Harvard University Press. Searle, J. R. (1969).Speech Acts. An Essay in the Philosophy of Language. Cambridge: Cambridge University Press.  Brinker, Holly, von Polenz Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

18. 3. SOME RESULTS OF THE PRAGMALINGUISTIC ANALYSIS Link between task and solution text The lecturer INSTRUCTS the students to explain certain contents. The lecturer opens a knowledge frame for the student. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

19. 3. SOME RESULTS OF THE PRAGMALINGUISTIC ANALYSIS Link between task and solution text The lecturer INSTRUCTS the students to explain certain contents. The lecturer opens a knowledge frame for the student. A task definition is limited by restrictive attributes, e.g.: Describe briefly ... Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

20. 3. SOME RESULTS OF THE PRAGMALINGUISTIC ANALYSIS Link between task and solution text The lecturer INSTRUCTS the students to explain certain contents. The lecturer opens a knowledge frame for the student. A task definition is limited by restrictive attributes, e.g.: Describe briefly ... Task definitions are dominated by action predicates, e.g.: mark. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

21. 3. SOME RESULTS OF THE PRAGMALINGUISTIC ANALYSIS Link between task and solution text The lecturer INSTRUCTS the students to explain certain contents. The lecturer opens a knowledge frame for the student. A task definition is limited by restrictive attributes, e.g.: Describe briefly ... Task definitions are dominated by action predicates, e.g.: mark. Topic development patterns → “Help” Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

22. Teachers’ correction actions of free worded exercise solutions Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

23. Teachers’ correction actions of free worded exercise solutions 1) Grasping the complete student’s solution attempt or filtering the propositions to be dealt with 2) Comparison with the expected solution 3) Assessment of the student’s solution attempt with regard to the definition of the tasks 4) Total assessment of student’s solution and 5) Total assessment of all present solutions of a student. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

24. Potential correction actions • writing down expected solutions, • compiling a criteria grid and • determining the approach • – paper by paper or exercise by exercise Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

25. Individual assessment of a student’s exercise solution Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

26. Individual assessment of a student’s exercise solution 1) Envisioning of the exercise definition 2) Envisioning of the connection between exercise definition and solution 3) Expressing the first impression 4) Grasping the student’s solution attempts 5) Comparison with the expected solution 6) Assessing individual parts of the student’s solution attempt and 7) Correcting. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

27. Problematic and vague solution attempts Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

28. Problematic and vague solution attempts • problem solving strategies/(speech) acts, e.g.: They • PARAPHRASE from the student’s point of view, • INTERPRET, • SUMMARIZE, • EXPLAIN, • SPECIFY own solution texts, • ELUCIDATE the task definition, and • search for parts of the solution text which may still be considered as correct, • use aids like professional literature, • ASK colleagues/superiors and • MAY MAKE COMPROMISES. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

29. 4. TEACHER ASSISTING FEEDBACK TOOL • oriented towards free worded exercise solutions • standardized • brief • without grafics • „Borland Delphi 5.0“ • Prototype Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

30. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

31. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

32. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

33. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

34. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

35. Rule Assistant (Boole’s operators) AND (UND): Which words/phrases must be includedin the solution? Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

36. OR (ODER) and PREFERRED (BEVorzugt): Please enter acceptable alternatives to the required words/phrases. If you prefer certain alternatives, please tick them. (BEV-Operator) Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

37. OR POSSIBLE (ODEREVTL): Now enter solutions that you would accept even though they are not completely correct. AND POSSIBLE (UNDEVTL): Now enter solution text parts that may be included additionally. POINT (PLUS): When would a solution text be particularly good? Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

38. NOT (NICHT): Which words/phrases must not be included in the solution? What would not be completely wrong but can nevertheless not be accepted by you? What would be a serious mistake? Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

39. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

40. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

41. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

42. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

43. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

44. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

45. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

46. 5. CONCLUSION • Feedback tool • supports teachers in their correction work • by making arrangements with them • regarding the solution texts and assessments • which requires the acquisition of certain rules and initial training • An automatic correction of all solution • texts cannot and should not be expected! Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

47. Current works the use of a semantic word net in order to e.g. capture synonyms and statistical data. Chemnitz University of Technology – Communication on Technology – Dr. Sonja Ruda

48. Thank you for your attention! Contact: Dr. Sonja Ruda Chemnitz University of Technology Communication on Technology D-09107 Chemnitz sonja.ruda@phil.tu-chemnitz.de