Peter Chalk & Dafna Hardbattle CCTM & Write Now CETL Mini-Project

1. The role of personal development planning in improving the reflective writing skills of science and engineering students Peter Chalk & Dafna Hardbattle CCTM & Write Now CETL Mini-Project London Met Learning & Teaching conference July 2007

2. Research Context • At LondonMet, the Personal Development portfolio (PDP) is embedded in the curriculum with PDP related assessment at each level, in ‘core spine’ modules • Certificate Level (1st year) HEO module (‘HE Orientation) • Intermediate Level (2nd year) Employability module • Honours Level (3rd year) Project/ dissertation

3. Research question • Considering the first year, first semester HEO module: • Is there evidence of • Reflective writing in their PDPs? • Improved performance as a consequence? • Or is there a problem? • If so, can we solve it?

4. Terry King, Development of Student Skills in Reflective Writing, ICED 2002 Why Science Students? “In a culture of increasing emphasis on critical reflection by students, it is not surprising that the external examiners… have looked for evidence of this in student dissertations and other assessments. Towards the end of 2000 it became apparent that the incidence and quality of reflective writing by computing students across all programmes needed to be improved.”

5. Comments by Science staff at LondonMet in 2007 (straw poll) • “We certainly find that students need time to develop this ability… . They tend to produce very factual pieces of work and lack the ability to weigh up arguments… Workshops would certainly help.” • “…weaker students struggle with the level of critical/analytical report writing that they should be achieving.” • “Science students find it hard to write reflectively… PDP writing helps them develop these skills.”

6. Background: the work of Jenny Moon • Moon (2004: 75) cites the evaluative tool of Hatton and Smith (1995) as a useful method for categorising levels of reflection: • (Description) • Descriptive reflection • Dialogic reflection • Critical reflection

7. Moon’s categories (qualitative) • 1. “Descriptive writing: This is a description of events or literature reports. There is no discussion beyond description. • 2.Descriptive reflection: … some evidence of deeper consideration in relatively descriptive language. There is no real evidence of the notion of alternative viewpoints in use.”

8. 3. “Dialogic reflection:… a ‘stepping back’ from the events. There is consideration of the qualities of judgements and of possible alternatives for explaining and hypothesising… analytical or integrative, linking factors and perspectives. • 4.Critical reflection: … aware that the same actions and events may be seen in different contexts with different explanations associated with the contexts.”

9. Research methodology (Feb-Jul 07) • Qualitative and quantitative data analysis • Data from 0506 and 0607 Spring semester first year module Computing Abstraction & Skills • Collect (relevant) evidence from PDPs • Categorise writing according to Moon levels 1-4 and agree ratings • Identify examples of high level reflective writing & tasks that encouraged it • Interview students & analyse data against performance (in progress)

10. Rating Tool (quantitative) • Moon’s Categories: • Level 1 Descriptive writing • Level 2 Descriptive reflection • Level 3 Dialogic reflection • Level 4 Critical reflection • Jenny MoonReflection in Higher Education Learning,LTSN, 2001

11. Level 1 (?) illustrative example (‘descriptive’)

12. Level 2.5 (?) illustration (‘stepping back’?)

13. Example A (how would you rate it?)

14. Example B (how would you rate it?)

15. Example C (how would you rate it?)

16. Example D (how would you rate it?)

18. Preliminary results – 0506 cohort • Answers to questions on literature research, personal skills and group work in PDPs • 17 students, 1-5 extracts of text from each, total 52 extracts analysed • Ratings independently agreed for 16 students (Rating Tool validated? Discrepancy resolved; correlation 0.3 not significant?) • Mostly unreflective, 83% rated 1 or 1.5 • Some evidence of reflection at level 2 / 2.5

19. Performance & reflective writing – do they correlate? • The left-hand column shows student module result average across 3 semesters since start in 0506b • There is another weak positive correlation with DH’s RW grading • However, to show that improving reflective writing improves performance would require a more interventionist research study…

20. Evidence from use of inter-rating tool seems to confirm other findings, e.g. • “Student writing can be used as evidence for the presence or absence of reflective thinking… [but]Identifying textual elements within journals and allocating them to the finer levels of reflection within a more complex model of reflective thinking was, however, more problematic and considerably less reliable.” (Wong 95)

21. Preliminary conclusions • Most science and engineering student's writing in the first semester is at the level of descriptive, with some reflective writing when discussing group work • Many PDPs paraphrased example ‘reflective’ text (e.g. ‘did you feel shy when you started working in a group?’) • Some confirmation from a similar module’s PDPs in Science • Is the problem subject or task related?

22. Setting the task • The PDPs for another HEO module in the Business School were also analysed • Tasks were specifically situated within a reflective writing framework: • ‘What, So what, Now what?’ • Lectures, tasks, draft feedback & marking criteria all related to reflection/ critical thinking • Evidence of writing to levels 3 and 4

23. Teaching reflection • In another Computing HEO module, reflective writing is taught explicitly with reference to Kolb, Ramsden, Moon and others (Wilson-Medhurst, 2005: 92) • Task requires students to ‘relate the material on the module to their own experience’ • Formative feedback and group work assessment are two features that ‘work’

24. Guided reflection • Moon (2004: 70) cites Johns (1994) work on ‘guided reflection’ questions for training professionals - commenting that additional questions are required for ‘critical reflection’ in learning • can we adapt it to teaching and learning in science & engineering? • what questions might best guide reflection?

25. What task/ experience/ support leads to ‘better’ reflective writing? • Be able to answer questions like: • What did I do? (describe) • Why did I do it? (analyse) • What if I did it another way? (evaluate) • How would I act again in another context? (‘How did it affect my assumptions and beliefs?’ – Moon: 70)? • Group work tasks and questions seemed to extract ‘best’ reflection in first semester, at least – what other tasks might do this? Subject specific?

26. Possible research outcomes • Teaching and Learning tasks with questions (not examples?) that promote reflective writing by science and engineering students (use Write Now mentors?) • Guide to lecturers: how to categorise and assess levels of reflective writing (Moon inter-rater tool?) • Designing a reusable learning object incorporating these findings, similar to RLOs at www.rlo-cetl.ac.ukor www.bournemouth.ac.uk/study_support/reflective_writing.html • Rating ‘answers’: A=1.5, B=2, C=2.5, D=1 (??)

27. References • Hatton, N, and Smith, D (1995) ‘Reflection in teacher education – towards definition and implementation’, Teaching and Teacher Education, 11, (1), pp 33-49 • Johns C (1994) ‘Nuances of reflection’, Journal of Clinical Nursing, 3, pp 71-75 • King T (2002) ‘Development of Student Skills in Reflective Writing’, ICED [accessed 25/9/06]//www.csd.uwa.edu.au/iced2002/publication/Terry_King.pdf • Moon J (2004) Reflection in Learning and Professional Development: Theory and Practice, Kogan Page, London • Wilson-Medhurst S (2005) ‘Supporting Student Development Using Reflective Wriitng’, Investigations, 2, (2), pp 89-93 • Wong F et al (1995) ‘Assessing the level of student reflection from reflective journals’, Journal of Advanced Nursing, 22, (1), pp 48-57 • With thanks to Debbie Holley and Sarah Atchia

28. This project is supported by