Lynn Berry, Garry Collins, Peter Copeman Rowena Harper, Linda Li, Sue Prentice

1. Counting what counts in individual consultations: developing and applying a set of evaluation instruments Lynn Berry, Garry Collins, Peter Copeman Rowena Harper, Linda Li, Sue Prentice

2. Background of our study • Individual Consultations (ICs) are still core ASC business • Increasing demand for (ICs) at UC but decreasing capacity to meet this demand • Need to ensure quality & effectiveness of ICs • Inspiration from Stevenson & Kokkinn’s (2009) AALL paper on evaluating ICs

3. Purpose • Develop, trial and analyse the effectiveness of a set of instruments for evaluating individual consultations • Secondary purpose – to enhance our teaching and student learning in ICs

4. The research design

5. Methodology • Instruments: • Peer-observation (PO) tool • Self-reflection (SR) tool • Student questionnaire (Q-tool) • Participants: • 7 ASC staff; • Students • Time:4 week period - Semester 2, 2011

6. The research process Shared data Anonymous student feedback PO checklist de-identified & stripped of comments Responses to request for contributions about SR process (non-identifiable) IC Student Eval Pre-PO PO Post PO SR

7. The peer observations: a collaborative model A structured approach Establishing a set of principles and criteria a collaboratively developed peer-observation tool, scheduled pre- and post-observation discussions integrated with self-reflection inclusive, staff active involvement, non- judgmental, a collaborative process

8. Peer-observations: positive outcomes • Enthusiastic staff participation • Sharing of approaches to good practice and strategies • Strengthening the collegiality of the team • Benefits of learning from each other

9. An example of staff comment • Thank you for the opportunity to observe your consultation. In spite of my observer status, I was actively involved in the learning and teaching process as I observed its development as the consultation unfolded. The consultation made an important contribution to my learning and professional development by stimulating reflection on my own teaching.

10. Peer-observations: challenges and constraints • Some initial staff reservation • The limiting effect of the peer-observation tool • The constraint of time

11. The self-reflection process: reflective dialogues with self and others • Scheduled self-reflection time • A guided approach with a self-reflection tool aligned • with the peer observation tool • Linked to the peer observations through • post-observation discussions with colleagues • Team debrief session to reflect on the whole process

12. Self-reflections: positive outcomes • The self-reflection tool served as a reminder or structural tool • Self-reflection was stimulated by observing another and through peer-exchanges. • Reflection as a process with ongoing informal discussions

13. An example of staff self-reflection One thing I would remind myself in future sessions is not to jump to conclusion too quickly… It was only after listening to his explanations that I realised I made an assumption too quickly without getting the real picture! This reminds me of the importance of listening to the student, not just at the beginning of the session, but throughout the session, and the importance of holding back my own judgment and letting the student make his/her own decision.

14. Self-reflections: challenges and constraints • Some staff criticism of the self-reflection checklist • Need to modify the tool • ‘Self-reflection can’t take place in a vacuum’

15. An example of staff self-reflection By and large, my reflection took place while I was observing other staff. Upon seeing a particular approach, my instinct was to consider how I would approach the situation if I was the adviser. I learned a lot of useful strategies this way…I’m not sure I can apply ‘reflection’ to my own teaching in a sort of vacuum. When I try to do this, I am only evaluating how I fell short of my own tried and tested approaches. I feel it’s difficult to improve this way. I feel I need to see alternative approaches in practice so I have points for comparison.

16. The student questionnaire: focus on student learning • An improved version: more than quality assurance • Encourage the students to reflect more explicitly on their learning

17. The student questionnaire: focus on student learning Evaluation Of teaching Evaluation of learning

18. The student questionnaire: positive outcomes • A constructivist view of learning • Student expectation and student learning • Focus on most needed aspects of learning

19. Student comments • I am very happy with consultation. It gave me a lot of confidence. • Thank you for giving me the chance to correct my assignment. • I am happy with the ASC service and it helped me to improve my knowledge and my result. • Thank you for all your help it has greatly advanced my grades and understanding!!

20. The student questionnaire:challenges and constraints • Focus on what rather than how and how well • Unlinked to peer-observations and self-reflections

21. Have we achieved our goals?

22. Have we achieved our goals? • We successfully developed,trialled and analysed the effectiveness of Q-tool, PO & SR • The Q-tool contributed to greater student awareness of their own learning, but did not add significantly to feedback on our teaching • PO and SR contributed to better shared understanding of ICs and improved IC teaching practices

23. What have we learnt from this project? • Better , shared teaching strategies for ICs • We learnt more from observation than being observed or reflection • Necessity to link PO and SR to improve IC teaching practices • Using evaluative instruments such as PO and SR requires training and experience

24. What next? • Plan for next round of PO & SR: • Refine the PO, SR & Q-tool instruments • Pilot a focus group of students to elicit more in-depth comments/suggestions on ASC’s ICs • Keep an on-going record of effective IC teaching strategies

25. Acknowledgements We would like to thank our colleagues Emmaline Lear, Sharon Gibson, Jo Caffery, Ros Byrne and all the participating students for their contribution to this project.