Maria João Loureiro

1. The quality of peer assessment in a wiki based online context:a qualitative study in a post-graduation module Maria João Loureiro

2. topics • context

3. Coimbra Oporto Aveiro • context of the study - Portugal

4. context (U. Aveiro) • created in 1973 • 10 to 11.000 students • courses (grad and pos-grad) • teachers education • sciences, math, … • engineering • health • … • member of the European Consortium of Innovative Universities

5. context • PhD program in Multimedia in Education module • Distance Education (DE) • blearning regime • face-to-face and at a distance group work • module aim - develop research competences related with literature reviewing • search, select, systematize, synthesize and literature related with DE; • communication, collaborative work and assessment competences (self- and peer- assessment)

6. Context (literature) • focus on learner self-directed and collaborative learning -> increase attention to alternative/authentic/integrated assessment methods • HE assessment methods are quite narrow – assessment students’ outcomes test based – students are not implicated • peer assessment (PA) can empower students (critical thinking, reflection) • reliability and validity of PA – no consensus • literature on the quality of online (web 2.0) peer assessment scarce Boud and Falchikov (2007), Joordens et al. (2009), Li et al. (2009, 2010), Peng (2010), Rourke et al. (2008), Sluijsmans et al. (2004), Topping (2000, 2008, 2010), Zundert et al. (2010)

7. Context (literature) • PA benefits: • promote higher order thinking, developing students’ critical thinking, communication, lifelong learning, problem solving… • increase the amount of feedback, from teachers and from peers • enhance students’ sense of ownership, responsibility and students’ motivation, since they find it useful, attractive and enjoyable • support active and autonomous learning • can avoid the involvement of free-riders in group work, i.e, facilitate the identification of individual contributions

8. Context (literature) • PA pitfalls: • students’ attitudes toward PA may hinder the above mentioned benefits –students’ unconfident towards PA - judging friends – increase students’ anxiety and resistant toward PA • PA is time-consuming – training, preparation, and monitoring are needed • PA can be subjective – students might be too hard, critical or boasting, or not used to assess their peers • PA advantages increase – formative feedback

9. course organisation and development • 1st week online • Diagnosis of students’ perceptions about how to do literature review and about “Distance Education” (individual). • Familiarisation with the module guide and discussion of the proposed activities, including assessment (individual). • Individual readings of recommended literature and or other materials related with theoretical and methodological issues to be taken into account to do a literature review. • Use of the synthesis sheet – instrument developed to help students to analyse and document their readings (individual).

10. course organisation and development • 1st face-to-face session • Teacher, tutor and students’ presentation and definition of work groups (classroom). • Negotiation of the module activities and schedule and of the assessment framework (classroom). • Session about online literature search tools (made by a library technician). • Presentation of tips on how to do a literature review (teach). • Discussion and syntheses of the students’ readings about how to do a literature review (groups). • Information search (groups).

11. course organisation and development • 2nd weekonline • 1st ind.reflection about the ongoing activities and students’ learning • Definition of the literature review themes related with DE and of the work plan (groups) • Formative assessment of the work plan (teachers) • 3rd and 4th weeksonline • 2nd ind.reflection about the activities and the developed competences • Development of the 1st version of the literature review (gr.) • Formative assessment of the ongoing work (teachers and PA) • Revision of the paper and preparation for its presentation (groups).

12. course organisation and development • 2nd face-to-face session • Presentation, discussion and assessment (teachers and peer summative assessment) of the groups work. • Self and peer assessment of the developed collaborative competences (intra group) and final reflection concerning students own learning during the module (individual) • Module assessment (individual).

13. Methodology • Qualitative approach - study recent phenomena (like the quality of online PA) in their own contexts • Data gathering and analysis techniques: • observation mediated by the wiki (PA was delivered by the students using the wiki) – content analysis (2 researchers, reliability 100%, external validation) • inquiry (a questionnaire was applied to evaluate the module in the final face to face session) – descriptive statistics

15. Examples • negotiated criteria • “(…) the topics and sub-topics that underlie the work are well framed [i.e. the structure is adequate]. Its coherence and rigour are remarkable and the group demonstrates synthesis competences”. • adequacy of the chosen vocabulary • “the paper does not present significant gaps in what concerns cohesion, textual coherence or even orthographic or syntactic mistakes.” • constructive feedback • “the work does not include the research question that guides the development of the work”

18. Final thoughts • PA could have more quality (suggestions, questions, …) • Results • In line with the literature – lack of quality can be explained by some of the pitfalls of PA (students’ attitudes towards PA, uncomfortable, lack of confidence, unreliability + fear of being exposed in the social web – in this case while agreement about publishing PA) • PA needs training

19. Contributions • framework to assess the quality of PA in web 2.0 online contexts, • for some of the criteria, quality PA can be achieved • reports students’ perceptions and attitudes towards the assessment of collaborative work, including PA

20. Thank you for your attention • questions???

21. courses organization Ramos (2004)