Best Practices in Classroom Peer Review Edward F. Gehringer Department of Computer Science

1. Best Practices in Classroom Peer Review Edward F. Gehringer Department of Computer Science North Carolina State University The Expertiza project has been funded by the National Science FoundationPlease visit our Web site: http://tinyurl.com/expertiza-site

2. Credits … • Arlene Russell, Calibrated Peer Review • Chris Schunn, SWoRD • Steve Joordens & Dwayne Paré, Peer Scholar • Eric Ford & DmytroBabik, Mobius SLIP • Helen Hu & David McNaughton, uJudge Gehringer, Classroom peer review efg@ncsu.edu

3. What’s good about peer review? F2F vs. online peer review Rubrics Rating vs. ranking Formative vs. summative Quality control Who reviews whom? Online apps for peer review Outline Gehringer, Classroom peer review efg@ncsu.edu`

4. Advantages of peer review? Gehringer, Classroom peer review efg@ncsu.edu

5. Some advantages of peer review • Feedback is • more extensive • quicker • scalable • Can’t blame the reader! • Forces students to think metacognitively Gehringer, Classroom peer review efg@ncsu.edu

6. Face-to-face vs. online peer review Gehringer, Classroom peer review efg@ncsu.edu

7. Why face-to-face peer review? • Easier to set up • Communicate more interactively • Exchange non-verbal cues • Instructor can intervene Gehringer, Classroom peer review efg@ncsu.edu

8. Why online peer review? • Doesn’t consume class time • Read/write more reflectively • Easier to get multiple reviews • Easier for author to refer back • Can be used summatively • Can be perused in deciding on grade • Rubric can perhaps be more detailed Gehringer, Classroom peer review efg@ncsu.edu

9. Rubrics • Why use a rubric? • Tell students what to look for • “Fairness” in assessment • Students can helpcreate the rubric • How detailed? Gehringer, Classroom peer review efg@ncsu.edu

10. Rubric advice

11. Rating vs. ranking • Should students rate others’ work on a Likert scale, • or rank students against each other? • Rating • Easier to rate than rank using a rubric • Can give 2 students the same rating • Ranking • May not be compatible with F2F review • More robust when reviewers are not experts • Can use a slider to show nearness Gehringer, Classroom peer review efg@ncsu.edu

12. Mobius SLIP’s approach to ranking Gehringer, Classroom peer review efg@ncsu.edu

13. Formative vs. summative peer review • Formative—text feedback • Summative—Likert scale • Should peer review be used summatively? Gehringer, Classroom peer review efg@ncsu.edu

14. Quality control • You can’t take review quality for granted. • Approaches • Metareviewing • Calibration • Reputation system Gehringer, Classroom peer review efg@ncsu.edu

15. Metareviewing • “Review the reviewer” “Rate the rater” • Who performs metareviews? 3 choices • Author? • Instructor? • 3rd party? • Can we automatethe process? Gehringer, Classroom peer review efg@ncsu.edu

16. Calibration • Basic idea: Training course for reviewers • How they do  how much credence they get • Before students review peers, they get 3 works to review • 1 exemplary • Their agreement with instructor  Reviewer Competency Index • Others have known defects Gehringer, Classroom peer review efg@ncsu.edu

17. Reputation algorithm s1 gets the same scores from reviewers in both situations. Should it get the same grade? Gehringer, Classroom peer review efg@ncsu.edu Gehringer, Classroom peer review efg@ncsu.edu

18. Reputation algorithm, cont. r2 and r3 agree with their co-reviewers … Gehringer, Classroom peer review efg@ncsu.edu

19. Reputation algorithm, cont. r2 and r3 agree with their co-reviewers … while r1 gives higher scores. So, s1’s grade may be inflated. Gehringer, Classroom peer review efg@ncsu.edu

20. Reputation algorithm, cont. In this situation, r1 agrees with his co-reviewers, while r2 and r3 give lower scores. So in this case, s1 was reviewed by “harder” graders, and thus deserves a higher grade. Gehringer, Classroom peer review efg@ncsu.edu

21. Reputation systems—how reliable? • Two studies on Coursera MOOC [2013] • Piech et al.: ≥ 26% of grades ± 5% from “ground truth.” • Kulkarni et al.: 40% of grades off by 1 letter grade! • But … • no calibration, metareviewing • this was, after all, a MOOC Gehringer, Classroom peer review efg@ncsu.edu

22. Who reviews whom? • Simplest: Each student reviews k other students • Reviewing in groups—case study, etc. • Individuals review teams • Dynamic assignment, to make sure all get reviewed Gehringer, Classroom peer review efg@ncsu.edu

23. The PR app landscape • Most widely used: CPR • Sharable assignments for many disciplines • But, you probably want to adapt. Gehringer, Classroom peer review efg@ncsu.edu

24. SWoRD • Perhaps the most-researched system …from Pitt’s Learning Resource Development Ctr. Gehringer, Classroom peer review efg@ncsu.edu

26. Peer Scholar • Came from U. of Toronto • Now sold by Pearson in Canada • Free (for now) in the US • Supports (& recommends) revision and resubmission Gehringer, Classroom peer review efg@ncsu.edu

27. Mobius SLIP • Origin in case-study courses • Based on ranking • “Double loop” Gehringer, Classroom peer review efg@ncsu.edu

28. Expertiza • “Reusable learning objectsthrough peer review” • Supports signing up for topics/parts of a project • Students (or instructor) form teams • Individuals review teams • Teammates review each other Gehringer, Classroom peer review efg@ncsu.edu

29. Signup sheet Gehringer, Classroom peer review efg@ncsu.edu

30. Viewing results Gehringer, Classroom peer review efg@ncsu.edu

31. Reasons for doing peer review F2F vs. online peer review Rubrics are important Rating vs. ranking Formative vs. summative Quality control Who reviews whom? Online apps for peer review Summary Gehringer, Classroom peer review efg@ncsu.edu`