Expertiza : Peer Review and Social Networking for Co-Producing Learning Objects

1. Expertiza: Peer Review and Social Networking for Co-Producing Learning Objects Edward F. Gehringer Dept. of Computer Science North Carolina State University Supported by NSF DUE under a CCLI grant NCSU Faculty Center for Teaching and LearningNCSU LITRE (Learning in a Technology-Rich Env.) Center for Advanced Computing and Communication Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

2. After this talk, you will … Have a glimpse of the wide variety of ways in which peer feedback can be used, Know of various software tools for gathering peer feedback, and Understand how students can assist in producing course materials using peer review. Learning Goals Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

3. Expertiza Rationale Co-Production Experiments and Results Brief Expertiza Demo Handling Team Assignments New Features Coming to Expertiza Outline Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

4. There are many Web-based systems for peer review of student writing. Expertiza takes this one step further—enabling co-production of the course! Advantages … Gets students working together to improve others’ learning experiences, Helps them learn, by performing tasks that are similar to real-world responsibilities, Gives them experience in writing their ideas up for an audience of their peers, Allows each cohort to “stand on the shoulders” of students in earlier classes. Introduction Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

5. For students to demonstrate mastery of the subject. Every student does the same thing— redundant effort. Work is graded and thrown away, never benefiting anyone but the student who did it. In Expertiza, students (or teams) sign up for different work Now the best work can be reused, to help others learn. Homework, traditionally … Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

6. Expertiza discourages plagiarism Students are not doing the same work. Their work must go through multiple review cycles Expertiza better utilizes scarce resources Students do most of the reviewing. Instructor/TA spend less time grading. With Expertiza, large classes are an advantage! Major advantages Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

7. Expertiza Rationale Co-Production Experiments and Results Brief Expertiza Demo Handling Team Assignments New Features Coming to Expertiza Outline Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

8. Co-Production Experiments and Results Exercises for a Textbook Active-Learning Exercises for a Course Assessment of Wiki Contributions Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

9. Results: Material for a Textbook Our class Master’s level course CS & ECE Substantial DE enrollment In Fall 2005, we used a new OOD text for the first time, Dale Skrien’s, An Introduction to Object-Oriented Design Using Java Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

10. Improve an explanation of a topic covered in the text. Create a new example of a concept covered in the text. Write a new exercise for a chapter in the text. All students did not do these exercises in the same order. Three Homework Assignments Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

11. Learning and Enjoyment • 17 student submissions selected for text! Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

12. Last spring, I had my CSC 216 students create active-learning exercises. They worked in teams. They created two exercises during the semester. One of the exercises was used later in the class In the summer, 4 student-generated AL exercises were used in class. Results: Active-Learning Exercises Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

13. “I learned a lot from doing active-learning exercises in class.” Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

14. “I enjoyed doing the AL exercises” Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

15. “I enjoyed devising an AL exercise” Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

16. Comparing Instructor- & Student-Generated Exercises No significant difference Significant at 99% level Significant at 90% level Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

17. Results: Wiki assessment We studied ten NCSU classes that used wikis in 2007-08. CSC 216, Programming Concepts—Java CSC/ECE 506, Architecture of Parallel Computers CSC/ECE 517, Object-Oriented Languages & Systems COM 598W, Gaming and Social Networks ECI 306, Middle Years Reading ECI 521, Teaching Literature for Young Adults ECI 525, Contemp. Approaches in Teaching Social Studies HI 216, Latin America since 1826. HI 453/553, U.S. and Latin America. TOX 415, Environmental Toxicology & Chemistry Three classes used peer assessment; seven classes used “expert” assessment. 40 respondents 62 respondents Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

18. Peer Review vs. Other Assessment Classes that used peer assessment are shown in gold. CSC 216, Programming Concepts—Java CSC/ECE 506, Architecture of Parallel Computers CSC/ECE 517, Object-Oriented Languages & Systems COM 598W, Gaming and Social Networks ECI 306, Middle Years Reading ECI 521, Teaching Literature for Young Adults ECI 525, Contemp. Approaches in Teaching Social Studies HI 216, Latin America since 1826. HI 453/553, U.S. and Latin America. TOX 415, Environmental Toxicology & Chemistry We compared students’ reactions to wiki assignments and assessment. Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

19. “The experience of using a wiki helped me to think critically about the subject matter of this sassignment.” “Using a wiki made it easy to collaborate with other students.” “I received adequate feedback on the quality of my work.” “The feedback I received helped me to improve my work.” All questions rated on a scale of 1=strongly agree to 5=strongly disagree. Comparing Peer Review with Expert Review 95% 95% Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

20. Expertiza Rationale Teaching with Wikis Brief Expertiza Demo Handling Team Assignments New Features Coming to Expertiza Outline Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

21. The Review Process: How It Begins One or more times during the semester, a peer-reviewed project is assigned Students select a topic from a list. Several students are allowed to select the same topic, But the number of slots is limited. Students are usually allowed to work in teams Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

22. Step 1: Signing up for a Topic Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

23. After the Initial Review • Resubmission phase. 2–7 days to revise work in response to reviewer comments. • Final review. 3–7 days to make final comments and assign scores. • Metareview phase. Students review each other’s reviews. Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

24. Alternate Tool: Calibrated Peer Review cpr.molsci.ucla.edu ● For review of writing ● Well developed assgts. for many fields Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

25. Expertiza Rationale Co-Production Experiments and Results Brief Expertiza Demo Handling Team Assignments New Features Coming to Expertiza Outline Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

26. Teamwork mimics the work environment Teaming skills are important to success Teams can tackle larger projects, which diminishes the effort of integration Why team assignments? Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

27. All team members have access to a common submission area. Any member can revise the submission. Individuals review teams. This simplifies reviewing, and provides for more reviews. Teammates assess each other’s contribution to the team. This score can be factored into the student’s grade. How does Expertiza support teams? Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

28. Team formation By instructor, from the user interface By instructor, by importing a file By students, by issuing invitations Reviewing Feedback from author to reviewer Teammate review "Virtual" team demo Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

29. Let’s log in as an instructor … The login screen Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

30. Setting up a team assignment Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

31. Let’s create some teams … Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

32. In the beginning, there are no teams Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

33. But let’s create one … Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

34. Team created; let’s add members Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

35. 1st member is added; now for the 2nd … Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

36. Team now has two members Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

37. Team formation By instructor, from the user interface By instructor, by importing a file By students, by issuing invitations Reviewing Feedback from author to reviewer Teammate review "Virtual" team demo Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

38. Importing teams Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

39. Here’s the file that’ll be imported Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

40. Here’s how we import it Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

41. Now there are five teams … Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

42. Team formation By instructor, from the user interface By instructor, by importing a file By students, by issuing invitations Reviewing Feedback from author to reviewer Teammate review "Virtual" team demo Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

43. Students can create teams too … Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

44. Click on this assignment, Team test 2 Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

45. Now, click on “Your team” Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

46. This user needs to set up a team Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

47. Let’s create a team Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

48. Invite another member to join … Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

49. Invitation has been sent Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza

50. Now the invited user logs in Apr. 22, 2009 Ed Gehringer efg@ncsu.edu http://research.csc.ncsu.edu/efg/expertiza