Constructivist Learning with Participatory Examinations

1. Constructivist Learningwith Participatory Examinations Dezhi Wu, Michael Bieber, S. Roxanne Hiltz Information Systems Department College of Computing Sciences New Jersey Institute of Technology Hyo-Joo Han Information Systems Department College of Information Technology Georgia Southern University 1

2. Outline • Motivation • Participatory Exam approach • A bit of theory • Experimental results • Interesting issues 2

3. Motivation • To increase learning of course content • Learning through active engagement • involve students as active participants • with the full exam life-cycle • through peer evaluation • Minimize overhead for instructors 3

4. Outline • Motivation • Participatory Exam approach • A bit of theory • Experimental results • Interesting issues 4

5. All entries posted on-line PE Process • Each student creates 2 exam problems • Instructor edits the problems if necessary • Each student solves 2 problems • Students evaluate (grade) the solutions to the problems they authored, writing detailed justifications • Other students evaluate each problem a second time • Instructor gives a final grade • optional: Students can dispute their solution’s grade, by evaluating it themselves and writing detailed justifications • Instructor resolves the dispute 5

6. Screen Shot WebBoard System 6

7. Instructor Control Process Student Learning Process Course Design Process Flow: Learning from doing the PE activities Make up problems Set up on-line environment Read- other problems - other solutions - grade justifications - disputes additional learning from reading everything peers write Solve problems Exam Process Control Assign ID Edit questions Assign who answers questions Assign level-2 graders Level-1 and Level-2 graders grade solutions Determine Final Grades Resolve Disputes Dispute final grade 7

8. Instructor Control Process Student Learning Process Confirmation ID, understand process Course Design Make up problems Set up on-line environment Read- other problems - other solutions - grade justifications - disputes Solveproblems Exam Process Control Assign ID Edit problems Assign who solves problems Assign level-2 graders Level-1 and Level-2 graders grade solutions Determine Final Grades Resolve Disputes Dispute final grade 8

9. Evaluation (grading) • Evaluation includes: • Written critique or “justification” (positive or negative) • Optional: separate sub-criteria to critique • Solution result is correct and complete (40%) • Solution was well explained (30%) • Solution demonstrated class materials well (10%) • Solution cited appropriate references (20%) • Grade • Evaluation may be disputed (optional) • Student must re-evaluate own solution when disputing example of four sub-criteria(totals to 100%) 9

10. Instructor should provide… • Detailed instructions and timetable • Solution: what is expected • Critiquing and grading guidelines 10

11. Outline • Motivation • Participatory Exam approach • A bit of theory • Experimental results • Interesting issues 11

12. Constructivism(Learning Theory) • The central idea is that human learning is constructed, that learners build new knowledge upon the foundation of previous learning{learning throughout the exam process} • Two classic categorizations • Cognitive Constructivism (Piaget’s theory) • Social Constructivism (Vygotsky’s theory) 12

13. Learning • People learn as they navigate to solve problems (Koschmann et al, 1996) and design representations of their understanding (Suthers 1999) • Learning requires cognitive flexibility (Spiro et al. 1991), and results from interaction with people having different experiences and perspectives (Goldman-Segall et al. 1998) 15

14. Where is Knowledge Constructed in PE? • In all PE stages:constructing problems, solutions, grade justifications, dispute justifications • When reading everything their peers write • Students also are motivated to learn more when peers will read their work (McConnell, 1999). 17

15. Outline • Motivation • Participatory Exam approach • A bit of theory • Experimental results • Interesting issues 19

16. Course Information NJIT CIS677: Information System Principles • Graduate level introductory survey core course (Masters/Ph.D.) • Aim: study how IS/IT can be used effectively • Both on-campus and distance-learning sections • software: WebBoard • Traditional Exam: • Three-hour, in class, 3-4 essay questions, 6 pages of notes • Used PE 5 times between Fall 1999 and Summer 2002 • We compared control groups without PE and treatment groups with PE • Also, we used with shorter essay questions in CIS 365, undergraduate course on file structures in Fall 2002, with similar survey results. 20

17. Enjoyability Cronbach’s Alpha=0.68 SA - strongly agree (5 points); A - agree (4); N - neutral (3); D - disagree (2); SD - strongly disagree (1); the mean is out of 5 points; S.D. - standard deviation 21

18. Perceived Learning Cronbach’s Alpha=0.88 22

19. Recommendation: Do Again! Similar results for CIS365: undergraduate file structures course using short essay questions (Fall 2002) 23

20. Outline • Motivation • Participatory Exam approach • A bit of theory • Experimental results • Interesting issues 24

21. Trade-offs • Trade-offs for students (traditional vs. PE) • Participation: solutions only vs. entire exam life-cycle • Timing: concentrated vs. drawn-out (2.5 weeks) • Access to information: limited vs. the Internet • Trade-offs for professors • Fewer solutions to evaluate, but each is different • Timing: concentrated vs. drawn-out process • Much more administration 25

22. What students liked best • Active involvement in the exam process • Flexibility • Reduction in tension 26

23. Extending the PE Approach • Degree of Evaluation • Currently: students only evaluate solutions • What about evaluating: • quality of problems (how good was the problem?) • quality of evaluations/grades (how good was the grading?) • All could be disputed 29

24. Full Collaboration • Groups for: • Problems, solutions, evaluation, dispute arbitration • Requires group process support • Group roles: leader, scheduler, etc. • Process: work on each activity together or separately, internal review • Grading of individual group members • Process Tools: brainstorming, voting, etc. 30

25. Support Software • We plan to develop support software • Guide students (what to do next) • GSS tools for collaboration • Manage administration for instructor • Minimize overhead for students • Minimize overhead for instructors 31

26. Thank you! Questions, please? PE: Contributions • Systematic technique to increase learning • Constructivist approach, actively engaging students in the entire problem life-cycle • Minimize overhead for students and instructors • Experimental evaluation • Supporting software • Looking for collaborators to try this out with us! 32