Labrador: A Tool for Automated Grading Support in Multi-Section Courses

1. Labrador: A Tool for Automated Grading Support in Multi-Section Courses Drexel University Programming Learning Experience (DUPLEX) http://duplex.mcs.drexel.edu Christopher D. Cera, Robert N. Lass, Bruce Char, Jeffrey L. Popyack, Nira Herrmann, Paul Zoski Drexel University Mathematics and Computer Science

2. Roadmap • Introduction • Problems and Solution Goals • Labrador • Discussion

3. The Group • Bruce Char • Professor, Computer Science • Nira Herrmann • Professor and Head, Mathematics • Jeffrey L. Popyack • Associate Professor, Computer Science • Paul Zoski • Instructor, Math and Computer Science • Christopher D. Cera • Computer Science Graduate Student • Robert N. Lass • Computer Science Undergraduate Student

4. Who Am I • First TA in MCS to experiment with WebCT in December 2000 • Courses I have TA’ed using WebCT • 3 x Introductory Programming I, II • 1 x Object Oriented Programming • Migrated course content to WebCT from previous course website • HTML assignments and labs into question database • Gradebook maintainer • Wrote demo’s and documentation to train other TAs • Developer of software supplements to WebCT to support course administration

5. The Course • Intro Programming I, II • Variety of Majors • Computer Science • Computer Engineering • Information Systems • Mathematics • Digital Media • Class • 1 x 1 hour lecture • 2 x 1 hour labs • People • 250-300 Students • 2-3 Professors • 10-12 Teaching Assistants

6. The Duplex Project: An Overview • Take advantage of advances in Information Technology to improve instruction and reduce costs for computer programming courses • Modular Structure • Multiple Entry Points • Multiple Audiences • Multiple levels of knowledge (Bloom’s Taxonomy) • Computer Supported Cooperative Work (CSCW) in student labs • Online Services – Today’s Topic

7. Roadmap • Introduction • Problems and Solution Goals • Labrador • Discussion

8. Before WebCT • Hundreds of people involved in paper exchanges of handwritten assignments and quizzes • Testing programs required floppy exchanges • No Chat • No newsgroup-style threads • Feedback and grades are not online

9. Before WebCT • Large Classes • Managing several hundred students is difficult due to the logistics involved with organizing paperwork and people.

10. WebCT • Service from Information Resources & Technology (IRT) department (v3.1) • MCS started Dec. 2000 (v3.5) • Upgraded to v3.7 recently • Transition to v3.8 planned in Fall

11. Noteworthy Features • General Course Website • Labs: Online Quizzes with Automated Grading • Homework: Online Assignments • Joint Staff--Student Chat • Discussion Threads

12. Software Design Goals • Bulk Assignment Downloading (prior to 3.7) • Bulk Quiz Downloading • Section Sorting for bulk downloads, or only one section

13. Software Design Goals • Post-Processing student files • Archive Extraction (tar, zip) • Decompress (gz, zip) • Decode (uue) • Minimal staff intervention when transferring submissions between systems, eg. Plagiarism Detection Systems (PDS) • Automatically collate source code • Generate electronic documents to facilitate grading and archiving • Remote Execution • downloading to computer x (on campus) while operating at computer y (off campus)

14. The Bigger Picture • Course-Specific Tasks • Not all processing should be done on the server • Select files have to be transferred to a different system for further processing and analysis by staff • Client-side support is needed to perform this, preferably automated and not necessarily using a web browser.

15. Roadmap • Introduction • Problems and Solution Goals • Labrador • Discussion

16. Labrador: Our Solution • Client-side WebCT Supplement • Implemented in Perl • Works for users with TA and Designer access to WebCT

17. Perl • Practical Extraction and Reporting Language • Created in 1987 to replicate Unix shell functionality • Powerful text manipulation • High-level: rapid prototyping and development • Large library of modules and active development community • Perl is used by many applications, including WebCT

18. A Taste of Labrador Perl Programming

19. Labrador works on Windows, Mac, Linux, Other Unix Variants, etc. • Perl has been ported to all major operating systems • Perl programs can be compiled: • No need for the end user to install Perl • Perl2exe for Windows support (www.perl2exe.com)

20. User Interface • Different users / Different UI preferences • Command-line • Interactive • Configuration File • GUI (in development)

21. Required Info http://webct.drexel.edu/SCRIPT/CS164_Fall2002/scripts/designer/dropbox_edit.pl?DROPBOX_ASSN_VIEW+_side_nav++1006285909 • Username/Password • Server URL webct.drexel.edu • Course ID CS164_Fall2001 • Submission Name or ID 1006285909 • Optional Username List

22. Components JPlag Submission Downloader Section Sorting Post Processing PDF Generator Moss

23. Bulk Downloader • Web-Crawler: simulates clicks of actual staff member • Downloads actual web content (HTML) • Parses HTML to find desired text or URLs to crawl to next • Uses HTTP operations • Works on assignments and quizzes • Only component in Labrador which interacts with WebCT

24. Organizing Submissions by Section • Not supported in original Labrador prototype, added later on. • Organizes files into directories • 3 possible input methods: • Creating a Section column in gradebook • “Username, Section” CSV File • Username file

25. PDF Generation for Electronic Mark-up • Adobe Portable Document Format can be viewed on all major platforms • With Adobe Acrobat, PDFs can be annotated by graders • Sony VAIO Slimtop PC (PCV-LX920) • Wacom Pen Tablet.

26. VAIO / Wacom

27. PDF Markup Example

28. Redistribution • How can we return annotated PDF’s back to students using WebCT? • WebCT Mail • WebCT Groups • Version 3.8 now addresses this issue

29. Heterogeneous Systems • Different systems want different formats • Plagiarism Detection Systems • Moss has been used most extensively • Began experimenting with JPlag more recently • Future work: Automatic program compiling and testing

30. Automated Plagiarism Detection • Digital formats make "borrowing" easy • Browsing similar works needs a simple and quick user interface. • Careful review by faculty to assess results and present to students

31. Moss • Processes C, C++, Java, ML, Lisp, Scheme, Pascal, and Ada programs. • Common code feature reduces false positives

32. Moss Interface

33. JPlag • Processes C, C++, Scheme, and Java programs • For plain text files, it matches a user specified number of words appearing in succession • Could be used for any course grading written (text) documents

34. Command Line Invocation

35. Interactive

36. Quiz Download with Config File

37. Roadmap • Introduction • Problems and Solution Goals • Labrador • Discussion

38. Recent WebCT Enhancements • Relevant to this talk: • 3.7 • Addressed bulk download issue for assignments • 3.8 • Attaching documents to an assignment

39. Future WebCT Enhancements • Power users will need functionality not yet supported • Every domain will also require additional functionality • Not feasible for all domain-specific functionality to run on the WebCT server

40. Wanted • API for non-administrators • Stateful protocol so clients can be built by 3rd parties

41. HTTP: Insufficient Protocol • Inconvenient content interchange • Heavy client interaction • A HTTP based approach will be sensitive to the exact location of web pages, and format of text within them.

42. Labrador Availability • Contact Us • http://duplex.mcs.drexel.edu

43. Project Support • The Pew Learning and Technology Program at the Center for Academic Transformation • National Science Foundation, Division of Undergraduate Education, DUE-#0089009 • The Ramsey-McCluskey Family Foundation, Margaret Ramsey, '84 • Drexel University

44. References • [1] Alex Aiken. Moss: A system for detecting software plagiarism (unpublished), http://www.cs.berkeley.edu/~aiken/moss.html. • [2] Jeffrey L Popyack, Bruce Char, Paul Zoski, Nira Herrmann, and Christopher D. Cera. Managing course management systems. In Proceedings of the thirty-third SIGCSE technical symposium on Computer Science Education, Birds-of-a-Feather Sessions, page 423, 2002. • [3] L. Prechelt, G. Malpohl, and M. Philippsen. Jplag: Finding plagiarisms among a set of programs. Technical Report 2000-1, Fakultat fur Informatik, Universitat Karlsruhe, Germany, March 2000. • [4] Larry Wall, Tom Christiansen, and Jon Orwant. Programming with Perl. O’Reilly and Associates, 3rd edition, 2000. • [5] Michael J. Wise. Yap3: Improved detection of similarities in computer program and other texts. In Proceedings of the twenty-seventh SIGCSE technical symposium on Computer Science Education, pages 130–134. ACM Press, 1996.

45. Roadmap • Introduction • Problems and Solution Goals • Labrador • Discussion • Bonus Slides

46. Blooms Taxonomy • Knowledge • remembering of previously learned material; recall (facts or whole theories); bringing to mind. • Comprehension • grasping the meaning of material; interpreting (explaining or summarizing); predicting outcome and effects (estimating future trends). • Application • ability to use learned material in a new situation; apply rules, laws, methods, theories.   • Analysis • breaking down into parts; understanding organization, clarifying, concluding. • Synthesis • ability to put parts together to form a new whole; unique communication; set of abstract relations.  • Evaluation • Ability to judge value for purpose; base on criteria; support judgment with reason. (No guessing).