1 / 28

Out of the Blue: Electronic Assessment Replaces Paper Bluebooks

Learn how electronic assessment technology is replacing traditional paper bluebooks for efficient and accurate grading. Discover the benefits for students and scorers, as well as the successful implementation process.

jjohnstone
Download Presentation

Out of the Blue: Electronic Assessment Replaces Paper Bluebooks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Out of the Blue: Electronic Assessment Replaces Paper Bluebooks Greg Robinson, Ph.D. Academic Technology Services College of Arts and Sciences University of NC at Chapel Hill Melissa Graham Meeks Doctoral Candidate Department of English

  2. Agenda • Placement Exam: Context of Securexam Pilot • Outline of Placement Exam Objectives • Brief Demonstration of Software • Logistics, Outcomes, Feedback • Student Interviews • UNC-Chapel Hill Status • Comments and Questions

  3. The Problem: Time • The placement exam generates hundreds of pages of student writing, which must be duly processed within the first week of class so that students can be placed in the appropriate courses. • Grading handwritten exams is slower than reading typed exams.

  4. Context: The Writing Program • The WP consists of three courses: ENGL 10, 11, and 12. • Undergraduate requirements mandate that students receive credit for or be continuously enrolled in 11 and 12. • Placement is initially determined by the verbal/English scores for the SAT, ACT, SAT II, or AP exams.

  5. The Placement Exam • Students have the option to take the Placement Exam—a timed, two hour exam requiring students to read a passage and analyze its rhetorical features and argumentative effectiveness. • Summer Bridge participants, junior transfers, nontraditionally aged students, and students who are dissatisfied with their placement usually take this exam. About 10% of the FY class, or 300 students, take the exam in any given year.

  6. ETS-based Process of the Placement Exam • Exam Session • Anchoring: Selects exams that match the 6 rubric descriptions. • Calibration: Trains scorers to match anchor exams to the rubric. • Scoring: Each exam receives two scores. • Rescoring: Exams with scores that differ by more than 1 point are scored by a 3rd grader. • Distribution analysis • Placement decisions posted, and students reschedule their classes as necessary.

  7. Convenience is bad pedagogy, so why ask students to type? • Global revision, which the WP emphasizes, is easier. • Editing is also easier. • For most students, typing is also easier than handwriting. • The WP does not accept handwritten essays in any other context.

  8. How might Type Affect Scoring and Scores? • Because our goal is to place students in the course that will most benefit them, we are delighted that typed exams give students more opportunity to revise and edit: A stronger writing sample means a stronger placement decision. • Typed exams eliminate the “noise” of legibility in the scoring process. • Potentially, it reduces the time necessary to read each exam.

  9. How Can We Get Type? We needed a software that: • Did not significantly alter the writing space with which most students could be expected to be familiar • Could be integrated into the exam process without compromising the timeline • Could deliver typed exams with minimal technical fuss

  10. Securexam Software • Purpose: Secure Electronic Assessment • Setting: Exams administered in a secure environment using Microsoft Word, locking out access to all other files, programs, and the Internet. http://www.softwaresecure.com

  11. Three Security Methods • Lockdown Effects • Alarm Security System (detects exit attempts) • Post-exam Encryption prevents access

  12. Software Demonstration

  13. Implementation Logistics • Students download and install software from the Website before exam. • Provide IT help before and during exam. • Provide extra outlets per room • Collect exam files on USB drives (secure, reliable, network-free, and cheap) at multiple exiting stations. • Batch Decrypt, Print, and Archive files.

  14. Results of Implementation • 172 exams were typed. • 14 were handwritten, by election or technical failure. • The reduced number of total exams is related to the upward shift of SAT scores, not technophobia. • Every stage of the scoring process was timely.

  15. Feedback #1: Laptop Preference • “If you had the option to complete this exam with pencil and paper or using your laptop, which option would you choose and why?” (n = 134 students) • Laptop: 76.3%. ~ Easy edits, speed, comfort, efficiency.

  16. Feedback #2: Laptop Disadvantages • Q#2: “Disadvantages to using a laptop?” • Apprehension about a new environment • Impetus? • Needless layer of complexity

  17. Feedback #3: Essay Quality • “Do you feel that using your laptop allowed you to write a more cohesive and coherent essay?” • Yes: 80.2%

  18. Feedback #4: Implementation • “Do you feel that you would be comfortable using your laptop for other in class assessments such as short answer essays or exams?” • Yes: 86.6%

  19. Summary: Reported Benefits • Benefits for students • Ease of Editing and Revision • Rapid Composition • Improved Organization and Cohesiveness • Review, Depth, Content-Rich • Benefits to scorers • 100% “Easier to grade” • Instant conversion to legible essays • Better student performance

  20. Observed PE Challenges • Student Responsibilities • Uninformed and Unprepared • Skipped Exiting Station (!) • User Apprehension • Technical Challenges…

  21. Technical Challenges • Spyware, P2P apps, Office Add-ins • Macs* and Some PC Hardware • Win 98 and Office 97 = trouble. • Power – Dead batteries, outlets, etc.

  22. Lessons Learned • Students follow instructions poorly outside of the exam setting. • Post removal tools (Spybot S&D, AdAware) • Post Office Add-ins Disable Tool • Technical: Restart required before transferring the file to USB drive.

  23. Securexam Pros • Students already know software (Word) • Integrates with CCI (Laptop Initiative) • Hardware Infrastructure already in place • Office Software already licensed • Blackboard Infrastructure • Potential on campus load? • Recovery Capability • Accommodates batch processing

  24. Student DVD Feedback

  25. Present Status at UNC-CH • Purchased and Used 1000 licenses across the College of Arts and Sciences: English, Music, Communication Studies, EXSS, Political Science, AFAM Studies, American Studies, Public Policy, German. • Implementations: • English Composition Placement Exam • Midterm and final exams • Graduate Qualifying exams

  26. Standard Course Implementation • Post Application to Course Blackboard Site • Collection: Blackboard or email (not USB) • Student Information: http://securexam.unc.edu

  27. Closing Comments • Positive Student Feedback • Positive Instructor Feedback • 100% (n=10) who used in Fall 2004 exams requested for Spring 2005 • 100% (n=10) would recommend to peers, chair, and IT Administration. • Securexam Failsafe Feature: Word Backup Files • IT Support Goal: Instructor Autonomy, Campus Support, Facilities

  28. Contact Information UNC-Chapel Hill Greg Robinson, Ph.D. Senior Academic Technology Liaison College of Arts and SciencesUNC-Chapel HillGreg_Robinson@unc.edu http://securexam.unc.edu Vendor: Software Secure, Inc.http://www.softwaresecure.com

More Related