1 / 49

Security Strategies for Every Stage of the Testing Process

Security Strategies for Every Stage of the Testing Process. DeDe Hedlund , Creighton University Jeff Place, Questionmark Evangelist NCTA – Minneapolis, MN Friday Aug. 3, 2012, 11:30-12:30 pm. Goals of this Session. There are no shortage of test security challenges

nyx
Download Presentation

Security Strategies for Every Stage of the Testing Process

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Security Strategies for Every Stage of the Testing Process DeDeHedlund, Creighton University Jeff Place, Questionmark Evangelist NCTA – Minneapolis, MN Friday Aug. 3, 2012, 11:30-12:30 pm

  2. Goals of this Session There are no shortage of test security challenges At each stage of the assessment process (Authoring, Scheduling, Delivery, and Reporting) solutions exist to meet security challenges Let’s Identify the threats to watch out for Show you today how certain product functionality can meet your test security needs by walking through a typical examination process and some real life examples

  3. Threats Impersonation Content Theft Cheating

  4. Issues Fraud Triangle Threats • Rationalization • Opportunity • Motivation • Impersonation • Content Theft • Cheating

  5. Low/High to High/High Stakes Tests What’s at Stake? Higher Stakes Life and Limb Promotion & Jobs & Legal Concern Educational Exams Tests Elearning & Surveys

  6. Monitoring Tests Securely ID Fraud Protect Content Minimizes Cheating √ √ √ √ √ √ √ √ √ √ √ √ √ X √ √ X X

  7. Providers ID Fraud Protect Content Minimizes Cheating √ √ √ √ √ √ √ √ √ √ √ √ √ X √ √ X X

  8. Assessment Management Process

  9. Authoring: Item Banking and Security • Security impacts • Validity of assessment results • Fairness to participants • One approach: security rights that are role-based • Set access permissions by role or profile • Limit an author’s or reviewer’s access to the various functions • Limit access to topic (item) folders • Limit what the author may do in those folders

  10. Scheduling • Create participants and schedule assessment delivery • Manage participants individually or in hierarchical groups • Schedule access to assessments: • Limit dates • Limit attempts • Require proctor/invigilator • Deliver to certified test center • Online or offline delivery • Email Broadcast:Notifications to Participants

  11. Blended Delivery Levels of Monitoring Stakes of Assessment

  12. Secure Browser Regular Browser Questionmark Secure • Stops people from printing questions • Stops people from typing in their own URL • Always display in full screen and it’s not possible to maximize or minimize • Stops people starting a new task • Does not provide menu options or icons • Disables control keys and task switching • Disables right-click menu options • Disables the ability to start new applications • Prevents going backwards to a previous page • Stops people exiting in a high stakes, proctored, environment • Hides the HTML source • Provides an API to control certain functions of a browser from the server • Server can command Questionmark Secure to display a toolbar

  13. Browser Check Ensures Test Takersuse a compatible browser

  14. Delivery Security • Perception can be configured to use SSL to ensure communication between participants and the server are secure • Data in item repositories are encrypted (QML is encrypted) providing added item bank security • All participant scoring is conducted on the Perception server (rather than on client/participant PCs) • Scoring algorithms are not sent in any way to participant PCs

  15. Delivery Security Features

  16. Limiting Item Exposure • Question by Question (QxQ) delivery options limit the amount of content shown at one time • Randomization of questions and choices to prevent cheating

  17. Reporting • Score List Report • IP addresses (as expected?) • Time taken (too quick, too long?) • Easy flagging of participants who get perfect scores for further investigation • Coaching Report • Drill down on an individual participant, comparison to benchmark, use in appeal process • Item Analysis Report • Determine psychometric performance and examine item difficulty drift over time • Test Analysis Report • Assessment defensibility review and test difficulty drift

  18. Case studies Case studies from Creighton University • What their test security needs were • How our software addressed those needs • Demo of solutions

  19. Creighton University School of Pharmacy & Health Professions • Private Jesuit University located in Omaha, NE • 2001 started the first and only accredited truly distance based full time Doctor of Pharmacy program • Issues to resolve and security requirements • How to securely deliver and administer exams to students located throughout the country • Paper and pencil were too expensive and it’s difficult to distribute and collect approximately 5600 exams per semester • Control who sees what test and when • Lock down the computer to eliminate printing, instant messaging, browsing the internet during the tests • Ability for students to “mark” an identified area of a graphic

  20. Creighton University School of Pharmacy & Health Professions Solutions offered to meet requirements: • Provided online testing solution and eliminated almost all paper and pencil tests to reduce costs • 22 Question types including Hotspot allowing online exams to replicate past paper exam formats with the additional ability to provide instant feedback. • Ability to randomly deliver different variations of an exam • Manage large item banks of questions including security for collaborative authoring among faculty • Lock down browser • Scheduling (right test to right student) Demos • Hotspot question type & feedback text/images • Password system & variations of an exam • Item Banking and security for collaborative authoring • Questionmark Secure • Scheduling

  21. Demonstration

  22. Variations of an Exam

  23. Item Banking

  24. Creighton University School of Pharmacy & Health Professions Additional security measures • Established guidelines for choosing new proctor sites • Formalized the process of communicating what is expected from a proctor site • Proctor Certification forms are required from every student who completes an exam • Exam specific instructions are emailed to the proctors

  25. Remote Video Monitoring Custom Hardware Laptops and PCs

  26. Candidate (Participant) Logs in As Software Secure and Questionmark integration is for higher stakes exams a Participant will be referred to as a Candidate

  27. Schedule Appointment Exam Candidate selects exam and “Schedules Appointment” Exams (4) Surveys (1)

  28. Schedule Day and Time

  29. Review Schedule If the exam is to be schedule in the next 3 days or if the participant might be expected to pay some of the cost of the exam then there are more steps

  30. Payment Process

  31. Candidate Takes Exam When It Was Scheduled

  32. Candidate (Participant) Logs in

  33. Selects Exam to Take Exams (4) Surveys (1)

  34. ProctorU / Software Secure • Camera and Microphone Access • Identity Checks • Captures Photo, Proctor Checks against files • Proctor checks has Participant scan the room

  35. Security Checks Complete Candidate Takes Exam • After Authentication Candidate takes Exam • Video and screens are monitored during exam

  36. Potential Applications for Mobile Delivery

  37. Mobile Test Centers • Use notebooks/3G laptops/iPads to create mobile test centers • Enables greater flexibility in where you deliver assessments • Assessments can be conducted in ‘on location’ environments Slide 10

  38. “Securing” an iPad… • An app should prevent candidates from easily getting to other URLs • But – access to “home button” allows task-switching • Hardware solutions are available to enable use of iPad in a “kiosk” mode Slide 11

  39. Considerations: • The higher the stakes, the higher the propensity to cheat –still require human monitoring! • Bandwidth – Be sure to test the wifi and/or 3G signal strength prior to event Slide 12

  40. Cheat-resistant questions and assessments

  41. Considerations for cheat-resistant questions and assessments Randomization Refresh items regularly Build large item banks Track item parameters Tests that are unique for each test taker make it hard to share answers Adaptive tests provide different questions based on responses to previous questions

  42. Improve Your Multiple Choice Questions Answer choices should be roughly the same length and kept as short as possible. Provide a minimum of three answer choices and a maximum of five. Four is considered optimal. Keep your writing clear and concise – you’re testing knowledge, not reading comprehension. Make sure that you’re putting the correct answer in the first two positions as often as the last two positions.

  43. Avoid Previously Delivered Questions

  44. Extended Matching Question Type • The number of answer options depends on the logical number of realistic options for the test taker.) • The same answer choice could be correct for more than one question in the set • Some answer choices may not be the correct answer for any of the questions • So it is difficult the answer this type of question correctly by chance. • A well-written lead-in question is so specific that students understand what kind of response is expected, without needing to look at the answer options.

  45. Scenario Based Questions

  46. Scan a QR Code with your mobile to start an assessment!

  47. Closing and questions • Thank you for your time and attention! • Questions/Comments? • Visit our booth to talk to us more • We have a whitepaper on this topic (Delivering Computerized Assessments Safely and Securely) • Visit our website to download: http://www.questionmark.com/us/whitepapers/index.aspx

  48. Thank you for attending! DeDeHedlund dhedlund@creighton.edu Jeff Place jeff@questionmark.com

More Related