Optical Mark Reader Service

1. Optical Mark Reader Service OMR Machine in Print Services Supplied by DRS Marks 7000 student answer sheets per hour, provided sheets are completed correctly and in good condition.OMR Machine in Print Services Supplied by DRS Marks 7000 student answer sheets per hour, provided sheets are completed correctly and in good condition.

2. Slide 2 Background Marking Multiple Choice Questions (MCQs) Other available Exam styles (question types) require a specially-prepared answer sheet and a decode file for Multiquest Multiple Choice Questions (follow hyperlink) Give conceptual overview of what is a MCQ and how it can be marked using a master template. That example uses 5 part TFD MCQ and we will work with a 50 Question, 5-part TFD on this course (50TFD). Exam styles (follow hyperlink) illustrates the other 2 exam styles for which answer sheets and Multiquest decodes are available in Bristol.Multiple Choice Questions (follow hyperlink) Give conceptual overview of what is a MCQ and how it can be marked using a master template. That example uses 5 part TFD MCQ and we will work with a 50 Question, 5-part TFD on this course (50TFD). Exam styles (follow hyperlink) illustrates the other 2 exam styles for which answer sheets and Multiquest decodes are available in Bristol.

3. Slide 3 Overview of the OMR Service Essentially, five phases: Setting up prior to the exam In good time - immediate support cannot be guaranteed! The exam itself Scanning of answer sheets by Print Services Correcting errors (student responses that don�t scan) Reporting and adjusting marking scheme Preliminary reports As necessary, adjust scoring scheme and/or master answers Final reports (which can be exported to Excel) Overview the entire process. Use the next slide to illustrate each phase in more detail.Overview the entire process. Use the next slide to illustrate each phase in more detail.

4. Slide 4 Overview of the OMR Service Setting up The exam itself Scanning Correcting errors Reporting and adjusting Follow each hyperlink in turn.Follow each hyperlink in turn.

5. Slide 5 This course will cover Use of Multiquest software Procedures and timescales See APPENDIX C of training document omr-t1 Guidelines: For error-free scanning and administration For security and integrity of data Sources of further help and information Aims and objectives of the course include: To use the required Multiquest software to: Set up the files need to machine mark the exam Correct and report on the machine-marked exam To go through the necessary procedures, which must be followed within the prescribed timescales and which are summarised in Appendix C of the training document (and also available on the web). To ensure familiarity with the guidelines that ensure problem-free exam marking and security of the exam data. Note that everything is well documented in the handouts and on the web, so no need to take detailed notes. Note the URL of the web pages (a vital resource) Aims and objectives of the course include: To use the required Multiquest software to: Set up the files need to machine mark the exam Correct and report on the machine-marked exam To go through the necessary procedures, which must be followed within the prescribed timescales and which are summarised in Appendix C of the training document (and also available on the web). To ensure familiarity with the guidelines that ensure problem-free exam marking and security of the exam data. Note that everything is well documented in the handouts and on the web, so no need to take detailed notes. Note the URL of the web pages (a vital resource)

6. Slide 6 Before you can use the service Register (�250) Print Services will send you the SLA, a sample of each of the 3 types of student answer sheets, and some Answer Sheet Request Forms Ensure that at least 2 key users are trained If necessary, install Multiquest software (v2.38)and set up a secure exam filing system Registration covers the license for Multiquest. There are other costs, detailed on the web site. Print Services will send samples of available student answer sheets and forms to order these for the exam. Keep the forms until the date of the exam is known, then order the answer sheets immediately. The current supported version of Multiquest is 2.38(we will review this annually) Instructions for the installation of Multiquest are on the web site in the Technical section, but we cannot publish the necessary passwords to unlock the distribution. The passwords will be sent: On registration On request to the service mail help address help-omr@bristol.ac.uk Registration covers the license for Multiquest. There are other costs, detailed on the web site. Print Services will send samples of available student answer sheets and forms to order these for the exam. Keep the forms until the date of the exam is known, then order the answer sheets immediately. The current supported version of Multiquest is 2.38(we will review this annually) Instructions for the installation of Multiquest are on the web site in the Technical section, but we cannot publish the necessary passwords to unlock the distribution. The passwords will be sent: On registration On request to the service mail help address help-omr@bristol.ac.uk

7. Slide 7 Multiquest filing system Under C:\Speedwell\Multiquest (or similar) academic year\programme\year of study The program will be installed in a folder, possibly C:\Speedwell\Multiquest We recommend that the files that contain exam data (called the Response Files) are stored below that, with the files for kept in separate folders for each group: Academic year\programme\year of study The program will be installed in a folder, possibly C:\Speedwell\Multiquest We recommend that the files that contain exam data (called the Response Files) are stored below that, with the files for kept in separate folders for each group: Academic year\programme\year of study

8. Slide 8 Security considerations Confidentiality Secure location Appropriate file permissions (password protected) Involve Computer Support Officer Data integrity and reconciliation Clear, consistent naming Reliable and secure backup archive (eg, CD) Don�t store the response files in a network shared folder because networked folders are not secure. Best to involve the department�s Computer Support officer and point them at the technical pages on the web. They can get more help from us if wanted. Task 1 from omr-t1Multiquest files and security Don�t store the response files in a network shared folder because networked folders are not secure. Best to involve the department�s Computer Support officer and point them at the technical pages on the web. They can get more help from us if wanted. Task 1 from omr-t1Multiquest files and security

9. Slide 9 Setting up for the exam Go through entire slide, and then:Request Form Part 1 (follow hyperlink). The same Job Number (i.e., same Request Form) must be used: to order the sheets for a particular exam (date) and request the scanning of the completed sheets for that same exam. Go through entire slide, and then:

10. Slide 10 Candidate lists Candidate number (8-digit) Candidate names Surname Forename Middlename Next, set up the exam in Multiquest. To do this requires the candidate lists and for these lists to be in a particular format: 8-digit candidate numbers Names as Surname Forename Middlename (one field) Microsoft Excel is a suitable package to manipulate the candidate lists into the required format. Once manipulated in Excel, the candidate list can be exported into Multiquest.Next, set up the exam in Multiquest. To do this requires the candidate lists and for these lists to be in a particular format: 8-digit candidate numbers Names as Surname Forename Middlename (one field) Microsoft Excel is a suitable package to manipulate the candidate lists into the required format. Once manipulated in Excel, the candidate list can be exported into Multiquest.

11. Slide 11 Candidate numbers Current 5-digit University numbers Possible future 8-digit Answer sheets and decodes anticipate the future Convert the University�s current 5-digit candidate numbers to 8-digits by adding three trailing zeros. Perhaps the easiest way to do this is to use Excel�s CONCATENATION formula. Concatenate = to join What we want to do is �join� three zeros onto the end of the 5-digit candidate numbers. Task 2.1 to 2.5 from omr-t1Creating a candidate list (Excel) � candidate numbers Convert the University�s current 5-digit candidate numbers to 8-digits by adding three trailing zeros. Perhaps the easiest way to do this is to use Excel�s CONCATENATION formula. Concatenate = to join What we want to do is �join� three zeros onto the end of the 5-digit candidate numbers. Task 2.1 to 2.5 from omr-t1Creating a candidate list (Excel) � candidate numbers

12. Slide 12 Concatenating names =CONCATENATE (Surname cell," ",Firstname cell, " ",Middlename cell) from a worksheet to a new sheet, for example:  =CONCATENATE('Sheet 1'!B2," ",'Sheet 1'!C2, " ",'Sheet 1'!D2) Note that Middlename is not strictly necessary, but could help avoid ambiguity. What is important is that the entire name is in asingle field (column) in Excel and that the order is: Surname Firstname Middlename Task 2.6 from omr-t1Creating a candidate list (Excel) � candidate names Note that Middlename is not strictly necessary, but could help avoid ambiguity. What is important is that the entire name is in asingle field (column) in Excel and that the order is: Surname Firstname Middlename Task 2.6 from omr-t1Creating a candidate list (Excel) � candidate names

13. Slide 13 Generate from Excel Tab delimited text file for import into Multiquest Now Excel can be used to create: The candidate list as a tab-delimited text file for export to Excel. Two printed lists that must be sent to Print Services with the answer sheets for marking: Candidates in name order Candidates in candidate number order (re-sort) After the exam, on one or other of these lists (it doesn�t matter which) clearly mark the absentees. Task 3 from omr-t1Exporting a candidate list (Excel)Now Excel can be used to create: The candidate list as a tab-delimited text file for export to Excel. Two printed lists that must be sent to Print Services with the answer sheets for marking: Candidates in name order Candidates in candidate number order (re-sort) After the exam, on one or other of these lists (it doesn�t matter which) clearly mark the absentees. Task 3 from omr-t1Exporting a candidate list (Excel)

14. Slide 14 Consistency of naming Use this format of exam name for Multiquest response files Unit name Year of study Paper number Exam date Sci 1 paper1 220503 Write this name on the floppy disk sent to Print Services And use the same exam name on all paperwork Request Form and Answer Sheets Printed Candidate Lists This is an appropriate point to note the importance of consistent naming of files and paperwork connected with a particular examination. To ensure consistency and to avoid ambiguity, the exam name should be constructed as: Unit name Year of Study Paper Number Exam date For example, Sci 1 paper1 220503 Use that name: For the response files created by Multiquest On the label of the floppy disk sent to Print Services On all paperwork, namely: The Print Services Request Form The answer sheets themselves The printed candidate lists Point out Appendix A of the document as reminder.This is an appropriate point to note the importance of consistent naming of files and paperwork connected with a particular examination. To ensure consistency and to avoid ambiguity, the exam name should be constructed as: Unit name Year of Study Paper Number Exam date For example, Sci 1 paper1 220503 Use that name: For the response files created by Multiquest On the label of the floppy disk sent to Print Services On all paperwork, namely: The Print Services Request Form The answer sheets themselves The printed candidate lists Point out Appendix A of the document as reminder.

15. Slide 15 Setting a new examination (1) Set up empty Multiquest response files that tell the OMR machine how to read answer sheets by selecting the appropriate exam style (decode) Bristol Physiology � 50 TFD (sheet 50TFD) Bristol Anatomy � 60 A to E (sheet 60A-E) Medical 130 A to Z (sheet 30EMI) and setting the number of responses to read also, set grading scheme and scoring scheme (optional) The decodes are installed with Multiquest and tell Multiquest and the OMR machine how to read and interpret a particular layout of answer sheet. There is a decode for each of the 3 styles: 50TFD 60A-E 130EMI To tell the OMR machine how to read a particular exam, it is also necessary define the total number of responses to expect (follow hyperlink and explain). Task 4.1 to 4.7 from omr-t1Setting a new exam in Multiquest The decodes are installed with Multiquest and tell Multiquest and the OMR machine how to read and interpret a particular layout of answer sheet. There is a decode for each of the 3 styles: 50TFD 60A-E 130EMI To tell the OMR machine how to read a particular exam, it is also necessary define the total number of responses to expect (follow hyperlink and explain). Task 4.1 to 4.7 from omr-t1Setting a new exam in Multiquest

16. Slide 16 Setting a new examination (2) Now it is necessary to import the candidates into Multiquest. Note: Up to now, the choices made can be changed before the exam is taken, by editing the exam in Multiquest. Once the candidate list is imported the exam cannot be changed, so if new candidates were put forward the exam would need to be set up again from scratch (but candidates who have withdrawn can just be marked as absent on the printed candidate listing). Task 4.7 to 4.11 from omr-t1Setting a new exam in Multiquest Now it is necessary to import the candidates into Multiquest. Note: Up to now, the choices made can be changed before the exam is taken, by editing the exam in Multiquest. Once the candidate list is imported the exam cannot be changed, so if new candidates were put forward the exam would need to be set up again from scratch (but candidates who have withdrawn can just be marked as absent on the printed candidate listing). Task 4.7 to 4.11 from omr-t1Setting a new exam in Multiquest

17. Slide 17 Copy response files to floppy The response files must be sent to Print Services along with the student answer sheets for marking. The OMR machine will populate the files with marks. All three response files (RES RST RSX) must be copied to floppy disk. The disk must be clearly labelled with: The name and number of the exam contact. The Faculty and Department The exam name (using the standard naming convention)The response files must be sent to Print Services along with the student answer sheets for marking. The OMR machine will populate the files with marks. All three response files (RES RST RSX) must be copied to floppy disk. The disk must be clearly labelled with: The name and number of the exam contact. The Faculty and Department The exam name (using the standard naming convention)

18. Slide 18 Master answer sheet Two more things now need to be done: Task 6 of omr-t1Filling in the master answer sheet A master answer sheet must be marked up (this tells the OMR machine what the correct answers are) as in Prepare the master answer sheet (follow the hyperlink) Master candidate number designates the sheet as master for the OMR machine. It is always required and is 00000000 (eight zeros). The machine reads the marked-up number, not words. For fewer questions, eg 40 not 50, leave the last 10 questions blank. For a four-part exam (A-D, not A-E) then leave option E blankTwo more things now need to be done: Task 6 of omr-t1Filling in the master answer sheet A master answer sheet must be marked up (this tells the OMR machine what the correct answers are) as in Prepare the master answer sheet (follow the hyperlink) Master candidate number designates the sheet as master for the OMR machine. It is always required and is 00000000 (eight zeros). The machine reads the marked-up number, not words. For fewer questions, eg 40 not 50, leave the last 10 questions blank. For a four-part exam (A-D, not A-E) then leave option E blank

19. Slide 19 Student answer sheets Prepare instructions to students and ensure invigilator knows these How to write candidate numbers and record answers(eg, 4 or 5 part questions) Marking and care of answer sheets Task 7 of omr-t1Providing instructions to students Crucial. 15 minutes scanning paid for. This is 10 mins setup time and 5 mins scanning. At 7000 per hour that is 350 answer sheets paid for under ideal conditions. If the scanning takes longer than 15 mins it is charged at an extra �5 per minute. Marking and care of answer sheets (follow hyperlink) Candidate numbers must be filled in correctly (most common cause of failure) and have three trailing zeros Horizontal mark in box in pencil (ease of correction) No text or other marks outside of boxes. Don�t mark or damage track marks, bend, fold or dogear. Note: If master is 4 part, student part E is ignored, but best avoid confusion by instructions not to mark box E. Departments are responsible for ensuring blank answer sheets arrive in exam hall and instructing students! Task 7 of omr-t1Providing instructions to students Crucial. 15 minutes scanning paid for. This is 10 mins setup time and 5 mins scanning. At 7000 per hour that is 350 answer sheets paid for under ideal conditions. If the scanning takes longer than 15 mins it is charged at an extra �5 per minute. Marking and care of answer sheets (follow hyperlink) Candidate numbers must be filled in correctly (most common cause of failure) and have three trailing zeros Horizontal mark in box in pencil (ease of correction) No text or other marks outside of boxes. Don�t mark or damage track marks, bend, fold or dogear. Note: If master is 4 part, student part E is ignored, but best avoid confusion by instructions not to mark box E. Departments are responsible for ensuring blank answer sheets arrive in exam hall and instructing students!

20. Slide 20 Send to Print Services Student answer sheets Task 8.1 of omr-t1The marking and return of the examination After the exam, collect together all the above and ensure these are correctly marked up and labelled. Task 8.1 of omr-t1The marking and return of the examination After the exam, collect together all the above and ensure these are correctly marked up and labelled.

21. Slide 21 Print Services Request FormPart 2: Order for scanning Hand-deliver in a box to avoid damage. Must include the Request Form with Part 2 completed. This must be the same Job Number (Request Form) used to order the answer sheets. You should specify Negative Marking Yes/No (but this should be checked after scanning and can be changed). Hand-deliver in a box to avoid damage. Must include the Request Form with Part 2 completed. This must be the same Job Number (Request Form) used to order the answer sheets. You should specify Negative Marking Yes/No (but this should be checked after scanning and can be changed).

22. Slide 22 If some sheets won�t scan �Surgery�at Print Services Task 8.2 of omr-t1The marking and return of the examination Phone call from Print Services for the academic responsible for the exam to call in. This will only happen if one or more entire papers have failed to scan. Minor errors of scanning individual answers will be corrected in the department. The academic may need to: Correct candidate numbers Retranscribe a ruined paper onto a fresh answer sheet. The aim is to ensure that all answer sheets scan, because it is a lot of work and prone to error to manually type an entire answer sheet into Multiquest.Task 8.2 of omr-t1The marking and return of the examination Phone call from Print Services for the academic responsible for the exam to call in. This will only happen if one or more entire papers have failed to scan. Minor errors of scanning individual answers will be corrected in the department. The academic may need to: Correct candidate numbers Retranscribe a ruined paper onto a fresh answer sheet. The aim is to ensure that all answer sheets scan, because it is a lot of work and prone to error to manually type an entire answer sheet into Multiquest.

23. Slide 23 Returned from Print Services Task 8.3 of omr-t1The marking and return of the examination Print Services will return, by secure delivery: The Candidate Lists The Master Answer Sheet All Student Answer Sheets, in two piles: Scanned OK Had minor errors The floppy disk containing the three now-populated response files Task 9 of omr-t1Copying response files to the Multiquest filing system Never open the response files on floppy disk as this will corrupt them! No need to do task 9 in the training room as there are some populated response files in the Sci_2 folder. Task 8.3 of omr-t1The marking and return of the examination Print Services will return, by secure delivery: The Candidate Lists The Master Answer Sheet All Student Answer Sheets, in two piles: Scanned OK Had minor errors The floppy disk containing the three now-populated response files Task 9 of omr-t1Copying response files to the Multiquest filing system Never open the response files on floppy disk as this will corrupt them! No need to do task 9 in the training room as there are some populated response files in the Sci_2 folder.

24. Slide 24 Correcting errors Reasons for unscanned responses Missing (or unclear) response Multiple response Reasons for unscanned responses: Missing response (part a) Unclear-to-machine (hence missing) response (part b) Multiple response (part c) Multiple-to-machine response (part d), e.g. a changed answer not fully rubbed out. In Multiquest, correct the response if appropriate. If can�t unambiguously interpret the response, enter D Interpreting and correcting unclear responses is a job for academic staff If responses are changed in Multiquest they should be signed-for on back of the student�s answer sheet. Multiquest also keeps its own audit trail. Task 10 from omr-t1Correcting unscanned student responses Reasons for unscanned responses: Missing response (part a) Unclear-to-machine (hence missing) response (part b) Multiple response (part c) Multiple-to-machine response (part d), e.g. a changed answer not fully rubbed out. In Multiquest, correct the response if appropriate. If can�t unambiguously interpret the response, enter D Interpreting and correcting unclear responses is a job for academic staff If responses are changed in Multiquest they should be signed-for on back of the student�s answer sheet. Multiquest also keeps its own audit trail. Task 10 from omr-t1Correcting unscanned student responses

25. Slide 25 Preliminary reporting Absentees Candidate listing (summarised results) Correct answers (master answer sheet) Print Item analysis Print Score distribution histogram Print Preliminary reports help to check the examination � by comparing Multiquest�s absentee listing against the printed Candidate Lists with absentees marked - by double-checking the master answers. and by examining the marks and various statistics about the examination (a job for academic staff) Printed copies of the following will be found useful by academics who scrutinise the examination: Correct Answers Report Item Analysis Report Score Distribution Histogram Task 11 from omr-t1Generating ReportsPreliminary reports help to check the examination � by comparing Multiquest�s absentee listing against the printed Candidate Lists with absentees marked - by double-checking the master answers. and by examining the marks and various statistics about the examination (a job for academic staff) Printed copies of the following will be found useful by academics who scrutinise the examination: Correct Answers Report Item Analysis Report Score Distribution Histogram Task 11 from omr-t1Generating Reports

26. Slide 26 Consider your examination As appropriate, adjust the marking scheme Negative or weighted marking Withdraw question parts Withdraw entire questions Change the expected (master) answer When the marks have been considered, academic opinion may be that a particular question was unclear, unfair, or incorrectly transcribed onto the master answer sheet. For instance, if all or most candidates get a question wrong, this should prompt further consideration. Adjustment of the weight given to right/wrong answers Withdrawing part of a question Withdrawing an entire question Changes to the expected (master) answer Note that this is not the same as adjusting (correcting) unscanned student responses. Run Adjusting the marking scheme (follow hyperlink) to show how < always counts as zero (null, no mark) regardless of whether it is a student or master mark. Task 12 from omr-t1Adjusting the exam scores and withdrawing questions When the marks have been considered, academic opinion may be that a particular question was unclear, unfair, or incorrectly transcribed onto the master answer sheet. For instance, if all or most candidates get a question wrong, this should prompt further consideration. Adjustment of the weight given to right/wrong answers Withdrawing part of a question Withdrawing an entire question Changes to the expected (master) answer Note that this is not the same as adjusting (correcting) unscanned student responses. Run Adjusting the marking scheme (follow hyperlink) to show how < always counts as zero (null, no mark) regardless of whether it is a student or master mark. Task 12 from omr-t1Adjusting the exam scores and withdrawing questions

27. Slide 27 Exporting results Export percentage scores to Excel(or another package) as a Comma Delimited file Many people find it convenient to do final manipulations of the exam results in Excel (for instance, to process the MCQ section as part of a larger examination). However, the results can be exported into any other package that accepts delimited text files. And, of course, it may be desirable to export into Excel and from there transfer the data into another package (e.g., wordprocessor or database) Task 15 from omr-t1Setting sub-exams (optional) Task 16 from omr-t1Reporting sub-exams (optional) Show this by reference to the slide above.Many people find it convenient to do final manipulations of the exam results in Excel (for instance, to process the MCQ section as part of a larger examination). However, the results can be exported into any other package that accepts delimited text files. And, of course, it may be desirable to export into Excel and from there transfer the data into another package (e.g., wordprocessor or database)

28. Slide 28 Archiving Archive the finalised exam files onto CD Consider security of data on the floppy disk used to transport response files in files deleted when tidying up when re-deploying PCs used for exams The finalised response files and files containing finalised exported data should be archived: onto CD � two copies, locked away in separate locations Be careful with the floppy disks: they are not secure long-term storage and should be re-formatted to completely destroy the files (or a secure file deletion program). If you then tidy up the files on the PC (for instance, to delete transitional import/export files), beware that files deleted using Windows delete commands: End up in the Recycle Bin Or can be undeleted by people who know what to do Similarly, be careful when redeploying or scrapping a PC that has contained exam marks. Involve the department�s computer support officer. Task 14 from omr-t1Archiving the examinationThe finalised response files and files containing finalised exported data should be archived: onto CD � two copies, locked away in separate locations Be careful with the floppy disks: they are not secure long-term storage and should be re-formatted to completely destroy the files (or a secure file deletion program). If you then tidy up the files on the PC (for instance, to delete transitional import/export files), beware that files deleted using Windows delete commands: End up in the Recycle Bin Or can be undeleted by people who know what to do Similarly, be careful when redeploying or scrapping a PC that has contained exam marks. Involve the department�s computer support officer. Task 14 from omr-t1Archiving the examination

29. Slide 29 Sub exams (optional) Set up before creating response files Allow reporting of defined sections of the exam For instance, in a 40 question exam you might want to set the first 20 questions on anatomy and the last 20 questions on physiology and produce separate reports on the student�s performance in each section. This can be done by dividing an exam into sub-exams. This should be done: at the time of setting up the exam at the end of Step 4.6 on page 11. For instance, in a 40 question exam you might want to set the first 20 questions on anatomy and the last 20 questions on physiology and produce separate reports on the student�s performance in each section. This can be done by dividing an exam into sub-exams. This should be done: at the time of setting up the exam at the end of Step 4.6 on page 11.

30. Slide 30 Summary Essentially, five phases: Setting up prior to the exam In good time - immediate support cannot be guaranteed! The exam itself Scanning of answer sheets by Print Services Correcting errors (student responses that don�t scan) Reporting and adjusting marking scheme Preliminary reports As necessary, adjust scoring scheme and/or master answers Final reports (which can be exported to Excel) Overview the entire process. Use the next slide to illustrate each phase in more detail.Overview the entire process. Use the next slide to illustrate each phase in more detail.

31. Slide 31 Some key points Security of exam data Consistent naming of each exam (Appendix A) and use of the same Job Number for each exam Zero-filled candidate numbers: with three trailing zeros to bring them up to 8-digits. Clear instructions to students and care of sheets Stick to the timetable (Appendix C)

32. Slide 32 Help and support Consult: the training document (document omr-t1) the OMR web site (other documents, FAQs) other local users as appropriate, your local IT support person

33. Optical Mark Reader Service