Industry White Paper - Prometric

1. Third Quarter 2016 | Industry White Paper Industry White Paper Five Key Findings for Best-in-Class Computerised Testing Executive Summary Technologically advanced societies rely on the development of numerous specialised professions in order to ensure the proper and safe operation of critical societal functions, and the continuing discovery of new knowledge and advancement of technology. Fundamental to all professions is some technique or mechanism for admitting new students and practitioners and for making sure they are current with the state of their art or science. Usually this takes the form of a test that assesses an individual student’s or practitioners’ current knowledge. The practice of simply replicating a paper test on a computer, however, should still adhere to best-practice approaches and standards learned over decades of experience by testing and assessment organisations. This becomes a critical path to follow for emerging computer-based testing programs, especially when the subject of single-day testing has become such a topic of great interest in India recently. This report presents data derived from a comparative analysis that indicates gaps and security risks between multi-day test administrations with smaller numbers of test takers and single-format testing of large numbers of test takers. Additionally, the report highlights the use of standard best practices adopted around the world by leading test sponsor organisations. www.prometric.com | 1 Copyright © 2016 Prometric, Inc. All Rights Reserved. |

2. Third Quarter 2016 | Industry White Paper Comparative Analysis Delivering a high-stakes test to large numbers of test takers in a single-day format can pose challenges in delivering valid and reliable exams, managing security risks and ensuring the testing environment allows honest test takers the ability to demonstrate their knowledge and qualifications accurately and fairly. Alternatively, delivering tests over a test administration window with a smaller test taker footprint enables a number of quality measures, security safeguards and test taker advantages. In July of 2016, Prometric conducted a comparative analysis between these two types of testing models to identify the critical components required for a best-in- class testing and assessment program. Five key findings are detailed in this report: 1. Test Administrations Require a Robust Lab Set Up. If the technical infrastructure of a testing environment is comprised of third-party, “rented” test workstations, then precautionary steps should be taken to mitigate the risks involved. A best-practices approach entails that all third- party test workstations be completely isolated and rebuilt to ensure an absolute security equivalent to official, corporate test sites. For example, critical security threats to the exam program would occur if laptops were simply brought in and connected to workstations by browsers. Lab Set Up and Security Multi-Day Testing Single Day Testing Value Ensure that the central testing network is completely and physically isolated from test workstations. Yes No The physical separation of test workstations from the central testing network mitigates security risks from problems that could spread beyond the existing network into the rental facility. Install proxy servers at testing sites. Yes No The test workstations are further protected by being connected to a specific and dedicated proxy server built from scratch at the testing site. Install dedicated servers for test workstations stations at sites. Yes No Instead of a laptop-configured server brought to the site, a dedicated test workstation should be built on site. Such an approach provides extra security because laptop servers can be compromised when moved from location to location. Incorporate best-practices when formatting workstations and the operating system (OS) as well as the installation of additional software. Yes No All test workstations should be flattened, and licensed operating systems (Windows) should be installed from scratch – including the installation of anti-virus software. Set a limit of up to 40 test workstations per testing site. Yes No To deliver better performance and security, a proxy server and a server per 40 workstations should be provided instead of up to 200 test workstations with only one server. www.prometric.com | 2 Copyright © 2016 Prometric, Inc. All Rights Reserved. |

3. Third Quarter 2016 | Industry White Paper 2. Staff Recruitment/Management Call for Careful Governance. When organised cheating occurs, it is dependent upon complicit people being in place who are able to act with little oversight. For some organisations, there is a tendency to send in one or two staff members to work with a testing site’s established teams. If cheating is well organised and well financed, though, there is a need for additional layers of protection to be put in place. By having numerous staff recruited from two different entities – and by using corporate staff – organisations can minimize the possibility of testing staff engaged in bad and unlawful practices. Staff and Security Multi-Day Testing Single Day Testing Value Avoid having a heavy reliance upon third-party, “rental” test sites and provided staff. No Yes The heavy reliance of staff connected to the rental site can provide a dangerous exposure to malfeasance. A best-practices approach typically includes using a consultancy company to help recruit third-party staff. This practice places additional staff on site to reduce the opportunity for site staff to be compromised. Employ dedicated, auditing staff and a robust process. Yes No In addition to corporate staff and third-party staff, it is a good business practice to employ yet another redundant layer of auditors who are recruited through an unbiased agency as a deterrent to proxy schemes. In order to run a proxy scheme at a testing center, culprits must be able to negotiate with staff. Having staff recruited by three different entities greatly reduces the chance that tests can be “bought” from unscrupulous site staff. Ensure there is a high proctor- to-candidate ratio. Yes No Best-in-class test delivery providers offer a 1:10 ratio of proctors to test takers versus a 1:30 ratio offered by second-tier providers. The greater the ratio, the greater the chance of candidate misbehavior that cannot be observed nor reported. Have unbiased test experts process test results in secure servers outside of India. Yes No The use of outside resources in the processing of the score results provides yet another layer of protection. There is a much less likelihood for in-market bad actors to influence professional test experts located in other countries. 3. Best-in-Class Processes Need to Be Measured for Performance Improvement. High-stakes tests around the world call for clear and robust processes that can be audited and managed for performance improvement to reduce the probability of organised or even opportunistic foul play. Process and Security Multi-Day Testing Single Day Testing Value Set up a secure zone at the testing location. Yes No A secure zone is a dedicated area that is demarcated and enforced. The secure zone creates a buffer between the facility and the testing workstations. A secure zone also helps to prevent access by unauthorised personnel. Require test takers to check in and check out – leveraging the secure zone. Yes No Test takers should be checked in and out of the testing labs to ensure that the right person is taking the right test at the right time. This helps eliminate the risk of proxy testers completing a portion of the exam and also acts as a deterrent by being a visible reminder of enforced security measures. www.prometric.com

4. Third Quarter 2016 | Industry White Paper Inspect test takers carefully. Yes No Test takers are inspected through frisking, wanding and other checks to ensure that no foreign materials such as electronic devices, chunky jewelry and slips of paper are brought into the testing area. Set video monitoring at a low ratio of test takers to cameras. Yes No Fixed cameras that feed to DVR storage should be used to eliminate the possibility of malfeasance. Using a handycam for “passes through” will not provide adequate protection. Additionally, fixed cameras and DVR storage should be combined with a high-proctor ratio and walkthroughs in the testing area to provide robust security at the time of testing as well as post-test forensic evidence for all matters that need further investigation. Perform biometric capture with real-time feedback. Yes No With advanced ID management, all test taker biometric data are captured and securely managed to identify suspected proxy testers and prevent “blacklisted” test takers from entering the testing labs. 4. Operational Convenience Should Benefit Test Takers When Delivering High-Stakes Exams. The technology capabilities of computer-based testing and longer testing administration windows offer test takers greater conveniences. Test Taker Convenience Multi-Day Testing Single Day Testing Value Offer test takers a range of dates from which to select. Yes No Frequently, test takers may have challenges in their personal life that could interfere with a single date. This can impact them in many ways – even though they may be able to physically attend, they may be faced with emotional hardships that can affect their performance. Take into account accidents and unforeseen instances that may impact test takers. Yes No In every test administration window, there are test takers who have accidents or encounter weather/public action inconveniences that can make them miss the testing day. Multiple sessions help to mitigate these risks so that they can realise a successful testing event. Afford test takers flexibility so that they have the best chance to recover from a disaster. Yes No There is always the chance that significant events can disrupt a testing schedule, including everything from typhoons, strikes and social unrest. Wider testing administration windows offer greater flexibility for test takers so that they can test successfully despite such instances. 5. Valid, Reliable and Fair Tests Start with a Knowledge of Test Content Development. High-stakes tests must be of the highest quality, accurately measure the necessary knowledge and skills and be fair to all test takers. Developing psychometrically sound, legally defensible tests requires significant investment in time, knowledge and resource. Valid, Reliable and Fair Tests Multi-Day Testing Single Day Testing Value Develop test content in accordance with industry- recognised standards. Yes No The need for stricter content, statistical guidelines and validity evidence increases as the stakes regarding test use increase. Test development practices should meet the high standards of the ETS www.prometric.com

5. Third Quarter 2016 | Industry White Paper Standards for Quality and Fairness. The core of any test development process should adhere to industry best practices and support content validity and reliability of results. Conduct thorough psychometric analyses to keep exam performance sound. Yes No Psychometric analyses determine how effective test items are and how they perform. Each test and its items should be subjected to various analyses aimed at determining the quality of the assessment. Adopt an equating model when assessing the comparability of scores across different test forms. Equating Normalization The comparability of scores across different test forms of a standardised assessment has been a focus of educational measurement and the testing industry for the past 90 years. Normalisation is done based on the assumption that "in all multi- sessions of a test, the distribution of abilities of test takers is the same across all the sessions." There is no evidence supporting that test taker groups who take different forms are of equal ability. Equated scores are not affected by the problems that occur with the normalisation because equating adjusts for differences in the difficulty of test forms and test taker abilities. Item response theory (IRT) anchoring and calibration is used for hundreds of high-stakes testing programs around the world and has been demonstrated to be highly defensible to legal challenges of fairness in scoring. Secure all test development processes and exam content. Yes Yes Every precaution should be taken to prevent unauthorised disclosure of examination content and assure the integrity of examinations. All test items and item banks should be securely created, stored, accessed and delivered with full encryption to ensure maximum security of intellectual property. Executive Summary High-stakes examinations help derive important, life-changing decisions – assisting individuals to realize their human potential, be it a certification, assessment, licensure or entrance exam. While India has begun to embrace the digital era, it is critical as never before that licensing, certification and academic organisations rely on best-in-class testing and assessment processes to ensure the integrity and reliability of their testing programs. Testing and assessment service providers have the responsibility to ensure that the exam content they deliver on behalf of a test sponsor organisation as well as their test centers give candidates the opportunity to demonstrate their knowledge and qualifications accurately and fairly. Copyright© 2016 Prometric, Inc. All Rights Reserved. www.prometric.com