Myths and facts about online testing Eugene Burke Director Design Innovation

1. Myths and facts about online testingEugene BurkeDirector Design & Innovation

2. Overview Demand side characteristics � the context for what is happening Technology drives search for and expectancies of efficiencies Pressures on recruitment budgets War for talent and talent capture Supply side issues � the challenge to the industry Fakeability and cheating � impact on accuracy of assessment Validity and defensibility against a tight assessment window User experience and candidate care � expectations and equity in the technological age

3. Context Focus will be on high-stakes testing � testing that leads to decisions to hire or not to hire Focus will also be on the recruitment funnel � moving through large numbers of applicants through to smaller numbers of candidates to the final shortlist of hires Examples will be drawn from SHL�s experience over the past 48 months in developing and deploying applications in various countries world-wide

4. Demand side characteristics

5. Shift in value from technology

6. Supply side issues and challenges

7. High stakes testing & zones in the recruitment funnel

8. Assessment at the top of the funnel

9. Issues of cheating & fakeability Once past the killer questions, is it possible to screen on, say, competencies? Ellington, Sackett & Hough (1999) showed that Likert scales and social desirability corrections do not appear to work. Hunt and Warr (1999) showed that forced-choice (ipsative) formats give more accurate assessments. Bartram (1996) showed that when dimensions > 15, suitable degrees of freedom are given for statistical manipulations. Saville et al. (1996) showed that Likert and ipsative formats gave equivalent predictions of manager�s competence.

10. Evidence of value Story #1: Manufacturing Competency application questionnaire designed to tap Team Working, Flexibility, Reliability (Dependability) and Quality Orientation predicted employee attendance (low to zero absenteeism), work performance (productivity) and managers� ratings of behaviour and attitude (low to zero performance management problems) Story #2: Distribution, Graduate and Manager Selection Senior line managers ratings of direct reports on Managing Others, Building Relationships, Making Decisions and Improving the Business were all predicted from competency application form. Those scoring in the upper quartile on the application form were rated between 21% and 29% higher than their counterparts scoring in the lower quartile on the application form

11. Unsupervised ability testing online

12. Moving timed testing up the funnel

14. Cheating & verifiability

15. Unsupervised personality testing online

16. Is the validity if personality measures still an issue?

17. Let�s put that into context �

18. Moving personality up the funnel Study conducted by Bartram & Brown (2003) N=1809 with samples drawn from UK and Hong Kong Internet controlled versus paper-and-pencil supervised modes of administration Data were from real in vivo assessments Analyses showed scale reliabilities and scale relationships maintained

19. Figure shows the weighted average of effect sizes of the five matching pairs of samples. When averaged, effect sizes for most scales are smaller than quarter a sten and are not bigger than half a sten. Note that Effect sizes for different pairs of samples seem to be about as large as differences between the pairs. These analyses suggest that there is little if any systematic distortion in OPQ scores attributable to mode of supervision. Figure shows the weighted average of effect sizes of the five matching pairs of samples. When averaged, effect sizes for most scales are smaller than quarter a sten and are not bigger than half a sten. Note that Effect sizes for different pairs of samples seem to be about as large as differences between the pairs. These analyses suggest that there is little if any systematic distortion in OPQ scores attributable to mode of supervision.

20. To sum up �

21. Facts & myths covered Paper has explored issues in high-stakes internet testing Open � ipsative competency-based questionnaires Controlled � dynamic generated testlets Supervised to Controlled � ipsative personality (self-assessment) questionnaires Paper has sought to provide examples of how issues of cheating and fakeability can be tackled, and that the value of objective assessment is maintained Paper has also sought to show the process efficiencies and increased ROI that moving up the funnel offers clients

22. Challenges going forward Evidence to date shows that applicants have positive perceptions of structured internet delivered assessments � �shows someone is serious� � but � the issue of candidate care requires careful preparation of applicants prior to the assessment the issue of equity also need to be carefully marketed to applicants The assessments still need to be defensible � what is being predicted and how do we know it is important? What about user training � what should a user have access to and what do they need to make effective decisions? How does a client know that the �product� is fit-for-use and trustworthy � what evidence should a provider give to demonstrate that their application does handle the problems of open and controlled assessments? what is best practice?

23. Thank you for attending!