Psychometric Assessment: Tests, test use and preparation

1. Psychometric Assessment: Tests, test use and preparation 13th July 2017 Loughborough University

2. Overview • Part 1: • A quick recap on psychometrics • Employers’ use of tests • Helping students prepare • Part 2: • New developments • Future trends

3. Psychometric tests • “an objective and standardised measure of a sample of behaviour” Anastasi and Urbina (1997) • ‘psycho’ = mind • ‘metric’ = measurement

4. Psychometric tests • administered in a standardised way • scored and interpreted in a standardised way • reliable – meaning they are accurate • valid – meaning they are appropriate for purpose • rigorously developed according to best practice standards

5. Psychometric tests

6. Types of tests & assessment • Cognitive tests – learning, reasoning, capacity (what) • Personality – style, preferences (how) • Motivation – drive, energy, engagement (why) • SJTs / simulations – judgement, practical knowledge, ‘common sense’ (what & how)

7. Cognitive tests • Focus of much of today as they are the most… • widely used • widely researched • open to preparation • feared! • They cover tests of ability, aptitude and attainment

8. Verbal ability Abstract ability ‘g’ or general ability Numerical ability Types of cognitive test • Ability tests • Assess what a person is capable of doing • Typically involve ‘reasoning’ or problem-solving • May assess general or specific abilities

9. Types of cognitive test • Attainment tests • Assess what a person has learnt • e.g. educational qualifications, accountancy exams • Influenced by ability, so association • Aptitude tests • Abilities related to acquiring specific skills • Future-oriented, i.e. used to predict • In practice, often similar to ability tests

10. Personality and motivation • Self-report measures • No(?) ‘right’ or ‘wrong’ answers • Assess typical behaviours • Personality - distinction between ‘trait’ and ‘type’ • Motivation related to drives and values – increasing emphasis, especially on fit

11. SJTs • Can assess behaviours, skills, competencies • Based on understanding and responding to scenarios • Provide ‘low fidelity simulations’ • Relatively distinct from other assessments

12. SJTs

13. Structured interviews Astrology Unstructured interviews Graphology SJTs Ability tests Assessment centres References 0 Effectiveness 1 Self-assessment Personality assessments Why are tests used? • Validity • ‘Level playing field’ • Objectivity • Accuracy

14. Graduate population Employer’s local norm General population Test score How good is good enough?

15. Popularity • CIPD Resourcing and Talent (2017) • Competency-based interviews – 78% • Interviews on contents of CV / app form 74% • General ability tests – 41% • Assessment centres – 39% • Personality / aptitude questionnaires – 35% • Online tests – 23% • Gamification – 2% Based on 1068 UK organisations

16. Popularity • Over 95% of FTSE100 companies • c70% of SMEs • Tendency for greater use in private sector than public or voluntary sectors

17. Popularity – Gartner (formerly SHL) • Assessed more than 5 million people in over 160 countries in the last 12 months • Worked with 10,000 customers • More than 60% of the Fortune Global 500 used SHL: • 80% of Technology companies • 80% of Food and Beverage companies • 70% of Financial Services companies • More than 80% of the FTSE 100 used SHL, including every Telecommunications company • Current estimate of 25 – 30 million tests per year

18. Preparation and test performanceorPractice ≠ Preparation

19. Research on test preparation • Almost exclusively on cognitive tests and ‘exams’ • Partial and incomplete • Assumptions and ambiguities

20. Purpose of testing - employers • Efficiency • Managing risk • Creating a ‘level playing field’ • Fairness • Understanding potential • Finding the right fit • Attraction

21. Purpose of testing – job seekers • To ‘show myself off’ – it’s an opportunity • Do I have the right knowledge / skills / abilities? • Will I fit in? • Do I want this job enough?

22. Purpose of testing • It’s a two-way thing… • …though loaded in favour of the employer • It helps if it is seen as an opportunity, not a threat

23. What the research tells us - ability • ‘Familiarisation’ vs ‘coaching’ • Re-test effect, c0.5 standard deviations • Research during the 1980s by ETS • Test-specific familiarisation – modest effect • Coaching – variable, not as big as claimed • Numerical more susceptible than verbal • Effect of test complexity • Well designed prep, c4 hours, as effective as coaching courses

24. What the research tells us – personality & motivation • Less coachable than ability – job specific • Natural changes under assessment conditions • ‘Gaming’ the assessment • Fit • Validation through interview

25. What the research tells us - SJTs • Very little good research available • Re-test effects similar to cognitive tests • Familiarity is likely to be important – variation in question formats • Know the company – what do they want? • Typical behaviour and fit

26. Action Exploration Reflection Creation Helping students prepare

27. Action • What do I know about tests? • What tests am I being asked to take? • Get some practice

28. Reflection • Key questions to ask: • How did I do? • Did I understand the instructions properly? • How did I feel – confidence vs anxiety? • Did I manage speed vs accurately well? • What skills do I need to develop?

29. Reflection Speed Optimum performance Accuracy

30. Creation • Use Reflection points to set targets: • Know the test and its requirements • Confident and positive • Develop speed and accuracy • Brush-up on knowledge / skills assessed by test • Identify gaps in current knowledge

31. Exploration • Put it into practice • Practice tests • Information • Use feedback to inform approach

32. Action • Get some practice… • … and apply what I now know / can do

33. Test-taking skills Test performance Low High Ability Test-taking skills Weak Strong

34. Any questions?

35. Trends in Psychometric Assessment 13th July 2017 Loughborough University

36. Overview • Developments with current tests • Strengths-based assessment • Game-based assessment • Looking further ahead

37. Developments with current tests • More tests at all stages of recruitment • Technology is everywhere, increasing accessibility • Testing earlier on in recruitment process • Ability tests • High use • Online, unsupervised • Validation? • Item banks and adaptive testing

38. Developments with current tests • Personality assessment • Increasingly used as a screening tool • Bespoke versions, focus on key attributes • Can be used to derive competencies • SJTs • Typically used as an early filter • Most likely to be bespoke to organisation • Vary in terms of ‘fidelity’ • Most advanced ones are ‘game-like’

39. Going mobile • 45% of job seekers use their mobile to search for jobs daily • 89% believe a mobile device is an important tool for job searching • 48% think mobile will be the most common way to search for jobs in two years or less • 90% of the Fortune 500 company career sites do not support a mobile applications (Glassdoor surveys, 2014)

40. Tests going mobile

41. Strengths-based assessment • Identifies individual strengths, interests, motivators • May be an assessment or interview • Part of a wider recruitment process • Predictive of fit and retention • Open to preparation?

42. Strengths-based assessment

43. Game-based assessment • Some definitions • Games • Gamification • Game-based assessments • Attractive • Equitable? • Valid?

44. Looking further ahead • Diversity vs validity • Diversity is increasingly important to organisations • Traditional tests, esp ability, are not great here • Personality, simulations, GBAs are better here… • … but are they equally valid? • AI / natural language processing • Digital footprint analysis • Up to 90% of employers review candidates’ social media profiles • Rise of companies offering social media reports

45. Looking further ahead

46. Any questions? www.realisepotential.com www.bpstesttraining.co.uk