Lecture 10: Workplace Application – Skills & Abilities Testing

1. Lecture 10: Workplace Application – Skills & Abilities Testing PSY 605

2. The Workplace Applications - Overview

3. Workplace Application – Skills & Abilities Testing • In selection contexts… • Tests of cognitive ability • Tests of job-related skills • In performance appraisal & management contexts… • Tests of job performance • Tests of ‘potential’

4. Cognitive Ability Testing • General cognitive ability (g) long considered the strongest single predictor of performance across jobs (Hunter & Hunter, 1984) • Coefficients, after corrections for unreliability, reported in meta as .51 (performance ratings) and .56 (training performance) (Schmidt & Hunter, 1998) • Though traditional tests of g are not without controversy… (Murphy et al., 2003) • Diversity issues

5. Not-so-general Cognitive Ability Testing “Thinking critically is the primary responsibility of any manager, in any organization, and a leader’s capacity to engage in this process is largely determined by his or her intelligence.” (p.102, Menkes, 2005) • The validity findings on g are based on general cog. ability tests; however validity can be even stronger when more targeted assessment tools are used • Example: the ‘executive intelligence interview’ presented in Menkes (2005)

6. Job-related Skills Testing • Work Sample Testing: applicant performs set of tasks that are physically and/or psychologically similar to those on job (Ployhart, 2005; Roth, Bobko, & McFarland, 2005) • Examples for managerial jobs: the Leaderless Group Discussion; In-Basket technique • Assessment Centers (e.g., Arthur et al., 2003; Thornton & Gibbons, 2009) • Conceptually, multi-dimensional work sample testing • Tests that most closely mimic job performance will (in general) be the best predictors of future job performance • Tend to hold up well in court – easy to demonstrate job-relatedness, most applicants view as fair

7. Tests of Performance • Performance appraisal = • Formal process used to evaluate what individual workers do at work (Brannick, Levine, & Morgeson, 2007) • Systematic description of job-relevant strengths & weaknesses within & between employees or groups (Cascio & Aguinis, 2010) • Performance Appraisal tools: • Various methods of performance rating systems – validity of tool depends on its foundation (must begin with sound job analysis!) • BARS: Behaviorally-anchored rating scales • BOS: Behavioral Observation Scales

8. BARS Example

9. BOS Example

10. How well can performance be captured? • Objectively… • Very rarely are objective indicators even available • When they are available, they never capture full construct domain (limiting validity from the start) • Subjectively… • Performance ratings are only weak predictors of true performance (e.g., Murphy, 2008) • Ratings impacted by many sources of error (some random, most systematic), including motivations of rater

11. What’s a psychometrically trained I/O psychologist to do? 1) Define performance: • Start with the foundational best practices of measurement: Define your construct and testing purpose (use job analysis to define job performance) 2) Develop behaviorally-based Tool • BARS or BOS best options 3) ID Sources of error • Identify any sources of error in performance appraisal 4) Attack Sources of Error • Provide raters with incentives, tools, and opportunities to observe & recall performance accurately & honestly 5) Know your Limits • Limit inferences to those supported by validation of your tool