1 / 25

To Use the Teamwork Test -- Or Not? A Psychometric Evaluation

To Use the Teamwork Test -- Or Not? A Psychometric Evaluation. Janet L. Kottke California State University, San Bernardino Kimberly A. French University of South Florida Rhiannon J. Kirchner California State University, San Bernardino Presented to PTC of Southern California, July 23, 2013.

helmut
Download Presentation

To Use the Teamwork Test -- Or Not? A Psychometric Evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. To Use the Teamwork Test -- Or Not? A Psychometric Evaluation Janet L. Kottke California State University, San Bernardino Kimberly A. French University of South Florida Rhiannon J. Kirchner California State University, San Bernardino Presented to PTC of Southern California, July 23, 2013

  2. TKSA Development • Stevens and Campion (1994; 1999) develop the Teamwork KSA Test • 35 items, multiple choice format (4 options, dichotomous scoring) • Reviewed teamwork literature • Identified 14 KSAs identified with effective teamwork Kottke, French, & Kirchner PTC July 23, 2013

  3. 14 KSAs → 5 facets • These 14 KSAs represent • Five facets • Conflict resolution • Collaborative problem solving • Communication • Goal setting and performance management • Planning and task coordination Kottke, French, & Kirchner PTC July 23, 2013

  4. 5 facets → 2 dimensions • Two higher-order dimensions: • Interpersonal KSAs • Conflict resolution • Collaborative problem solving • Communication • Self-management KSAs • Goal setting and performance management • Planning and task coordination Kottke, French, & Kirchner PTC July 23, 2013

  5. Teamwork KSA Structure O’Neill, Goffin, & Gellatly, 2012, p. 37 Kottke, French, & Kirchner PTC July 23, 2013

  6. Sample Question • When you set work goals for yourself or your work team, what are the best goals to set? A. Set goals to "do your best." B. Set general and broad goals. C. Set specific and detailed goals. D. Set easy and simple goals. Kottke, French, & Kirchner PTC July 23, 2013

  7. Criterion validity • Several studies have found predictability of supervisor and observer ratings of team effectiveness (rs range .20 to .56) • Ellis, Bell, Ployhart, Hollenbeck, & Ilgen, 2005 • Leach, Wall, Rogelberg, & Jackson, 2005 • McClough& Rogelberg, 2003 • Morgeson, Reider, & Campion, 2005 • Mixed results in predicting team member peer ratings Kottke, French, & Kirchner PTC July 23, 2013

  8. Questions about predictive validity • Miller, 2001 • Issues raised about Stevens & Campion validation studies • Team performance was predicted less well (r = .44) than task work (r = .56) • TKSA predicted team performance only slightly better than did an aptitude test • But: TKSA added incremental variance beyond GMA • Aptitude data suggest individual aptitude, not teamwork aptitude measured • Miller’s own study finds no significant relationship between team performance and TKSA scores Kottke, French, & Kirchner PTC July 23, 2013

  9. Questions about psychometrics • O’Neill, Goffin, & Gellatly (2012) examine the TKSA • Comprehensive analysis • Reviewed all known studies using TKSA • Classic test theory analysis • Confirmatory factor analysis • Exploratory factor analysis • Convergent, discriminant validation • Criterion validity analysis • Disappointing results all around Kottke, French, & Kirchner PTC July 23, 2013

  10. Our plan • Replicate O’Neill CTT & CFA analyses with fresh samples • Data sources • Had used the TKSA Test for a group project in several classes (sample 1) • Collected new data to address unexpected finding from sample 1 that aptitude was negatively related to teamwork interest (sample 2) • Collected data for a thesis project on teamwork and task interdependence (sample 3) Kottke, French, & Kirchner PTC July 23, 2013

  11. The 3 samples • Students from a mid-size southwestern University. • Sample 1 = 251 college students • Cohort groups, upper level undergraduate-level psychology course, term-length group project. • Sample 2 = 279 college students • Recruited from psychology and business courses • Sample 3 =404 college students • From upper-level undergraduate courses in both psychology and business departments, either a long-term (i.e., several weeks) or term-length group project Kottke, French, & Kirchner PTC July 23, 2013

  12. Work experience of samples • Although students, vast majority were employed • 96% in sample 1 • 82.4% in sample 2 • 88.4% in sample 3 • Work experience ranged from means of 5.5 to 7.3 years • SDs ranged from 5.6 to 6.3 years Kottke, French, & Kirchner PTC July 23, 2013

  13. Kottke, French, & Kirchner PTC July 23, 2013

  14. Analyses conducted • Analytic Approach • In each sample we evaluated • Classical test theory • Item, subscale, and total measure properties, • Means, standard deviations, and item-total correlations • Subscale and total measure means, standard deviations, and internal reliability • Structure using categorical confirmatory factor analysis (Mplus) • One factor model, two factor model, and a five factor model • For multi-factor models, both a covaried model (all factors were allowed to correlate), as well as a higher-order model (includes higher-order general teamwork KSAfactor) Kottke, French, & Kirchner PTC July 23, 2013

  15. Supplemental • Supplemental analysis • To ensure psychometric issues are not a result of student samples, ran classical test analyses on the subset of working respondents from each sample. • Results from working samples were not substantially different than full samples • For example, average absolute difference in factor and total means was .11 for sample 1 [SD = .07], .21 for sample 2 [SD = .19], and .14 for sample 3 [SD = .21]. • The average difference in inter-item correlations was .02 [SD = .01]) • Thus, only the results from the full samples are presented here. Kottke, French, & Kirchner PTC July 23, 2013

  16. RESULTS: Item means, SDs, item total correlations Kottke, French, & Kirchner PTC July 23, 2013

  17. More Results: Scale means, SDs, inter item correlation means, KR 20s Kottke, French, & Kirchner PTC July 23, 2013

  18. Confirmatory Factor Analysis Results on following slide Kottke, French, & Kirchner PTC July 23, 2013

  19. *p < .01, **p < .001 Kottke, French, & Kirchner PTC July 23, 2013

  20. Discussion • To be fair to the test authors • They recommend the subscales be used for training and developmental purposes only • In all of their publications, they use only total scores • So, whether the test authors intended a hierarchical structure is not a certainty • Yet, with their careful development that categorized the 14 KSAs into specific facets, one would expect meaningful structure • Criterion validity has been found, but issues here as well • TKSAtest correlates highly with general mental ability • Some incremental predictability Kottke, French, & Kirchner PTC July 23, 2013

  21. Conclusion • Very limited evidence for adequate psychometric properties • Use with caution • Recognize the test’s limitations Kottke, French, & Kirchner PTC July 23, 2013

  22. Thank you Kottke, French, & Kirchner PTC July 23, 2013

  23. Questions? Kottke, French, & Kirchner PTC July 23, 2013

  24. References Kottke, French, & Kirchner PTC July 23, 2013

More Related