to use the teamwork test or not a psychometric evaluation n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
To Use the Teamwork Test -- Or Not? A Psychometric Evaluation PowerPoint Presentation
Download Presentation
To Use the Teamwork Test -- Or Not? A Psychometric Evaluation

Loading in 2 Seconds...

play fullscreen
1 / 25

To Use the Teamwork Test -- Or Not? A Psychometric Evaluation - PowerPoint PPT Presentation


  • 172 Views
  • Uploaded on

To Use the Teamwork Test -- Or Not? A Psychometric Evaluation. Janet L. Kottke California State University, San Bernardino Kimberly A. French University of South Florida Rhiannon J. Kirchner California State University, San Bernardino Presented to PTC of Southern California, July 23, 2013.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'To Use the Teamwork Test -- Or Not? A Psychometric Evaluation' - helmut


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
to use the teamwork test or not a psychometric evaluation

To Use the Teamwork Test -- Or Not? A Psychometric Evaluation

Janet L. Kottke

California State University, San Bernardino

Kimberly A. French

University of South Florida

Rhiannon J. Kirchner

California State University, San Bernardino

Presented to PTC of Southern California, July 23, 2013

tksa development
TKSA Development
  • Stevens and Campion (1994; 1999) develop the Teamwork KSA Test
    • 35 items, multiple choice format (4 options, dichotomous scoring)
  • Reviewed teamwork literature
    • Identified 14 KSAs identified with effective teamwork

Kottke, French, & Kirchner

PTC July 23, 2013

14 ksas 5 facets
14 KSAs → 5 facets
  • These 14 KSAs represent
    • Five facets
      • Conflict resolution
      • Collaborative problem solving
      • Communication
      • Goal setting and performance management
      • Planning and task coordination

Kottke, French, & Kirchner

PTC July 23, 2013

5 facets 2 dimensions
5 facets → 2 dimensions
  • Two higher-order dimensions:
    • Interpersonal KSAs
      • Conflict resolution
      • Collaborative problem solving
      • Communication
    • Self-management KSAs
      • Goal setting and performance management
      • Planning and task coordination

Kottke, French, & Kirchner

PTC July 23, 2013

teamwork ksa structure
Teamwork KSA Structure

O’Neill, Goffin, & Gellatly, 2012, p. 37

Kottke, French, & Kirchner

PTC July 23, 2013

sample question
Sample Question
  • When you set work goals for yourself or your work team, what are the best goals to set?

A. Set goals to "do your best."

B. Set general and broad goals.

C. Set specific and detailed goals.

D. Set easy and simple goals.

Kottke, French, & Kirchner

PTC July 23, 2013

criterion validity
Criterion validity
  • Several studies have found predictability of supervisor and observer ratings of team effectiveness (rs range .20 to .56)
    • Ellis, Bell, Ployhart, Hollenbeck, & Ilgen, 2005
    • Leach, Wall, Rogelberg, & Jackson, 2005
    • McClough& Rogelberg, 2003
    • Morgeson, Reider, & Campion, 2005
  • Mixed results in predicting team member peer ratings

Kottke, French, & Kirchner

PTC July 23, 2013

questions about predictive validity
Questions about predictive validity
  • Miller, 2001
    • Issues raised about Stevens & Campion validation studies
      • Team performance was predicted less well (r = .44) than task work (r = .56)
      • TKSA predicted team performance only slightly better than did an aptitude test
        • But: TKSA added incremental variance beyond GMA
      • Aptitude data suggest individual aptitude, not teamwork aptitude measured
    • Miller’s own study finds no significant relationship between team performance and TKSA scores

Kottke, French, & Kirchner

PTC July 23, 2013

questions about psychometrics
Questions about psychometrics
  • O’Neill, Goffin, & Gellatly (2012) examine the TKSA
    • Comprehensive analysis
      • Reviewed all known studies using TKSA
      • Classic test theory analysis
      • Confirmatory factor analysis
      • Exploratory factor analysis
      • Convergent, discriminant validation
      • Criterion validity analysis
    • Disappointing results all around

Kottke, French, & Kirchner

PTC July 23, 2013

our plan
Our plan
  • Replicate O’Neill CTT & CFA analyses with fresh samples
  • Data sources
    • Had used the TKSA Test for a group project in several classes (sample 1)
    • Collected new data to address unexpected finding from sample 1 that aptitude was negatively related to teamwork interest (sample 2)
    • Collected data for a thesis project on teamwork and task interdependence (sample 3)

Kottke, French, & Kirchner

PTC July 23, 2013

the 3 samples
The 3 samples
  • Students from a mid-size southwestern University.
    • Sample 1 = 251 college students
      • Cohort groups, upper level undergraduate-level psychology course, term-length group project.
    • Sample 2 = 279 college students
      • Recruited from psychology and business courses
    • Sample 3 =404 college students
      • From upper-level undergraduate courses in both psychology and business departments, either a long-term (i.e., several weeks) or term-length group project

Kottke, French, & Kirchner

PTC July 23, 2013

work experience of samples
Work experience of samples
  • Although students, vast majority were employed
    • 96% in sample 1
    • 82.4% in sample 2
    • 88.4% in sample 3
  • Work experience ranged from means of 5.5 to 7.3 years
    • SDs ranged from 5.6 to 6.3 years

Kottke, French, & Kirchner

PTC July 23, 2013

analyses conducted
Analyses conducted
  • Analytic Approach
    • In each sample we evaluated
      • Classical test theory
        • Item, subscale, and total measure properties,
          • Means, standard deviations, and item-total correlations
        • Subscale and total measure means, standard deviations, and internal reliability
      • Structure using categorical confirmatory factor analysis (Mplus)
        • One factor model, two factor model, and a five factor model
          • For multi-factor models, both a covaried model (all factors were allowed to correlate), as well as a higher-order model (includes higher-order general teamwork KSAfactor)

Kottke, French, & Kirchner

PTC July 23, 2013

supplemental
Supplemental
  • Supplemental analysis
    • To ensure psychometric issues are not a result of student samples, ran classical test analyses on the subset of working respondents from each sample.
    • Results from working samples were not substantially different than full samples
      • For example, average absolute difference in factor and total means was .11 for sample 1 [SD = .07], .21 for sample 2 [SD = .19], and .14 for sample 3 [SD = .21].
      • The average difference in inter-item correlations was .02 [SD = .01])
      • Thus, only the results from the full samples are presented here.

Kottke, French, & Kirchner

PTC July 23, 2013

results item means sds item total correlations
RESULTS: Item means, SDs, item total correlations

Kottke, French, & Kirchner

PTC July 23, 2013

more results scale means sds inter item correlation means kr 20s
More Results: Scale means, SDs, inter item correlation means, KR 20s

Kottke, French, & Kirchner

PTC July 23, 2013

confirmatory factor analysis results on following slide
Confirmatory Factor Analysis Results on following slide

Kottke, French, & Kirchner

PTC July 23, 2013

slide20

*p < .01, **p < .001

Kottke, French, & Kirchner

PTC July 23, 2013

discussion
Discussion
  • To be fair to the test authors
    • They recommend the subscales be used for training and developmental purposes only
    • In all of their publications, they use only total scores
    • So, whether the test authors intended a hierarchical structure is not a certainty
  • Yet, with their careful development that categorized the 14 KSAs into specific facets, one would expect meaningful structure
  • Criterion validity has been found, but issues here as well
    • TKSAtest correlates highly with general mental ability
    • Some incremental predictability

Kottke, French, & Kirchner

PTC July 23, 2013

conclusion
Conclusion
  • Very limited evidence for adequate psychometric properties
  • Use with caution
    • Recognize the test’s limitations

Kottke, French, & Kirchner

PTC July 23, 2013

thank you
Thank you

Kottke, French, & Kirchner

PTC July 23, 2013

questions
Questions?

Kottke, French, & Kirchner

PTC July 23, 2013

references
References

Kottke, French, & Kirchner

PTC July 23, 2013