group level measurement n.
Skip this Video
Loading SlideShow in 5 Seconds..
Group-Level Measurement PowerPoint Presentation
Download Presentation
Group-Level Measurement

Loading in 2 Seconds...

play fullscreen
1 / 51

Group-Level Measurement - PowerPoint PPT Presentation

  • Uploaded on

Group-Level Measurement. Katherine Klein University of Pennsylvania CARMA Presentation February 2007. Why Group-Level Measurement?. Burgeoning of multilevel theory and research in last 25 years Great progress in conceptualizing and measuring group-level constructs

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Group-Level Measurement

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
group level measurement

Group-Level Measurement

Katherine Klein

University of Pennsylvania

CARMA Presentation

February 2007

why group level measurement
Why Group-Level Measurement?
  • Burgeoning of multilevel theory and research in last 25 years
  • Great progress in conceptualizing and measuring group-level constructs
        • Especially shared constructs
  • Continuing challenges and opportunities
        • Especially regarding configural constructs
a few terms and assumptions
A Few Terms and Assumptions
  • I’ll refer to groupsbut much or all of what I say will apply as well to organizations, departments, stores, etc.
  • I’ll focus on the creation and use of original survey measures to assess group constructs.
  • I’ll address statistical issues in passing only.
    • But see past CARMA presenters including James LeBreton, Gilad Chen, Paul Bliese, Dan Brass, Steve Borgatti, and others
  • Fundamentals: Theory First
    • Construct Types: Global, Shared, and Configural Constructs
  • Practicalities and Technicalities
    • Survey Wording
    • Sampling
    • Qualitative Groundwork
    • Single-source Bias
    • Justifying Aggregation
  • Opportunities and Challenges
    • The Configuration of Diversity
    • Social Network Analysis
fundamentals theory first
Fundamentals: Theory First
  • Constructs are our building blocks in developing and in testing theory.
  • High quality measures are construct valid.
  • The development of construct valid measures thus begins with careful construct definition.
  • Group-level constructs describe the group as a whole and are of three types (Kozlowski & Klein, 2000):
    • Global, shared, or configural.
global constructs
Global Constructs
  • Relatively objective, easily observable, descriptive group characteristics.
  • Originate and are manifest at the group level.
  • Examples:
    • Group function, size, or location.
  • No meaningful within-group variability.
  • Measurement is generally straightforward.
shared constructs
Shared Constructs
  • Group characteristics that are common to group members
  • Originate in group members’ attitudes, perceptions, cognitions, or behaviors
    • Which converge as a function of attraction, selection, socialization, leadership, shared experience, and interaction.
  • Within-group variability predicted to be low.
  • Examples:
    • Group climate, norms, leader style.
  • Measurement challenges are well understood.
configural group level constructs
Configural Group-Level Constructs
  • Group characteristics that describe the array, pattern, dispersion, or variability within a group.
  • Originate in group member characteristics (e.g., demographics, behaviors, personality, attitudes)
    • But no assumption or prediction of convergence.
  • Examples:
    • Rates, diversity, fault-lines, social networks, team mental models, team star or weakest member.
  • Measurement challenges are less well understood.
a related framework chan s 1988 composition typology
A Related Framework: Chan’s (1988) Composition Typology
  • Shared Constructs
    • Direct consensus models (e.g., group norms)
    • Referent shift models (e.g., team efficacy)
  • Configural Constructs
    • Dispersion model (e.g., climate strength)
    • Additive models (e.g., mean group member IQ)
  • Multilevel, Homologous Models
    • Process model (e.g., efficacy-performance relationship)
construct definition complexities an example shared leadership
Construct Definition Complexities:An Example: Shared Leadership
  • Shared leadership
    • “A dynamic, interactive influence process among individuals in work groups in which the objective is to lead one another to the achievement of group goals… [It] involves peer, or lateral, influence and at other times involves upward or downward hierarchical influence”
        • Conger & Pearce, 2003, p. 286
  • Is this a shared construct, or a configural construct, or … ?
construct definition complexities an example shared leadership1
Construct Definition Complexities:An Example: Shared Leadership
  • Well, how would you measure it?
    • Shared team leadership as a shared construct
      • “Team members share in the leadership of this team.”
      • “Many team members provide guidance and direction for other team members.”
    • Shared team leadership as a configural construct (network density):
      • “To what extent do you consider _____ an informal leader of the team?”
construct definition complexities an example shared leadership2
Construct Definition Complexities:An Example: Shared Leadership
  • Calling it a “referent shift” construct is not the answer.
    • Referent shift is a measurement strategy, not a construct type
  • Shifting the referent in an unthinking manner can be quite problematic:
    • The members of my team…
      • “Express confidence that we will achieve our goals”
      • “Will recommend that I am compensated more if I perform well”
      • “Are friendly and approachable”
      • “Rule with an iron hand”
a quick recap
A Quick Recap
  • Theory first: Define and explain the nature of your group-level constructs.
    • Is it a clearly objective description of the group?
      • If yes, a global construct.
    • Do you expect within-group agreement?
      • If yes, a shared construct.
    • Does it describe the group in terms of the pattern or array of group members on a common attribute?
      • If yes, a configural construct.
now what
Now What?
  • Having defined your constructs, the goal is to create measures that:
    • Are construct valid
    • Show homogeneity within (shared constructs)
    • Show variability between (all group-level constructs)
  • Practicalities and technicalities
    • Survey wording
    • Sampling
    • Qualitative groundwork
    • Minimizing single-source bias
    • Testing for aggregation
survey wording global constructs
Survey Wording:Global Constructs
  • Draw attention to objective descriptions of each group.
  • Gather data from experts and observers (SMEs) who can provide valid information about the groups in question.
      • No need to gather data from individual respondents within groups
  • Use language that fits your sample.
survey wording shared constructs
Survey Wording: Shared Constructs
  • Draw attention to shared group characteristics
  • Use a group referent rather than individual referent to enhance:
    • Within group agreement
    • Between group variability
    • Predictive validity
  • Gather data from individual respondents so within-group agreement can be assessed.
  • Actual consensus methods (discussion prior to group survey completion) work well but are labor-intensive.
survey wording configural constructs
Survey Wording:Configural Constructs
  • Draw attention to individual group member characteristics by using an individual referent.
  • Gather data from experts and observers (SMEs) who can provide valid information regarding individual group members, or gather data from individual respondents within groups.
  • The challenge is perhaps less in the survey wording than in operationalizing the array or pattern of interest.
  • Substantial between-group variability is essential. Seek samples in which groups vary considerably on the constructs of interest
      • Whether they are global, shared, or configural.
  • Statistical power reflects both:
    • Group sample size (n of groups)
    • Within-group sample size
      • When group size is large (number of respondents per group), measures of shared constructs are more reliable.
  • More research needed on power in multilevel analyses.
qualitative groundwork
Qualitative Groundwork
  • The survey wording and sampling guidelines seem fairly obvious and easy, but …
  • Check your assumptions in the field prior to survey data collection.
    • Are you measuring the right “groups”?
      • Example: Grocery stores or departments?
    • Is there meaningful between-group variability?
      • Example: Fast food chain
    • Are you measuring the right variables, and not too many of them?
      • Beware the blob.
single source bias
Single-Source Bias
  • Group-level correlations between measures of shared group constructs may be disturbingly high.
    • Examples:
      • Transformational and transactional leadership
      • Task, emotional, and procedural conflict
  • Aggregation does not “average away” response biases.
  • Rather, group members may share response biases
    • Halo, logical consistency, social desirability
  • Response bias may be particularly influential when respondents must make subtle distinctions among constructs.
single source bias beating the blob
Single-Source Bias:Beating the Blob
  • Survey measures
    • Choose and measure truly distinct constructs
    • Use different survey response formats
  • Survey design
    • Keep survey items measuring distinct constructs separate.
      • Help respondents recognize the distinction between leadership types, or conflict types, for example.
single source bias beating the blob1
Single-Source Bias:Beating the Blob
  • Survey analysis
    • Randomly split the within-group sample of respondents during data analysis.
      • All receive the same survey, but half provide IV and the other half provide the DV for analyses
  • Survey administration
    • Randomly split the within-group sample of respondents during data administration.
      • Respondents receive distinctive surveys. Half receive the IV survey and the other half receive the DV survey.
a quick recap1
A Quick Recap
  • Having
    • Defined our constructs
    • Written our survey items
    • Conducted qualitative groundwork
    • Sampled appropriately
    • Taken steps to reduce single source bias
  • We’re almost ready for hypothesis testing
  • But first: We need to justify aggregation
justifying aggregation
Justifying Aggregation
  • Why is this essential?
    • In the case of shared constructs, our very construct definitions rest on assumptions regarding within- and between-group variability.
    • If our assumptions are wrong, our construct “theories,” our measures, and/or our sample are flawed and so are our conclusions.
  • So, test both:
    • Within group agreement
      • The construct is supposed to be shared, but is it really?
    • Between group variability (reliability)
      • Groups are expected to differ significantly, but do they really?
justifying aggregation r wg j
Justifying Aggregation: rwg(j)
  • Developed by James. Demaree, & Wolf (1984)
  • Assesses agreement in one group at a time.
  • Compares actual to expected variance.
  • Answers the question:
    • How much do members of each group agree in their responses to this item (or this scale)?
  • Highly negatively correlated with the within group standard deviation
  • Valid values range from 0 to 1
  • Rule of thumb: rwg(j) of .70 or higher is acceptable
justifying aggregation r wg
Justifying Aggregation: rwg
  • Common to report average or median rwg(j) for each group for each variable:
    • If rwg(j) is below .70 for one or more groups, check:
      • Does the group have low rwg(j) values on several variables?
      • Do many groups have low rwg(j) values on this variable?
  • Remember: rwg(j) indicates within-group agreement, not between-group variability.
  • Beware: When variance in a group exceeds expected variance, out of range rwg(j) result.
justifying aggregation h 2
Justifying Aggregation: h2
  • Assesses between-group variance relative to total variance, across the entire sample.
  • Based on a one-way ANOVA
  • Answers the question:
    • To what extent is variability in the measure predictable from group membership?
  • The F-test provides a test of significance
    • The larger the sample of individuals, the more likely eta2 is to be significant.
  • Beware: h2 may be inflated when group sizes are small (under 25 individuals per group)
    • But, this is an easy way to begin tests of aggregation
justifying aggregation icc 1
Justifying Aggregation: ICC(1)
  • Assesses between-group variance relative to total variance
  • Based on a one-way ANOVA
  • Answers the question:
    • To what extent is variability in the measure predictable from group membership?
  • The F-test provides a test of significance
  • Based on h2 but controls for the number of predictors relative to the total sample size, so ICC(1) is not biased by group size.
justifying aggregation icc 2
Justifying Aggregation: ICC(2)
  • Assesses the reliability of the group means (i.e., between-group variance) in a sample, based on ICC (1) and group size.
  • Answers the question:
    • How reliable are between-group differences on the measure?
  • Reflects ICC(1) and within-group sample size
    • Example: If ICC(1) = .20 and:
      • Mean group size is 5, expected ICC(2) = .56
      • Mean group size is 20, expected ICC(2) = .71
a quick recap2
A Quick Recap
  • The hope is that we have successfully:
    • Defined our constructs.
    • Written our survey items.
    • Conducted qualitative groundwork.
    • Collected data from a large sample of groups.
    • Taken steps to reduce single source bias.
    • Justified aggregation.
    • And moved on to test our hypotheses.
  • So, what remains?
opportunities and challenges the configuration of diversity
Opportunities and Challenges:The Configuration of Diversity
  • Configural constructs describe the array, pattern, dispersion, or variability within a group.
    • The easy example is diversity
      • Demographic diversity
      • Climate strength
  • But even the easy example isn’t so easy: What is the definition of diversity? And how should it be measured?
the configuration of diversity
The Configuration of Diversity
  • A starting definition of diversity:
    • The distribution of differences among the members of a group with respect to an attribute, X, such as age, ethnicity, conscientiousness, positive affect or pay.
  • Okay, but what’s maximum diversity?
    • Which team has maximum age diversity?
      • 20, 20, 20, 70, 70, 70
      • 20, 30, 40, 50, 60, 70
      • 20, 20, 20, 20, 20, 70
      • 20, 70, 70, 70, 70, 70
the configuration of diversity1
The Configuration of Diversity
  • Diversity isn’t one thing.
  • It’s three things: Separation, Variety, or Disparity
  • The three types differ in:
    • Meaning or substance
    • Pattern or shape
    • Likely consequences
    • Appropriate operationalization
  • Blurring across these distinctions leads to fuzzy theory, misguided operationalizations, and potentially invalid research conclusions
the configuration of diversity example three research teams
The Configuration of DiversityExample: Three Research Teams
  • Team S
    • Members differ in their view of qualitative research.
      • Half of the team members respect it, half don’t.
  • Team V
    • Members differ in their discipline.
      • 1 psychologist, 1 sociologist, 1 anthropologist, etc.
  • Team D
    • Members differ in their rank
      • 1 senior professor, others are incoming graduate students.
diversity as separation
Diversity as Separation
  • Differences in group members’ position, attitude, or opinion along a continuum
  • Min: Every member has the same opinion
  • Max: Two polarized extreme factions
  • Theory: Similarity-attraction
  • Operationalization: Standard deviation
diversity as variety
Diversity as Variety
  • Differences in kind or category
  • Min: Every member is the same type
  • Max: Each group member is a different type
  • Theory: Requisite variety, cognitive resource heterogeneity
  • Operationalization: Blau’s index of categorical differences
diversity as disparity
Diversity as Disparity
  • Differences in concentration or proportion of valued assets or resources
  • Min: Every member has an equal portion of the resource
  • Max: One member is “rich” and all others are “impoverished”
      • Note: Disparity is asymmetric
  • Theory: Inequality, relative deprivation, tournament compensation
  • Operationalization: Coefficient of variation (SD/Mean)
the configuration of diversity a recap
The Configuration of Diversity:A Recap
  • Theory first
    • Separation is about position, attitude, or opinion
        • At maximum: Polarized factions
    • Variety is about knowledge or information.
        • At maximum: One of a kind
    • Disparity is about resources or power.
        • At maximum: One towers over others
  • Operationalize accordingly
    • The coefficient of variation is not a default or catch-all
opportunities and challenges social network analysis
Opportunities and Challenges: Social Network Analysis
  • Multilevel analysis and social network analysis have developed along separate paths.
  • Rich opportunities for cross-fertilization.
  • Social network analysis provides a means to conceptualize and operationalize configural constructs.
    • Illuminating the pattern or array of interpersonal ties within a group
opportunities and challenges social network analysis1
Opportunities and Challenges: Social Network Analysis
  • Many of our shared constructs appear to rest on tacit, often fuzzy, assumptions about interpersonal ties with groups.
      • Examples: Cohesion, communication, coordination, knowledge sharing, shared leadership, conflict
  • But we know little about the configuration of interpersonal ties – the structures – that underlie our shared constructs and measures.
an example social network analysis and shared team conflict
An Example: Social Network Analysis and Shared Team Conflict
  • When teams report high task or emotional conflict, what is the structure of interpersonal ties within the team?
  • As a starting point:
    • How dense are positive (advice) ties?
    • How dense are negative (difficulty) ties?
an example social network analysis and shared team conflict1
An Example: Social Network Analysis and Shared Team Conflict
  • Task and emotional conflict: The blob
    • r = .83
  • Advice density and negative tie density: More weakly correlated
    • r = -.36
  • Task conflict (mean task and emotional conflict), advice density, and negative tie density
    • Team Conflict and Advice Density: r = -.47
    • Team Conflict and Difficulty Density r = .40
social network analysis a recap
Social Network Analysis:A Recap
  • Social network analysis illuminates the configuration of interpersonal ties in groups.
    • What network structures underlie our shared constructs and measures?
    • Do network measures provide incremental validity?
  • Not just density, but centralization, cliques, and more.
  • What explains between-group differences in network structures?
in conclusion
In Conclusion
  • Theory first. Define your constructs.
    • Are they global, shared, or configural?
  • Measure constructs and collect data with care
    • Match item wording to the construct
    • Conduct qualitative groundwork
    • Sample appropriately
    • Take steps to reduce single source bias
    • Test for aggregation
  • Studying configural constructs remains a challenge and an opportunity
    • Conceptualizing and measuring diversity
    • Integrating social network analysis within our arsenal
some helpful references
Some Helpful References
  • Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research and methods in organizations (pp. 349-381). San Francisco: Jossey-Bass.
  • Borgatti, S. P. (2003). The network paradigm in organizational research: A review and typology. Journal of Management, 29, 991-1013.
  • Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234-246.
  • Harrison, D. A. & Klein, K. J. (2007). What’s the difference? Diversity as separation, variety, or disparity in organizations. Academy of Management Review.
  • Harrison, D. A. & McLaughlin, M. E. (1996). Structural properties and psychometric qualities of organizational self-reports: Field tests of connections predicted by cognitive theory. Journal of Management, 22, 313-338.
  • James, Demaree, & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85-98.
some helpful references1
Some Helpful References
  • Klein, K. J., Conn, A. B., Smith, B., & Sorra, J. S. (2001). Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. Journal of Applied Psychology, 86, 3-16.
  • Klein, K. J., Conn, A. B. & Sorra, J. S. (2001). Implementing computerized technology: An organizational analysis. Journal of Applied Psychology, 86, 3-16.
  • Kozlowski, S. W. J. & Klein, K. J. (2000). A multilevel approach to theory and research in organizations. In Klein, K. J. & Kozlowski, S. W. J. (Eds.), Multilevel theory, research, and methods in organizations (pp. 3-90). San Francisco: Jossey-Bass.
  • Morgeson, F. P. & Hofmann, D. A. (1999). The structure and function of collective constructs: Implications for multilevel research and theory development. Academy of Management Review, 24, 249-265.
  • Ostroff, C., Kinicki, A. J., & Clark, M. A. (2002). Substantive and operational issues of response bias across levels of analysis: An example of climate-satisfactoin relationships. Journal of Applied Psychology, 87, 355-368.