Measurement of Variables

1. Measurement of Variables

2. Operational Definition • A description of the ways a research will observe and measure a variable, so called because it specifies the operations that will be taken into account to measure the variable • Typical research lingo “How will you operationalize that variable?

3. Operational Defintions • Identify operational definitions of the following latent constructs: • Intelligence of an individual • Market value of a firm • Employee theft • Organizational performance • Accounting Fraud • Customer retention • Team diversity

4. Reliability

5. What is Reliability? • Consistency, dependability, or stability in measurements over observations or time • Degree to which we measure something without error (It is a theory of error) • Necessary but not sufficient condition for validity • Reliability forms the upper bound for validity

6. Classical Test Theory (True Score Model) X = T + E T = score or measurement that would be obtained under perfect conditions E = error because measurement is never perfect, assumes error is random `X expected to equal`T because`E = 0, thus E(X) = E(T)

7. Reliability - Variance Decomposition s2x = s2t + s2e In other words, the total variance associated with any measure is equal to true score variance plus error variance Where does error variance come from?

8. Correlation Between Observed and True Scores rxt= st2 sx2 In other words, the theoretical definition of reliability: Reliability is the portion of the overall variance in a measure which is true score variance

9. Correlation Between Observed and True Scores • \Correlation equal to the ratio of the standard deviation of true scores to the standard deviation of observed scores • Squared correlation indicates proportion of variance in observed scores due to true differences among people rxt2 = rxx = st2 / sx2

10. Reliability • Since rxt2 = rxx = st2 / sx2 • Reliability defined as ratio of true score variance to observed-score variance • Square root of reliability is correlation between observed and true scores - called the reliability index

11. Reliability s2x = s2t + s2e so, s2t = s2x - s2e reliability = s2t / s2x so by substitution, reliability = (s2x - s2e) / s2x so, reliability = 1 - (s2e/s2x) so, reliability can range from 0 to 1 and if rxx = .80 then 80% variance is systematic Does not distinguish between true variance and systematic error In reliability theory, unsystematic variance = error

12. Types of Reliability Estimates • Equivalent Forms • Test-Retest • Internal Consistency • Interrater/Interobserver (not considered a true model of reliability) • Differ in how they treat error

13. Test Retest • Test Retest - Observations taken at two different time periods using the same measure are correlated • Coefficient of stability • Error includes anything that changes between one administration of measures and next (including real changes) • Not good for unstable constructs

14. Equivalent Forms • Equivalent Forms - Observations using two different forms of measurement are correlated • Coefficient of equivalence (and stability if forms are administered at different times) • Error includes any changes between administrations plus anything that varies between one form and next (including real changes over time) • Not good for unstable constructs (e.g., mood) unless measures are taken within a short period of time

15. Internal Consistency • Internal Consistency - Items within a given measure are correlated • Coefficient of equivalence • Split-half • Even-odd • Early-late • Random split • Coefficient alpha • Does not assess stability of a measure over time

16. Split-Half • Can be viewed as variation of equivalent forms • Measures within a given form are split and correlated • Because correlation is based half of the items, can correct using Spearman-Brown rxx = 2r1/2 1/2 / (1 + r1/2 1/2) Where r1/2 1/2 is the correlation between halves • Flawed approach (many possible halves)

17. Cronbach’s Alpha (Coefficient Alpha) • Can be derived from Spearman-Brown formula • Assumes measures are homogenous • Average of all possible split-half reliability coefficients of a given measure • Does NOT tell you if there is a single factor or construct in your scale (does NOT indicate unidimensionality)

18. Coefficient Alpha k = number of items (or measures) Alternatively, the formula can be written: Where SC is the sum of all items in a covariance matrix With this formula, what effect should k have?

19. Internal Consistency • Number of items usually increases internal consistency reliability (if items are of equal quality) • Increased correlations between items increases internal consistency reliability • With enough items, alpha can be high even when correlations between items is low • Alpha should NOT be taken to indicate that the scale is unidimensional • Benchmark for “acceptable” alpha = .7 (Nunnally and Bernstein, 1994)

20. Interobserver or Interrater • Interobserver or Interrater - Measures between different raters are correlated • Coefficient of equivalence for raters • Not the same as agreement (agreement is absolute, but reliability is correlational/ relational) • Error includes anything that leads to differences between observer/judge ratings

21. Interrater Agreement • Often we need a measure of agreement - kappa is a measure of agreement between two judges on categories • Researchers often use % agreement - not appropriate • % agreement doesn’t take into account chance levels (e.g., two categories, chance = 50%) • kappa deals with this by using a contingency table • Use kappa if you need to measure interrater agreement on categorical coding

22. Interrater Agreement • kappa - a measure of agreement between two observers taking into account agreement that could occur by chance (expected agreement). kappa =  Observed agreement - Expected agreement  --------------------------------------  100% - Expected agreement

23. Interater Agreement vs Reliability • High reliability, low agreement (job applicants rated by interviewers on 1-5 scale) App 1 App 2 App3 Intv1 5 4 4 Intv2 4 3 3 Intv3 3 2 2

24. Considerations • Nature of construct being measured • Sources of error relevant to construct • Test/retest - all changes due to time • Equivalent forms - all changes due to time and all differences between forms • Internal consistency - differences between items within a measure • Interrater/Interobserver - differences between observers and judges

25. Standards of Reliability • Early stages of research in an area, lower reliabilities may be acceptable • Higher reliabilities are required when measures used to differentiate among groups • Extremely high reliabilities required when making important decisions • Rule of thumb = .7 minimum reliability; preferably .8+ (Nunnally & Bernstein, 1994) but .95+ may be TOO high

26. Effects of Unreliability • Attenuation of correlations rxy = r*xyÖ(rxxryy) r*xy=correlation between true scores of X,Y • Correction for attenuation r*xy= rxy/Ö(rxxryy)

27. Assignment 3 • Look at the hypothesis statements you wrote last week • For each variable (independent, dependent, mediator, moderator) from last week, discuss how you would operationally define and measure this variable. After you have identified your measure, indicate how you would assess the reliability of your measure, if possible. If it is not possible to assess reliability, why not?