Correcting for indirect range restriction in meta analysis determining the u t distribution
Download
1 / 22

Correcting for Indirect Range Restriction in Meta-Analysis: Determining the u T Distribution - PowerPoint PPT Presentation


  • 287 Views
  • Uploaded on

Correcting for Indirect Range Restriction in Meta-Analysis: Determining the u T Distribution. Huy Le University of Central Florida In-Sue Oh University of Iowa. The Importance of Correcting for Study Artifacts in Meta-Analysis.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Correcting for Indirect Range Restriction in Meta-Analysis: Determining the u T Distribution' - giolla


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Correcting for indirect range restriction in meta analysis determining the u t distribution l.jpg

Correcting for Indirect Range Restriction in Meta-Analysis: Determining the uT Distribution

Huy Le

University of Central Florida

In-Sue Oh

University of Iowa


The importance of correcting for study artifacts in meta analysis l.jpg
The Importance of Correcting for Study Artifacts in Meta-Analysis

  • The major goal of meta-analysis is to estimate the true relationships between variables (constructs) from observed correlations.

  • These observed correlations, however, are influenced by the effects of study artifacts  Meta-analysts need to take these effects into account in order to accurately estimate the true correlations.


Range restriction as a study artifact l.jpg
Range Restriction as a Study Artifact

  • What is Range restriction?

    • Occurs when the variance of a variable in a sample is reduced due to pre-selection or censoring in some way (Ree, Carretta, Earles, & Albert, 1994).

  • Effects of range restriction

    • Statistics estimated in such a restricted sample (incumbent sample) are biased, attenuated estimates of parameters in the unrestricted population (applicant sample).


Two types of range restriction l.jpg
Two Types of Range Restriction

Direct Range Restriction

Explicit Selection on X resulting

in distortion of correlation between

X and Y

X

Y

Indirect Range Restriction

Explicit Selection on a third variable Z

resulting in distortion of correlation

between X and Y (which are correlated

with Z); Always the case for concurrent validation studies

Suitability scores

Z

X

Y

GATB

Multivariate Range Restriction

An extension of indirect range restriction where explicit selections

occur on several variables


Effects of direct and indirect range restriction on correlations l.jpg
Effects of Direct and Indirect Range Restriction on Correlations

Direct Range Restriction

(Sr= 20%)

Unrestricted Correlation

Rho=.60

Rxx =.90

Ryy =.52

R = .22

Indirect Range Restriction

(Sr= 20% on Z; Rxz=.66; Ryz=.30)

R = .41

R = .32


Correction for range restriction l.jpg
Correction for Range Restriction Correlations

  • Direct Range Restriction: Thorndike Case II

  • Indirect Range Restriction: Thorndike Case III

Notes:

Ux = 1/uX; uX = sdx/SDx = Range restriction ratio of X (the ratio of standard deviation of the independent variable X in the restricted sample to its standard deviation in the unrestricted population).

UZ = 1/uZ ; uZ = Range restriction ratio of Z (the third variable where explicit selection occurs).


Correction for range restriction7 l.jpg
Correction for Range Restriction Correlations

  • Problems related to correcting for the effect of range restriction in Meta-Analysis:

    • Most studies are affected by indirect range restriction.

    • However, information required to correct for this effect of indirect range restriction (shown in the previous) is often not available.

    • The problem is even worst for meta-analysts who have to rely on information reported by primary researchers.


New range restriction correction methods l.jpg
New Range Restriction Correction Methods Correlations

  • Recently, Hunter, Schmidt, and Le (2006) introduced a new procedure (CASE IV) to correct for range restriction.

  • The procedure requires information about:

    • uT: Range restriction ration on the true score T underlying X

    • Rxxa: Reliability of X estimated in the unrestricted population.

  • Simulation study shows that the method is accurate (Le & Schmidt, 2006), outperforming traditional approach of using direct range restriction correction (when range restriction is actually indirect) in most situations.

  • Using this procedure, the researchers showed that traditional estimates of the validity of the GATB were underestimated from 24% - 45%!


New range restriction correction methods9 l.jpg
New Range Restriction Correction Methods Correlations

Hunter, Schmidt, & Le (2006) model for the combined effects of indirect range restriction and measurement error:

uS

S

RTP

uT

T

P

True validity

RTX=(RXX )1/2

RXY

X

Y

uX


Slide10 l.jpg

New Method for Range Restriction (Case IV) Correlations

Two key characteristics:

Beforeapplying Thorndike’s

Case II,

+ Ut(instead of Ux)

+ Correction for measurement

error before RR correction

+ Applying Thorndike’s Case II

+ Reintroducing unreliability

in predictor to estimate

true validity


Applying the new correction approach to meta analysis l.jpg
Applying the New Correction Approach to Meta-Analysis Correlations

  • Problem: The information needed to apply the new procedure is not available in every primary study.

  • In the past, meta-analysts addressed that problem by using artifact distributions.

  • This approach allows corrections to be made even when information of the artifacts is not available in each primary study.


Difficulties in estimating the u t distribution l.jpg
Difficulties in Estimating the CorrelationsuTDistribution

Problem when applying the artifact distribution approach to correct for indirect range restriction in meta-analysis:

  • Need the uT artifact distribution but uT is unknown (unobservable – unlike uX)!

  • Hunter et al. (2006) suggested uT be estimated from uX and Rxx (reliability of the independent variable in the unrestricted population) using the formula:


The artifact distribution of u t l.jpg
THE ARTIFACT DISTRIBUTION OF CorrelationsUT

Very Rare Case! Less representative! Dependent on Rxxa! Sometimes, cannot be computed even when Rxxa and Ux are simultaneously available

Individual studies

Study 1: Rxxa1 uX1 uT1 N1 rxy1

Study 2: Rxxa2 N2 rxy2

Study 3: uX3 N3 rxy3

Study 4: Ryy4 N4 rxy4

………… ........ ……. …… …… ….. ……

Study k: Rxxak Ryyk uXk uTk Nk rxyk

Artifact Distributions

of Rxx Ryy uX uT


Difficulties in estimating the u t distribution14 l.jpg
Difficulties in Estimating the CorrelationsuTDistribution

  • Doing so, however, renders the resulting distribution of uT is highly dependent to Rxxa the assumption of independence of the artifact distributions is violated.

  • Further, there are values in the distributions of uX and Rxxa which cannot be combined. For example, when uX = .56 (equivalent to selection ratio of 40%) and Rxxa =.60, we cannot estimate the corresponding value of uT . Current practice is to disregard these values.

     The uT distribution estimated by the current approach may not be appropriate  Meta-analysis results may be affected.


Estimating the u t distribution l.jpg
Estimating the CorrelationsuT Distribution

  • Our solution:

    • To go backward: Instead of combining values uX and RXXin their respective distributions to estimate the values of the uTdistribution, we systematically examine the appropriateness of different “plausible uT distributions” in term of how closely they can reproduce the original uX distribution when combined with the RXX distribution.

    • This approach is logically appropriate because uX results from uT and RXX (see Hunter et al., 2006; Le & Schmidt, 2006), not the other way around as seemingly suggested by the formula.


Estimating the u t distribution16 l.jpg
Estimating the CorrelationsuT Distribution

  • Procedure: Five steps

    (1) Selecting a “plausible distribution” for uT ( ). This distribution includes a number of representative values of uT , together with their respective frequencies.

    (2) The values of are then combined with all the values of Rxxin its distribution using the following equation to calculate the corresponding uX values (equation 8, p. 422, Le & Schmidt, 2006):

    (3) The resulting values of uX form a distribution with frequency of each value being the product of the corresponding frequencies of and Rxx in their respective distributions.


Estimating the u t distribution17 l.jpg
Estimating the CorrelationsuT Distribution

  • Procedure: (cont.)

    (4) This uX distribution is then compared to the observed (original) distribution of uX, based on a pre-determined criterion. If they are close enough, as determined by the criterion, the process terminates and the “plausible distribution” of specified in step (1) becomes the estimated uX distribution. Otherwise, the process continues in step (5);

    (5) A new plausible distribution is constructed by keeping the original values of but systematically changing their frequencies. A new iteration is then started (by returning to step 2 above).

    A SAS program was developed to implement the procedure (the program is available from Huy Le).


Estimating the u t distribution18 l.jpg
Estimating the CorrelationsuT Distribution

  • Demonstration of the procedure:

  • Note that the uX distribution for cognitive tests derived by Alexander et al. (1989) and the Rxxa distribution derived by Schmidt and Hunter (1977) were used.


Result the u t distribution for cognitive measures l.jpg
Result: The u CorrelationsT Distribution for Cognitive Measures


Result the u t distribution for educational tests l.jpg
Result: The u CorrelationsT Distribution for Educational Tests


Discussion l.jpg
Discussion Correlations

  • Procedures to correct for indirect range restriction are necessarily complicated, but the procedure described in this paper will allow better, more accurate estimation of the uT distribution  More accurate meta-analysis results.

    • The uT distributions estimated here can be used by researchers in their future research.

    • Alternatively, meta-analysts can apply the current procedure to any situations where there are only sparse information about range restriction and reliabilities in their data (i.e., primary studies).


Thank you l.jpg
Thank you! Correlations

  • Any Questions or Comments?


ad