Loading in 5 sec....

Overlooking Stimulus VariancePowerPoint Presentation

Overlooking Stimulus Variance

- By
**davis** - Follow User

- 105 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about ' Overlooking Stimulus Variance' - davis

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

### Overlooking Stimulus Variance

### The end error

Jake Westfall

University of Colorado Boulder

Charles M. Judd David A. Kenny

University of Colorado Boulder University of Connecticut

Cornfield & Tukey (1956):“The two spans of the bridge of inference”

50 University of Colorado undergraduates;

40 positively/negatively valencedEnglish adjectives

My actual

samples

All non-neutral visual stimuli

50 University of Colorado undergraduates;

40 positively/negatively valencedEnglish adjectives

Ultimate targets

of generalization

My actual

samples

All non-neutral visual stimuli

All CU undergraduates taking

Psych 101 in Spring 2014;

All short, common, strongly

valenced English adjectives

50 University of Colorado undergraduates;

40 positively/negatively valencedEnglish adjectives

Ultimate targets

of generalization

My actual

samples

All

potentially sampled

participants/stimuli

All non-neutral visual stimuli

“Subject-matter span”

“Statistical span”

50 University of Colorado undergraduates;

40 positively/negatively valencedEnglish adjectives

Ultimate targets

of generalization

My actual

samples

All

potentially sampled

participants/stimuli

Difficulties crossing the statistical span

- Failure to account for uncertainty associated with stimulus sampling (i.e., treating stimuli as fixed rather than random) leads to biased, overconfident estimates of effects
- The pervasive failure to model stimulus as a random factor is probably responsible for many failures to replicate when future studies use different stimulus samples

Doing the correct analysis is easy!

- Modern statistical procedures solve the statistical problem of stimulus sampling
- These linearmixed models with crossed random effects are easy to apply and are already widely available in major statistical packages
- R, SAS, SPSS, Stata, etc.

Illustrative Design

- Participants crossed with Stimuli
- Each Participant responds to each Stimulus

- Stimuli nested under Condition
- Each Stimulus always in either Condition A or Condition B

- Participants crossed with Condition
- Participants make responses under both Conditions
Sample of hypothetical dataset:

- Participants make responses under both Conditions

Typical repeated measures analyses (RM-ANOVA)

How variable are the stimulus ratings around each of the participant means? The variance is lost due to the aggregation

- “By-participant analysis”

Typical repeated measures analyses (RM-ANOVA)

4.00 3.67 6.33 7.33 3.67 6.33 8.00 6.00 8.00 4.00 5.00 5.33

Sample 1 v.s. Sample 2

“By-stimulus analysis”

Simulation of type 1 error rates for typical RM-ANOVA analyses

- Design is the same as previously discussed
- Draw random samples of participants and stimuli
- Variance components = 4, Error variance = 16

- Number of participants = 10, 30, 50, 70, 90
- Number of stimuli = 10, 30, 50, 70, 90
- Conducted both by-participant and by-stimulus analysis on each simulated dataset
- True Condition effect = 0

Type 1 error rate simulation results analyses

- The exact simulated error rates depend on the variance components, which although realistic, were ultimately arbitrary
- The main points to take away here are:
- The standard analyses will virtually always show some degree of positive bias
- In some (entirely realistic) cases, this bias can be extreme
- The degree of bias depends in a predictable way on the design of the experiment (e.g., the sample sizes)

The old solution: Quasi- analysesF statistics

- Although quasi-Fs successfully address the statistical problem, they suffer from a variety of limitations
- Require complete orthogonal design (balanced factors)
- No missing data
- No continuous covariates
- A different quasi-F must be derived (often laboriously) for each new experimental design
- Not widely implemented in major statistical packages

The analysesnew solution: Mixed models

- Known variously as:
- Mixed-effects models, multilevel models, random effects models, hierarchical linear models, etc.

- Most psychologists familiar with mixed models for hierarchical random factors
- E.g., students nested in classrooms

- Less well known is that mixed models can also easily accommodate designs with crossed random factors
- E.g., participants crossed with stimuli

Grand mean = 100 analyses

Mean analysesA = -5 MeanB = 5

Stim analysesulus Means: -2.84 -9.19 -1.16 18.17

The linear mixed-effects model errorwith crossed random effects

Fixed effects Random effects

Fitting mixed models is easy: Sample syntax error

R

library(lme4)

model <- lmer(y ~ c + (1 | j) + (c | i))

proc mixed covtest;

class i j;

model y=c/solution;

random intercept c/sub=i type=un;

random intercept/sub=j;

run;

SAS

MIXED y WITH c

/FIXED=c

/PRINT=SOLUTION TESTCOV

/RANDOM=INTERCEPT c | SUBJECT(i) COVTYPE(UN)

/RANDOM=INTERCEPT | SUBJECT(j).

SPSS

Mixed models successfully maintain the nominal type 1 error errorrate (α = .05)

Conclusion error

- Stimulus variation is a generalizability issue
- The conclusions we draw in the Discussion sections of our papers ought to be in line with the assumptions of the statistical methods we use
- Mixed models with crossed random effects allow us to generalize across both participants and stimuli

Further reading:

Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of personality and social psychology, 103(1), 54-69.

Download Presentation

Connecting to Server..