Sample Size calculations in multilevel modelling. William Browne University of Nottingham (With thanks to Mousa Golalizadeh and Lynda Leese). Summary. Introduction to sample size calculations. A simulation-based approach. PINT for balanced 2 level models. Effect of balance.
Sample Size calculations in multilevel modelling
University of Nottingham
(With thanks to
Mousa Golalizadeh and Lynda Leese)
Here RHS is sum of cases H0 true and H0 false.
Here we have looked at two examples with effect sizes 5 and 1 respectively. Assume σ takes the value 5 and so let us suppose we take a sample of size 25 Welshmen.
Case 1: 5/(5/√25)=1.645+z1-β,z1-β=3.355
Case 2: 1/(5/ √25)=1.645+z1-β,z1-β=-0.645
So here a sample of 25 Welshmen from a population with mean 180cms would almost always result in rejecting H0,
but if the population mean is 176cms then only 26% of such samples would be rejected.
We can plot curves of how power increases with sample size as shown in the next slide.
Here we see the two power curves for the two scenarios:
be used in many situations and hypothesis tests.
Construct research question -> Form null hypothesis that we believe false -> Collect appropriate data -> Reject hypothesis therefore proving our research question.
Note simulation curve is a good approximation of the theoretical curve although there are some minor (Monte Carlo) errors even with 5000 simulations per sample size.
We will here mainly consider 2-level models and take as our application area education, so we have students nested within schools.
When deciding on a sampling scheme we have many choices:
Our decision will depend on which parameter we wish to estimate in the model.
The design effect formula:
Design effect = 1 + (n-1)ρ
suggests that if we are to sample a fixed (balanced) number of pupils n*N then our best power results when n is smallest i.e. sampling one pupil each from 100 schools is better than sampling 100 pupils from the same school.
The effect of sampling policy is most important in scenarios where ρ is large e.g. repeated measures designs.
The simulation procedure gives approximately the same power curve and so in this simple example we have an easy to use formula.
The reason in practice for sampling several pupils from each school is purely the additional cost incurred in visiting additional schools.
Here we see good agreement from approaches. It appears that we need a large dataset to have strong power for this hypothesis.
Here the PinT curve appears to give slightly higher power suggesting that maybe the alternative predictor variances would be more appropriate.
Extremely unbalanced designs are really estimating the effect of the large school instead of the global mean and hence the level 2 variance is often estimated as 0.