Hierarchical Data Modeling in Education Impact Evaluation | MLM Refresher

1. Multi-level Modeling (MLM) Refresher Jessaca Spybrook Western Michigan University

2. MLM Refresher • Goal of session • Brief review of multilevel models • Establish common language • Establish common notation

3. Hierarchical Data • Individuals grouped into larger units • Examples • Students in schools • Citizens in communities • Focus Example • Africa Program for Education Impact Evaluation in the Gambia • Students in schools • Teachers in schools • Classrooms in schools

4. Hierarchical Data • Methods for analyzing data • Put everything at one level • Aggregate data up to level 2 • Model both levels together • Model both levels together • Improved estimation of individual effects • Questions related to cross-level effects • Partitioning of variance among levels

5. Hierarchical Data • Modeling both levels together • Hierarchical linear models, multilevel models, mixed effects models, random effects models, random coefficient models

6. Hierarchical Data • Scenario • The Gambia data (2008) • Students nested in schools • 2,657 ->2,008 students (pupils data) • 271 ->204 schools (head teacher data)

7. Hierarchical Data • Variables • DV: • Number of words read correctly in 60 seconds (reading fluency) [S2Q3_PP] • IVs: • Treatment (L2) [trmt –WSC and Grant] • Age (L1) [age_PP] • Mean school age (L2) [age_PP_m]

8. Guiding Questions (A) • Guiding Questions (A) • What is the mean reading fluency for all students? • How much variation in reading fluency is between schools? Within schools?

9. The Model Level 1 (students): Yijis reading fluency for student i in school j is the mean reading fluency for school j rij is the random error associated with student i in school j, Level 2 (schools): is the average school mean reading fluency across the schools u0j is the random error associated with school means,

10. The Model Combined Model: - Demo in HLM - Fill in Table as we go

11. ICC Intraclass Correlation: is the between school variance is the within school variance

12. Guiding Question (B) • Guiding Question (B) • Is there a difference in reading fluency at baseline for those in the treatment condition compared to those in the control condition?

13. The Model Level 1 (students): Yijis reading fluency for student i in school j is the mean reading fluency for school j rij is the random error associated with student i in school j, Level 2 (schools): is the average school mean reading fluency for the control schools Wj is the indicator for condition (1=treatment including WSC and Grant, 0=control) is the main effect of treatment, average difference in mean reading fluency for treatment and control schools is the random error associated with control school means, now a conditional variance,

14. The Model Combined Model:

15. Guiding Questions (C) • Guiding Questions (C) • What is the relationship between students age and reading fluency? • Consider 5 options • 1 - Age is group mean centered at L1 • 2 - Age is uncentered at L1 • 3 - Age is grand mean centered at L1 • 4 - Age is group mean centered at L1, grand mean centered at L2 • 5 - Age is grand mean centered at L1, grand mean centered at L2

16. The Model-Option 1 Level 1 (students): Yijis reading fluency for student i in school j is the average unadjusted mean reading fluency for school j is the average change in reading fluency for a 1 unit increase in student age in school j (within school age-reading fluency slope) rij is the random error associated with student i in school j, Level 2 (schools): is the average school mean reading fluency across the schools is the average age-reading fluency slope within schools u0j is the random error associated with school means,

17. The Model Combined Model:

18. Option 2 • What if we left age uncentered? • Same models, age uncentered • Intercept is now average school mean reading fluency for schools when age = 0 • Slope is now composite of within school age-reading fluency relationship and between-school age reading fluency relationship

19. Option 3 • What if we grand mean centered age? • Same models, age grand mean centered • Intercept is now average adjusted school mean reading fluency for schools • Slope is now composite of within school age-reading fluency relationship and between-school age reading fluency relationship

20. Option 4 • What if we group mean centered age at L1 and grand mean centered age at L2? • Need new model • Aggregate version of age for each school at L2

21. The Model – Option 4 Level 1 (students): Yijis reading fluency for student i in school j is the average unadjusted mean reading fluency for school j is the average change in reading fluency for a 1 unit increase in student age in school j (within school age-reading fluency slope) rij is the random error associated with student i in school j, Level 2 (schools): is the average school mean reading fluency across the schools adjusted for school mean age is the average change in school mean reading fluency for a 1 unit increase in school mean age across schools (between school mean age-reading fluency relationship) is the average age-reading fluency slope within schools u0j is the random error associated with adjusted school means, now a conditional,

22. Option 5 • What is we grand mean centered age at L1 and grand mean centered age at L2? • Same model, age grand mean centered at L1 • Intercept is same but adjusted mean • is same • is the compositional effect of age, weighted composite of the within and between slopes, difference between 2 students with same age value but who attend schools that differ by one unit of school mean age • Note:

23. Next Steps • Practice session in lab • Questions/comments via video session