Optimal design for longitudinal and multilevel research
Download
1 / 20

Optimal Design for Longitudinal and Multilevel Research - PowerPoint PPT Presentation


  • 460 Views
  • Updated On :

Optimal Design for Longitudinal and Multilevel Research. Jessaca Spybrook July 10, 2008 *Joint work with Steve Raudenbush and Andres Martinez. Outline. Overview Examples 2-level HLM 3-level HLM 3-level Randomized Block Design Writing up a Power Analysis Exercises. Overview.

Related searches for Optimal Design for Longitudinal and Multilevel Research

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Optimal Design for Longitudinal and Multilevel Research' - fionn


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Optimal design for longitudinal and multilevel research l.jpg

Optimal Design for Longitudinal and Multilevel Research

Jessaca Spybrook

July 10, 2008

*Joint work with Steve Raudenbush and Andres Martinez


Outline l.jpg
Outline

  • Overview

  • Examples

    • 2-level HLM

    • 3-level HLM

    • 3-level Randomized Block Design

  • Writing up a Power Analysis

  • Exercises


Overview l.jpg
Overview

Tool for planning studies

HLM notation

Connection to Howard’s formulas

Warning: Optimal Design is easy to use but can also be easily misused


Connecting the language l.jpg
Connecting the Language

Larry OD

2-level HLM Cluster Randomized Trial

3-level HLM Three level model with treatment at level three

3-level RBD Multi-site Cluster Randomized trial


Organization of optimal design l.jpg
Organization of Optimal Design

  • Optimal Design for Group Randomized Trials

    • Cluster Randomized Trial (2-level HLM)

    • Three level model with treatment at level 3 (3-level HLM)

    • Multi-site cluster randomized trial (3-level RBD)


Cluster randomized trial 2 level hlm example l.jpg
Cluster Randomized Trial (2-level HLM) Example

Program: A new math curriculum is randomly assigned and implemented at the school level.

Research question: Is the new math curriculum more effective than the traditional one?

Standardized math tests scores used as outcome measure. Based on previous research:

  • 18% percent of the variation lies between schools.

    (Students nested within schools).

  • Researchers expect the treatment to boost test scores by 0.25 standard deviations.

    With 90 students per school, how many schools are needed to detect the above effect size with 80% power?


Cluster randomized trial 2 level hlm example8 l.jpg
Cluster Randomized Trial (2-level HLM) Example

  • How many levels are in this study?

  • What is the unit of randomization?

  • Is there blocking?

  • What is the MDES?

  • What is a good estimate of the intraclass correlation?

  • What information is missing?


Cluster randomized trial 2 level hlm l.jpg
Cluster Randomized Trial (2-level HLM)

J – the total number of clusters

n – the total number of individuals per cluster

Rho – the intraclass correlation

R-squared – the percent of variation explained by level-2 covariate

Delta – the effect size


Three level model with treatment at level three 3 level hlm example l.jpg
Three level model with treatment at level three (3-level HLM) Example

Program: Third grade math curriculum implemented at the school level (group-level intervention). 40 schools are randomized. There are 6 teachers within each school and 25 kids per teacher.

Research question: Is the new math curriculum more effective than the traditional one?

Standardized math tests scores used as outcome measure. Based on previous research:

  • 8% percent of the variation lies between classes. 12% of the variation lies between schools.

    What is the MDES for power of 0.80? Is this reasonable?


Three level model with treatment at level three 3 level hlm example11 l.jpg
Three level model with treatment at level three (3-level HLM) Example

How many levels are in this study?

What is the unit of randomization?

Is there blocking?

What is the MDES?

What are good estimates of the intraclass correlations?

What information is missing?


Three level model with treatment at level three 3 level hlm l.jpg
Three level model with treatment at level three (3-level HLM)

K- the total number of schools

J – the total number of classrooms per school

n – the total number of individuals per classroom

Rho pi (level-2 ICC)– the intraclass correlation at classroom level

Rho beta (level-3 ICC) – the intraclass correlation at school level

R-squared – the percent of variation explained by level-3 covariate

Delta – the effect size


Multi site cluster randomized trial 3 level rbd example l.jpg
Multi-site Cluster Randomized Trial (3-level RBD) Example HLM)

Program: The following will be repeated across districts. 6 schools will be randomly assigned to either the new math curriculum or the regular curriculum. 150 students will be tested in each school.

Causal question: Is the new math curriculum more effective than the traditional one?

Standardized math tests scores used as outcome measure. Based on previous research:

  • 25% percent of the variation lies between districts.

  • We expect blocking will account for 30% of the variation.

  • Researchers expect the treatment to boost test scores by 0.20 standard deviations.

    How many districts are needed to detect an effect size of 0.20 with 80% power?


Multi site cluster randomized trial 3 level rbd example14 l.jpg
Multi-site Cluster Randomized Trial (3-level RBD) Example HLM)

  • How many levels are in this study?

  • What is the unit of randomization?

  • Is there blocking?

    • If so, how much of the variation is explained by blocking? How are blocks being treated – as fixed or random effects? If random effects, what is the variation of treatment effect across schools?

  • What is the MDES?

  • What is the intraclass correlation prior to blocking?

  • What information is missing?


Multi site cluster randomized trial 3 level rbd l.jpg
Multi-site Cluster Randomized Trial (3-level RBD) HLM)

  • K- the total number of blocks/sites (districts)

  • J – the total number of schools per district

  • n – the total number of individuals per school

  • Rho – the intraclass correlation

  • R-squared – the percent of variation explained by level-3 covariate

  • B – percent of variation explained by blocking

  • Sigma sq sub delta – effect size variability (specifying greater than 0 treats blocks as random effects)

  • Delta – the effect size


Writing up a power analysis l.jpg
Writing up a Power Analysis HLM)

  • Differs depending on the design (again it is critical to know your design!)


Exercise 1 l.jpg
Exercise 1 HLM)

Suppose a group of researchers want to examine the effectiveness of a professional development program intended to improve reading instruction for elementary teachers. They plan to randomly assign schools to the treatment (the new pd).Within each school, they select four 2nd grade teachers to implement the program. Each teacher has approximately 25 students. The outcome of interest is student reading achievement. They expect that students in the treatment schools will improve their reading scores by 0.25 standard deviations. They plan to use last years students posttest as a covariate. How many schools are necessary for 80 percent power? (Hint: use the best estimates for the intraclass correlation(s))


Exercise 2 l.jpg
Exercise 2 HLM)

Suppose that a team of researchers plan to evaluate the effectiveness of a scripted whole school reform model. They plan to randomly assign entire schools to either receive the new reform model or continue with their current practices. On average, there are 300 students in each school. They expect that students in the treatment schools will outperform students in the control schools by 0.20 standard deviations on a particular math test. They estimate that approximately 20 percent of the variation lies between schools. They plan to use last years math scores as a covariate. How many schools do they need to have 80 percent power? What if they only tested 50 kids in the school, how does that affect the power?


Exercise 3 l.jpg
Exercise 3 HLM)

Suppose that a team of researchers recruited 6 districts for a study of the effectiveness of a new science program. They plan to randomly assign 4 schools within each district to either the new curriculum or their current curriculum and to test 150 students within each school. They expect that by blocking on district they will remove 25 percent of the variation in science outcome scores. They estimate an intraclass correlation of 0.24 and have a pre-test covariate that they think will explain approximately 60 percent of the variation in test scores. What is the smallest effect size they can detect with power of 0.80? Is this reasonable? (Hint: remember to go through the steps of selecting the appropriate design and all the relevant parts.)