optimal design for longitudinal and multilevel research
Download
Skip this Video
Download Presentation
Optimal Design for Longitudinal and Multilevel Research

Loading in 2 Seconds...

play fullscreen
1 / 20

Optimal Design for Longitudinal and Multilevel Research - PowerPoint PPT Presentation


  • 458 Views
  • Uploaded on

Optimal Design for Longitudinal and Multilevel Research. Jessaca Spybrook July 10, 2008 *Joint work with Steve Raudenbush and Andres Martinez. Outline. Overview Examples 2-level HLM 3-level HLM 3-level Randomized Block Design Writing up a Power Analysis Exercises. Overview.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Optimal Design for Longitudinal and Multilevel Research' - fionn


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
optimal design for longitudinal and multilevel research

Optimal Design for Longitudinal and Multilevel Research

Jessaca Spybrook

July 10, 2008

*Joint work with Steve Raudenbush and Andres Martinez

outline
Outline
  • Overview
  • Examples
    • 2-level HLM
    • 3-level HLM
    • 3-level Randomized Block Design
  • Writing up a Power Analysis
  • Exercises
overview
Overview

Tool for planning studies

HLM notation

Connection to Howard’s formulas

Warning: Optimal Design is easy to use but can also be easily misused

connecting the language
Connecting the Language

Larry OD

2-level HLM Cluster Randomized Trial

3-level HLM Three level model with treatment at level three

3-level RBD Multi-site Cluster Randomized trial

organization of optimal design
Organization of Optimal Design
  • Optimal Design for Group Randomized Trials
      • Cluster Randomized Trial (2-level HLM)
      • Three level model with treatment at level 3 (3-level HLM)
      • Multi-site cluster randomized trial (3-level RBD)
cluster randomized trial 2 level hlm example
Cluster Randomized Trial (2-level HLM) Example

Program: A new math curriculum is randomly assigned and implemented at the school level.

Research question: Is the new math curriculum more effective than the traditional one?

Standardized math tests scores used as outcome measure. Based on previous research:

  • 18% percent of the variation lies between schools.

(Students nested within schools).

  • Researchers expect the treatment to boost test scores by 0.25 standard deviations.

With 90 students per school, how many schools are needed to detect the above effect size with 80% power?

cluster randomized trial 2 level hlm example8
Cluster Randomized Trial (2-level HLM) Example
  • How many levels are in this study?
  • What is the unit of randomization?
  • Is there blocking?
  • What is the MDES?
  • What is a good estimate of the intraclass correlation?
  • What information is missing?
cluster randomized trial 2 level hlm
Cluster Randomized Trial (2-level HLM)

J – the total number of clusters

n – the total number of individuals per cluster

Rho – the intraclass correlation

R-squared – the percent of variation explained by level-2 covariate

Delta – the effect size

three level model with treatment at level three 3 level hlm example
Three level model with treatment at level three (3-level HLM) Example

Program: Third grade math curriculum implemented at the school level (group-level intervention). 40 schools are randomized. There are 6 teachers within each school and 25 kids per teacher.

Research question: Is the new math curriculum more effective than the traditional one?

Standardized math tests scores used as outcome measure. Based on previous research:

  • 8% percent of the variation lies between classes. 12% of the variation lies between schools.

What is the MDES for power of 0.80? Is this reasonable?

three level model with treatment at level three 3 level hlm example11
Three level model with treatment at level three (3-level HLM) Example

How many levels are in this study?

What is the unit of randomization?

Is there blocking?

What is the MDES?

What are good estimates of the intraclass correlations?

What information is missing?

three level model with treatment at level three 3 level hlm
Three level model with treatment at level three (3-level HLM)

K- the total number of schools

J – the total number of classrooms per school

n – the total number of individuals per classroom

Rho pi (level-2 ICC)– the intraclass correlation at classroom level

Rho beta (level-3 ICC) – the intraclass correlation at school level

R-squared – the percent of variation explained by level-3 covariate

Delta – the effect size

multi site cluster randomized trial 3 level rbd example
Multi-site Cluster Randomized Trial (3-level RBD) Example

Program: The following will be repeated across districts. 6 schools will be randomly assigned to either the new math curriculum or the regular curriculum. 150 students will be tested in each school.

Causal question: Is the new math curriculum more effective than the traditional one?

Standardized math tests scores used as outcome measure. Based on previous research:

  • 25% percent of the variation lies between districts.
  • We expect blocking will account for 30% of the variation.
  • Researchers expect the treatment to boost test scores by 0.20 standard deviations.

How many districts are needed to detect an effect size of 0.20 with 80% power?

multi site cluster randomized trial 3 level rbd example14
Multi-site Cluster Randomized Trial (3-level RBD) Example
  • How many levels are in this study?
  • What is the unit of randomization?
  • Is there blocking?
    • If so, how much of the variation is explained by blocking? How are blocks being treated – as fixed or random effects? If random effects, what is the variation of treatment effect across schools?
  • What is the MDES?
  • What is the intraclass correlation prior to blocking?
  • What information is missing?
multi site cluster randomized trial 3 level rbd
Multi-site Cluster Randomized Trial (3-level RBD)
  • K- the total number of blocks/sites (districts)
  • J – the total number of schools per district
  • n – the total number of individuals per school
  • Rho – the intraclass correlation
  • R-squared – the percent of variation explained by level-3 covariate
  • B – percent of variation explained by blocking
  • Sigma sq sub delta – effect size variability (specifying greater than 0 treats blocks as random effects)
  • Delta – the effect size
writing up a power analysis
Writing up a Power Analysis
  • Differs depending on the design (again it is critical to know your design!)
exercise 1
Exercise 1

Suppose a group of researchers want to examine the effectiveness of a professional development program intended to improve reading instruction for elementary teachers. They plan to randomly assign schools to the treatment (the new pd).Within each school, they select four 2nd grade teachers to implement the program. Each teacher has approximately 25 students. The outcome of interest is student reading achievement. They expect that students in the treatment schools will improve their reading scores by 0.25 standard deviations. They plan to use last years students posttest as a covariate. How many schools are necessary for 80 percent power? (Hint: use the best estimates for the intraclass correlation(s))

exercise 2
Exercise 2

Suppose that a team of researchers plan to evaluate the effectiveness of a scripted whole school reform model. They plan to randomly assign entire schools to either receive the new reform model or continue with their current practices. On average, there are 300 students in each school. They expect that students in the treatment schools will outperform students in the control schools by 0.20 standard deviations on a particular math test. They estimate that approximately 20 percent of the variation lies between schools. They plan to use last years math scores as a covariate. How many schools do they need to have 80 percent power? What if they only tested 50 kids in the school, how does that affect the power?

exercise 3
Exercise 3

Suppose that a team of researchers recruited 6 districts for a study of the effectiveness of a new science program. They plan to randomly assign 4 schools within each district to either the new curriculum or their current curriculum and to test 150 students within each school. They expect that by blocking on district they will remove 25 percent of the variation in science outcome scores. They estimate an intraclass correlation of 0.24 and have a pre-test covariate that they think will explain approximately 60 percent of the variation in test scores. What is the smallest effect size they can detect with power of 0.80? Is this reasonable? (Hint: remember to go through the steps of selecting the appropriate design and all the relevant parts.)

ad