A Multilevel Study of the Effects of Opportunity to Learn on Reading Achievement: Issues of Measurement, Equity, and Val

1. A Multilevel Study of the Effects of Opportunity to Learn on Reading Achievement:Issues of Measurement, Equity, and Validity Jose-Felipe Martinez-Fernandez Social Research Methods Division School of Education, UCLA

2. Outline Opportunity to Learn (OTL) Legal Origins and Technical Issues Operationalization of OTL for this study Research Questions Measurement, Equity, Validity (OTL Effects) Sample Data and Variables Study Design and Analyses Summary and Implications

3. Opportunity to Learn Legal Origins Inequalities: From financial to instructional practices Rulings leading to Debra P. v. Turlington, 1983 Recent accountability efforts Technical Issues International comparisons (FIMS, TIMMS) OTL indicators (Binary, Time on Task, Practices) Source of OTL data (Teachers, students, observation) School effectiveness (Classroom and school context) Alternative Assessments (Performance Assessments)

4. Opportunity to Learn Operationalization in this study Time on Task Instructional Practices Interaction with Context

5. Research Questions 1) Measurement Are students and teachers consistent in their perceptions of the amount and nature of the opportunities to learn offered in the classroom? Factor analysis of Teacher and Student OTL questionnaires

6. Research Questions 2) Equity How is OTL distributed across levels of the educational system? To what extent does OTL differ across (students) classrooms and schools? Multilevel models of Student and Teacher OTL composites generated from Factor Analysis

7. Research Questions 3) Effects of OTL: Model Validity What are the effects of OTL on student achievement (SAT9 Reading)? What are the consequences of ignoring the classroom level in multilevel modeling of OTL effects? Are different conclusions obtained from three-level models, compared to two-level models? Two- and three- level HLMs of OTL effects on SAT9.

8. Research Questions 4) Effects of OTL: Predictive Validity Are similar patterns of OTL effects on student achievement observed using a Language Arts Performance Assessment, as opposed to SAT9 scores, as the measure of student achievement? Comparison of OTL results observed with HLMs of OTL effects on PA and SAT9 scores.

9. Data LAUSD validation of a Language Arts PA. 72,866 3rd to 5th grade students, 3,868 classrooms, 368 schools. Achievement Measures Student SAT9 (NCE) scores in 2001 and 2000 Language Arts PA (4-Point) scores in 2001 OTL Questionnaire Student Background � Sex, LEP, SES, Minority Classroom � Class size, Teacher experience & college courses, student composition. School-Parent education, teacher credentials, student composition

12. 1) Measurement Agreement between Student and Teacher reports of OTL

13. OTL Questionnaire Six 4-point Likert-scale Items 1.How often did you read literature? 2.How often did your teacher read aloud to you? 3.How often did you write compositions? 4.How often did you take notes on your ideas before beginning to write a composition? 5.How often did you use information from books to support ideas in your compositions? 6.How often did you rewrite your compositions to make them better?

15. Descriptive statisticsStudent and Teacher OTL composites

16. 2) Equity Distribution of OTL across classrooms and schools

17. 3-Level unconditional models of Student-reported OTL

18. Distribution of Student-, Teacher-reported OTL2- and 3- Level unconditional models

19. 3) OTL Effects: Model Validity Comparison of inferences drawn from 2- and 3- Level Models

20. 2- and 3- Level unconditional models of Student reading achievement (SAT9)

21. 2- and 3- Level variance components

23. 3-Level ANCOVA Model (Random)  Variation of OTL effects across Classrooms - Schools

24. Variation of OTL effects across classrooms

25. Intercepts-and-Slopes-as-Outcomes ModelClassroom and School Effects

26. 2-Level ANCOVA Model Variation of OTL effects across Schools

27. 4) OTL Effects: Predictive Validity Comparison of results with SAT9 and PA scores

28. Classroom and School Variance Componentsfor SAT9 and PA scores

29. 3- Level SAT9 and PA Conditional Models Comparison of Fixed and Random OTL Effects

30. Summary and Implications Measurement Similar Factor structures for Teachers and Students. Teachers report more reading OTL than students. Moderate correlations (under high skewness). Student OTL reports more powerful predictors of achievement (SAT9 or PA).

31. Summary and Implications Equity Schools are largely equivalent in terms of the average amounts of OTL they provide Differences are mainly between classrooms 60% of the variance in student reports is within classrooms: Measurement Error but also Classroom�Student Interaction. Personal component of OTL

32. Summary and Implications Model Validity Classroom placement as important (16% of variance), or more, as school attended (14%) 2-Level models overestimate school variance, but overall underestimate effect of the system Effect of Reading OTL varies across classrooms, not schools. Contextual effects at classroom, not school level. Consistent effects of Teacher quality variables 2-Level models distort perception of the mechanisms through which schools matter. Usefulness � Validity of school level aggregates for accountability is suspect

33. Summary and Implications Predictive Validity Larger between-classroom variance (21%) - PA more sensitive to instructional differences? OTL slopes vary across classrooms, not schools Similar effect sizes of Reading OTL Effect of Writing OTL in the Language Arts PA

34. Limitations of Application to Policy Easier to partition variance than to explain it Feasibility of collecting data from students Definition of opportunity for accountability Political and technical feasibility of moving accountability down to the teacher level Sample requirements Sample generalizability

35. Selected References