Technology’s Edge: The Educational Benefits of Computer-Aided Instruction

1. Technology’s Edge: The Educational Benefits of Computer-Aided Instruction Lisa Barrow Federal Reserve Bank of Chicago Lisa Markman Princeton University Cecilia Rouse Princeton University and NBER

2. Student achievement is critical for individuals and society • U.S. math skills have been increasing (NAEP), but proficiency levels remain low. • Math skills may explain a large portion of wage inequality (Grogger, 1996; Murnane, Willet, & Levy, 1995)

3. School districts are turning to advances in computers to… • Reduce administrative burden; • Compensate for poor teacher content knowledge (especially in districts that report difficulty recruiting and retaining teachers, particularly in math and science); • Allow more individualized student attention; students can progress at own pace.

4. Evidence on effectiveness of CAI is surprisingly weak • Poorly defined computer use. • Little use of randomized controlled study design.

5. In a 2001 review of the literature, Larry Cuban (2001, p. 179) concludes,“When it comes to higher teacher and student productivity and a transformation of teaching and learning … there is little ambiguity. Both must be tagged as failures. Computers have been oversold and underused, at least for now.”

6. Computer-Aided Instruction (CAI) Treatment: • Typically used in large urban districts; • Software and hardware package designed to deliver one-on-one instruction; • Software described as “meeting National Council of Teachers of Mathematics standards”; • Software may be configured to state standards; • Includes classroom management tools.

7. The Program • 5 components per lesson: • Pretest • Review • Lesson • Cumulative review • Comprehensive test • Required to achieve certain degree of mastery before advancing; • Teachers can monitor student progress.

8. The design of this experiment… • Randomly assign 8th and 9th grade classes in three large urban districts to be taught using computer-aided algebra and pre-algebra instruction. • Assess the impact on statewide tests and tests designed to target algebra and pre-algebra skills. • Note that the computer use was well-defined.

9. Research Design: Within-school random assignment to CAI or traditional instruction at the classroom level in three districts Research design addresses two forms of selection bias: • Non-random assignment of students to CAI; • Non-random assignment of teachers to CAI.

10. Table 1: Characteristics of the Districts in Our Study

11. Implementation of Random Assignment • Schools provided us with class schedules of pre-algebra and algebra classes; • Given option of eliminating particular teachers or classes from the experiment; • Randomization information provided to schools after students’ classes were scheduled.

12. Randomized Evaluation of a Computerized Math Curriculum Current School Schedule

13. Randomized Evaluation of a Computerized Math Curriculum Current School Schedule Below is a sample schedule that would be returned to each school after random assignment

14. Table 2a: Schools and Students in Study – District 1

15. Table 2b: Schools and Students in Study – District 2

16. Table 2c: Schools and Students in Study – District 3

17. Numbers of Schools Classes, Teachers, and Randomization Pools

18. Outcomes • Algebra and pre-algebra tests by Northwest Evaluation Association (NWEA) to be consistent with state and district standards; • State-wide administered math tests; • District benchmark tests in pre-algebra.

19. Table 3: Randomization of Treatment and Control Using Full Sample

20. Table 3 (cont.): Randomization of Treatment and Control Using Analysis Sample

21. Intent-to-Treat Estimates of the Effect of CAI on Algebra Achievement (with and without Teacher Fixed Effects)

22. Advantages/Disadvantages of the Intent-to-Treat Effect • Represents the gains a policy maker might reasonably expect to observe. • Does not necessarily represent the effectiveness of the program.

23. Amount of Time in the Computer Lab by the Random Assignment of the Student’s Class

24. Intent-to-Treat and Treatment on the Treated Estimates of the Effect of CAI (with and without teacher fixed effects)

25. Intent-to-Treat Estimates in District 1 Using Different Tests

26. Intent-to-Treat Estimates in Districts 2 and 3 Using Different Tests

27. We might expect to see an advantage of CAI in… • Classes where curriculum best suited to students; • Larger classes; • Classes with more disruptive students; • Classes with heterogeneous students.

28. Differential Intent to Treat Effects of the Computerized Instruction on Pre-Algebra and Algebra Achievement by Class Type

29. Differential Intent to Treat Effects of CAI on Pre-Algebra and Algebra Achievement by Baseline Test Score Quartile

30. Differential Intent to Treat Effects of CAI on Pre-Algebra and Algebra Achievement by Individual Attendance Rates

31. Differential Intent-to-Treat Effects by Class Characteristic: Attendance, Class Size, and Class Baseline Test Score S.D.

32. Overall we find: • On average, students in classes taught using CAI scored higher on algebra tests than students in traditionally-taught classes. • The effects appear larger for students in larger classes (especially large, heterogenous classes), those with worse attendance rates, and those in classes with lower average attendance rates.

33. Cost-Benefit Analysis