Thoughts on Relevance Robert C. Granger, Ed.D.

1. Thoughts on Relevance Robert C. Granger, Ed.D. Remarks prepared for Society for Research on Educational Effectiveness Alexandria, VA / March 1-3, 2009

2. Why relevance? • Relevant for what? • Relevant for whom? 2

3. Main Messages • The R&D cycle of developing, testing, or identifying innovations at a small scale and then “scaling up” these pockets of excellence does not work. • The role of research is to better sift through practices implemented at scale, not to “discover” innovations. • Instead of focusing on individual student or teacher performance, focus on classrooms. 3

4. “There are many small pockets of excellence in the education market. But because of its fragmentation, few of these have been able to scale. With the proper incentives and infrastructure, many innovations could be surfaced or created and more importantly, adopted broadly, resulting in dramatic increases in education production – getting greater student achievement gains per dollar invested.” Bill and Melinda Gates Roll Call Op Ed February 10, 2009 4

5. “A robust new federal Office of Educational Entrepreneurship and Innovation within the Department of Education would expand the boundaries of public education by scaling up successful entrepreneurs, seeding transformational educational innovations, and building a stronger culture to support these activities throughout the public sector.” Sara Mead and Andrew Rotherham Paper for Brookings Metropolitan Policy Program October, 2008 5

6. Comparison of Clinical Trial Phases Phase I Objective: Determine the metabolic and pharmacological actions and the maximally tolerated dose. Phase II Objective: Evaluate effectiveness, determine the short-term side effects, and identify common risks for a specific population and disease. Phase III Objective: Obtain additional information about the effectiveness on clinical outcomes and evaluate the overall risk-benefit ratio in a demographically diverse sample. Phase IV Objective: Monitor ongoing safety in large populations and identify additional uses of the agent that might be approved by the FDA. 6

7. Stages of Research in Research Prevention Cycle 7

8. IES Goal Structure for Grantmaking • GoalDescription • Identification Projects • Development Projects • Efficacy and Replication Trials • Scale-Up Evaluations • 5. Measurement Projects 8

9. 9

10. A Snapshot: Intervention Trials in 2006 • National Institute of Mental Health (NIMH): 51 trials; 3 implementation/dissemination trials; • National Institute of Drug Abuse (NIDA): 79 trials; 14 implementation/dissemination trials; • National Institute on Alcohol Abuse and Alcoholism (NIAAA): 53 trials; 2 implementation/dissemination trials. • National Academies Table F-1, page 522. 10

11. 11

12. “Contradiction and initially stronger effects are not unusual in highly cited research of clinical interventions and their outcomes. The extent to which high citations may provoke contradictions and vice versa needs more study. Controversies are most common with highly cited nonrandomized studies, but even the most highly cited randomized trials may be challenged and refuted over time, especially small ones.” Ioannidis, John P.A., JAMA, 2005;294:218. 12

13. “Part II (of the report) describes the substantial progress in prevention science since 1994. It describes “numerous” (quotation marks added) efficacious or effective prevention programs…thus far, however, prevention programs have generally not been widely implemented in schools and communities and have done little to reduce behavioral health problems in America.” National Academies, p. 295. 13

14. Barriers to Scaling Innovations • Program may not fit community needs, strengths, or capacities. • Real-world implementation may differ dramatically from the way originally tested. • Lack of ownership of the program. • Few evidence-based programs have the capacity to provide technical assistance and training. • An evidence-based program may not target outcomes relevant to community. 14

15. Barriers to Scaling Innovations (continued) • Key program components may be modified, thereby reducing outcomes. • Essential program components not always evident. • Lengthy period to develop community awareness, common vision, and program. • Potential for ineffectiveness or iatrogenic effects. • Challenges in obtaining funding for sustaining a unique program. 15

16. 16

17. The Decade of Experiment (1951-1961) • Recruitment and training programs for liberal arts undergraduates • Differentiated staffing with classroom aides • Increased use of technology (TV, tape, teaching machines) • Projects to introduce modern business methods into school management including a focus on teacher salaries. ($50 million in 1955 = $393 million in 2009). 17

18. Comprehensive School Improvement Program: 12 Practices • Team teaching • Use of non-professional personnel in schools • Flexible scheduling • Variable-size pupil groups for instruction and new space arrangements • Use of audio-visual resources, including educational television • Programmed instruction 18

19. Comprehensive School Improvement Program: 12 Practices (continued) • Language laboratories • Educational data processing by machine • Independent study • Advanced placement and early admissions • Non-graded school programs • School and university partnerships for curriculum improvement, and pre- and in-service preparation 19

20. Comprehensive School Improvement Program: Key Assumptions • The purpose of a school is to promote learning. • Learning is a continuous process and must be related to an individual student’s abilities and needs. • Curriculum in all content areas should be built on a continuum from the beginning of the completion of formal education, rather than be frozen by grade levels or age of pupil. • There needs to be a constant and continuous examination of the ways by which schools facilitate learning in order to take advantage of discoveries and developments. 20

21. The Summative Judgment on “Lighthouse” Projects “There was not a willingness on the part of the project’s neighbors to acknowledge its light giving nature…” A Foundation Goes to School, p.42. 21

22. 22

23. A Brief Aside “We must instill in our students the expectation of tedium and disappointment and the duty of thorough persistence, by now well-achieved in the biological and physical sciences. We must expand our students’ vow of poverty to include not only the willingness to accept poverty of finances, but also poverty of experimental results.” Campbell and Stanley, p. 2 & 3. 23

24. Another Approach (Practice at Scale) “To avoid disillusionment several aspects may be noted. First, the claims made (earlier) for the rate and degree of progress which would result from experiments were grandiosely over optimistic and were accompanied by an unjustified depreciation of non-experimental wisdom.” “The initial advocates assumed that progress in the technology of teaching had been slow just because (italics in original) (the) scientific method had not been applied: they assume traditional practice was incompetent just because it had not been produced by experimentation.” 24

25. Evolutionary Perspective on Cumulative Wisdom and Science “..applied practice and scientific knowledge are seen as the resultant of an accumulation of selectively retained tentatives remaining from the hosts that have been weeded out by experience.” “..but the selective, cutting-edge of this process of evolution is very imprecise in the natural setting.” “Experimentation enters at this point as the means of sharpening the relevance of the testing, probing, selection process. Experimentation thus is not in itself viewed as a source of ideas necessarily contradictory to traditional wisdom. It is rather a refining process superimposed upon the probably valuable cumulations of wise practice.” Campbell and Stanley, p. 4. 25

26. We know that about 15% of the variance in school-induced change in achievement is between schools and the rest is in the schools and that at least in relative terms, teachers who add value are widely dispersed. We also know that teachers matter but we are not sure why or how. In elementary school classrooms, 30% of the time is spent on administrative matters. 26

27. Why Classrooms? • Such settings are where development occurs. • Attempts to change individual behavior without affecting its setting are rarely effective. 27

28. Conceptual Articles on Classrooms as Systems Tseng, V., & Seidman, E. (2007). A systems framework for understanding social settings. American Journal of Community Psychology, 39(3-4), 217-228. Cohen, D. K., Raudenbush, S. W., & Ball, D. L. (2003). Resources, instruction, and research. Educational Evaluation and Policy Analysis, 25, 119-142. 28

29. Classrooms as Systems Macro Forces Resources Social Processes Macro Forces Youth Outcomes Social Settings Organization of Resources 29

30. Bob Pianta and colleagues at the University of Virginia http://www.virginia.edu/vpr/CASTL 30

31. Reed Larson and colleagues at the University of Illinois at Urbana-Champaign http://www.youthdev.uiuc.edu/ 31

32. Charles Smith and colleagues at the David P. Weikart Center for Youth Program Quality For staff: http://www.forumforyouthinvestment.org/about/staff For content: http://www.highscope.org/Content.asp?ContentId=117 32

33. Going Forward (Including a Modest Proposal for Secretary Duncan or Some Entrepreneurial Researchers) • Classrooms are the unit of analysis. • Improvements in measurement. • Improvements in non-experimental causal analyses. • Mixed-method work. • An engineering approach to the improvement of change strategies. 33

34. Dear Mr. Secretary: • $1 billion (100 @ $10 million). • Make the welfare waiver approach work for education. • Study shifts in policy but put classrooms in the middle. 34

35. THANKS! 35