1 / 36

Brian R. Flay, D.Phil. Department of Public Health Oregon State University

School-Based Randomized Trials of Social and Character Development Programs With preliminary results of the Positive Action program. Brian R. Flay, D.Phil. Department of Public Health Oregon State University. Why Social and Character Development. Increasing behavioral problems in schools

mac
Download Presentation

Brian R. Flay, D.Phil. Department of Public Health Oregon State University

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. School-Based Randomized Trials of Social and Character Development ProgramsWith preliminary results of the Positive Action program Brian R. Flay, D.Phil. Department of Public Health Oregon State University Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  2. Why Social and Character Development • Increasing behavioral problems in schools • Including violence, substance use, unsafe sex, and other disciplinary referrals • Seems to have worsened with increased attention to instruction for NCLB • Link of behavior management to teacher time on task (teaching) • Link of student behavior to student time on task (learning) Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  3. Limitations of Other Programs • Domain Specific • Usually only one behavior or one skill • Start Too Late • Upper elementary or middle school • Limited Intensity and Dose • Often only once a week for 10-20 sessions • Ecologically Limited • Usually only classroom • Also need school-wide, parent, community • Limited Effect Sizes • Average effect sizes in the 0.2 to 0.4 range • Effects not Sustained • Few effects beyond one year, let alone H. S. Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  4. IES Program of Research on Social and Character Development (SACD) • To conduct rigorous research on comprehensive social and character development (SACD) programs • To compare 7 comprehensive SACD programs in a national cooperative agreement • UIC/OSU to conduct a clustered (school-based) randomized controlled trial of the effectiveness of the Positive Action program on: • Parent involvement with schools • Teacher classroom management • Student character and behavior • Student school attendance and achievement test scores Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  5. Funded Projects • New York University: • Reading, Writing, Respect and Resolution (4 R’s) • Vanderbilt University: • Love in a Big World (LBW) Program • U of North Carolina, Chapel Hill: • The Competence Support Program (CSP) • U of Illinois at Chicago with Oregon State University: • The Positive Action (PA) Program • Brian Flay, David DuBois and Carol Allred • U of Maryland: • Second Step • Children’s Institute: • Promoting Alternative Thinking Strategies (PATHE) Curriculum • State U of NY Buffalo: • Academic and Behavioral Competencies (ABC) Program Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  6. The Positive Action Program Targets Multiple Behaviors • By teaching that doing positive actions helps: • individuals develop a positive self identity. • families develop a positive family identity. • schools develop a positive school identity. • communities develop a positive community identity. • By teaching that: • When you do good, you feel good • And there’s always a positive way of doing things Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  7. The Positive Action content is taught school-wide through six units: • Unit 1. Self-Concept: What It Is, How It’s Formed, and Why It’s Important (the Thoughts-Actions-Feelings Circle). • Unit 2. Physical and Intellectual Positive Actions for Body and Mind (includes motivation to learn) • Unit 3. Social/Emotional Positive Actions for Managing Yourself Responsibly • Unit 4. Social/Emotional Positive Actions for Getting Along with Others by Treating Them the Way You Like to be Treated (Character Education and Social-Emotional Learning) • Unit 5. Social/Emotional Positive Actions for Being Honest with Yourself and Others (Mental Health) • Unit 6. Social/Emotional Positive Actions for Improving Yourself Continually Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  8. The POSITIVE ACTION Program Components • K–12 classroom curriculum • over 1,200 lessons - using Teacher’s Kits (manuals and materials for each grade), classroom teachers present 15–20-minute lessons • Principal’s Kits (Elementary and Secondary) • a school-climate program to promote the practice and reinforcement of positive actions in the whole school population (students and staff) • Counselor’s Kit • used with selected individual students, small groups and families • Family Kit • contains prepared weekly home lessons paralleling the school program along with school parent-involvement activities • Community Kit • manuals and materials that align and encourage collaboration of all the environments (schools, families and community) involved in the program Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  9. Grade 3 Instructor’s Kit Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  10. Elementary Climate Kit • Positive Notes • Notepads • Positive Action News • Music – 27 songs • CDs • Song books • Posters • Video • Overview • Implementation Plan • Scope and Sequence Booklet • Principal’s manual • Support Staff manual • Parents’ manual • Assemblies • Certificates of Achievement • Positive Action Balloons • Positive Action Calendar • Words-of-the-Week Cards • Reinforcement Stickers • ICU Boxes and Notes • Positive Action Tokens Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  11. In a prior quasi-experimental study (Flay, Allred & Ordway, 2001), relative to elementary schools without PA, matched schools with PA reported: • Major reductions in problem behaviors • Up to 85% reductions in violence • Up to 71% reductions in substance use • Up to 90% fewer general disciplinary actions (with effects sometimes being larger in schools with higher levels of student poverty) • Up to 80% fewer suspensions • Up to 94% reductions in criminal bookings • Major improvements in school performance • Up to 60% reductions of absenteeism, • Up to 13% lower rates of chronic absenteeism. • Up to 100% improvements on standardized achievement scores, • Many schools report moving from one of the lowest scoring to one of the highest scoring in their district or state, • The matched-control study was replicated by Flay & Allred (2003) with the addition of pretest matching data. Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  12. Relative to middle/high schoolswith a low proportion of PA graduates, middle/high schools with a high proportion of PA graduates reported: Major reductions in problem behaviors Major improvements in school attendance and retention Major improvements in academic achievement Including proportions graduating from high school And higher proportions going on to higher education With some of these effects being larger in high-mobility schools Strong dose-response relationship for all outcomes, with stronger effects occurring in middle/high schools with greater numbers of PA graduates Flay & Allred (2003) also found long-term effects in middle high schools Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  13. Chicago Trial of Positive Action • UIC/OSU school-based randomized trial • Schools randomly assigned from 7 matched pairs • Matched on school-level demographic variables, achievement and disciplinary referrals • Data collected from students and their teachers and parents, and school leadership (principal and PA Coordinator) • Beginning and end of grades 3 & 4 and end of grade 5 • Program Implementation and Outcomes • Teacher amount and integrity of program delivery • Parent involvement with program and schools • Teacher classroom management • Student character and behavior • Student school attendance and achievement test scores • Also school records data on achievement, attendance and disciplinary referrals • Last wave of data collection Spring 2007 Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  14. Research Design • 7 matched pairs of schools randomly assigned to 2 conditions: • Early Starters: Start the program in the 2004-05 school year OR • Late Starters: Start the program 3 years later (2007-08 school year) • Schools eligible for inclusion were: • Community-based (No Magnet or Charter schools), Have not already used Positive Action, Not already participating in related projects (Project Northland, All Stars), • Enrollment > 50 and < 140 students per grade • Mobility rate no greater than 40% • More than 50% of students receive free or reduced price lunch • Less than 50% of students met achievement criteria on the ISAT • Schools matched into pairs before randomization: • Achievement scores, School size – enrollment, Ethnic Distribution, % Mobility, % Free/Reduced Lunch, Attendance and Truancy Rates, % Parent Involvement, % Teachers not meeting minimal requirements, CPS Region, Community crime statistics Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  15. Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  16. Research Design 2 • Follow one cohort of students from the beginning of grade 3 to the end of grade 5 • Signed parental consent – 98.3% returned, 79.7% Yes • Surveys of students, teachers and parents at beginning and end of grades 3 and 4, and end of grade 5. • Teachers complete behavior rating scales on students in their class at beginning and end of grades 3 and 4, and end of grade 5. • Some of the measures are multi-site – collected by a national contractor (Mathematica Policy Research – MPR) at each of the 7 sites – and not available to PIs until about a year later • Some of the measures are site-specific – specific to our evaluation of the Positive Action program – collected by UIC research staff Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  17. Measures and Analysis Plans • Cross-site data collection includes measures of • social competence, • character, • behavior and • achievement. • Site-specific data includes extensive process measures of • fidelity of implementation and • dosage of exposure for all components of the PA program • As well as additional measures of hypothesized mediators of the intervention’s effects. • We will use hierarchical statistical models (random regression models and GEE) that can accommodate longitudinal data with nested observations and missing observations. Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  18. Hawai`i Trial of Positive Action • Funded by NIH-NIDA • Schools randomly assigned from 10 matched pairs • Matched on school-level demographic variables, achievement and disciplinary referrals • Data collected from students and their teachers and parents, and school leadership (principal and PA Coordinator) • Beginning and end of grades 1/2 and end of grades 2/3, 3/4, 4/5 and 5/6 • Program implementation and outcomes • Teacher amount and integrity of program delivery • Parent involvement with program and schools • Teacher classroom management • Student character and behavior • Student school attendance and achievement test scores • Also school records data on achievement, attendance and disciplinary referrals • Last wave of data collection Spring 2006 Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  19. Baseline Equivalence: 2000-01 No differences are close to being statistically significant except % free/reduced lunch p = .099 Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  20. Some Preliminary Results from the Hawai`i Trial • Final wave of data not yet completely available or merged/cleaned • Preliminary results from the end of the 3rd year of the intervention • Simple graphs shown of examples of significant effects • Some variability in dose, so graphs show effects by level of implementation. Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  21. Teacher rating of child behavior scale Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  22. Student Self-Report of Behavior Scale by Level of Implementation and Control Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  23. Student Rating of Feeling About Self When They Do Each Behavior by Level of Implementation and Control Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  24. Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  25. Fig. 15: % of 5th graders reporting ever having had voluntary sex 5.00 C PA 4.00 3.00 2.00 1.00 0.00 -1.00 Girls Boys Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  26. Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  27. Parents Talk with Child Parents Listen/Attend/Help School PA Helps Their Child on Unit 2 - Intellectual Parents Help Their Child on Unit 6 – Self-Development Parents Negative Attention To Child Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  28. Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  29. Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  30. Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  31. Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  32. Relationships between teacher perceptions of school climate, beliefs about SACD, attitudes toward PA and three elements of program implementation Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  33. Student responsiveness partially mediates program outcomes The first (estimates outside the parentheses) is the model run with the clustering option specified using Wave 4 classrooms. The second (estimates inside parentheses) does not specify the clustering option. This is the reason for the varying Ns between the two model fit indices. Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  34. Summary of Findings • Baseline equivalence was established • Matching and random assignment was successful • We observed some variation between schools in level of PA Implementation • Evidenced in teacher process data • Also reflected in student behavior and achievement • Important positive program effects are emerging (and should be larger at later waves): • Teacher ratings, Student ratings, School Records • Clear differential effects by level of implementation • Reflected in both teacher and student data • The more PA is implemented, the larger the effects for both teachers and students, on both behavior and achievement • Some important causal models of effects of school leadership and student responsiveness Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  35. The Positive Action Program How Does Positive Action Work? Teachers: Improved Classroom Management Teachers and Students: Increased Time On Task Students: Improved Behavior Students: Improved Academic Scores Students: Increased Attendance Students: Improved Feelings About Self Students: Increased Motivation To Learn Parents: Increased Involvement in School Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

  36. Future Work/Needs • Larger scale trials • ICCs for attitudes (.03-.1) and behavior (.01-.05) are generally smaller than for achievement (.15-.2) • Still need Ns of 10-20 per condition rather than 5-7 • Improved measures of integrity and dosage delivered and received • Teacher, student and observer reports • Contractual reporting systems? • Longer term follow-ups • Effects take several years to even start emerging • Prior work suggests important long-term effects are possible • Methods of analysis to accommodate differential implementation • Propensity scoring, CACE, instrumental variable Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006

More Related