Too Good to Be True: Can a Brief Writing Intervention Really Close the Achievement Gap?

1. Too Good to Be True: Can a Brief Writing Intervention Really Close the Achievement Gap? October 13, 2010 School of Education Research Seminar Series Natasha K. Bowen UNC School of Social Work

2. Acknowledgments Kate M. Wegmann and Kristina C. Webber orchestrated the intervention at the middle school. Funding for teacher incentives was provided by a Pfouts grant from the School of Social Work.

3. Previous Research • Cohen, Garcia & Apfel, 2006, Science, 313, 1307-1310. • Report on two studies involving 119 African American and 124 European American 7th graders, over 1 school term • Random assignment; 2 conditions • treatment students were asked to indicate their most important value • control students their least important value. • Regression analyses • 2-way interactions • Dummy codes representing 3 teachers • Repeated measures, & means tests also presented

4. Previous Research • Cohen, Garcia & Apfel, 2006, Science, 313, 1307-1310. • Showed race by condition effects • .26 and .34 in GPA metric in the 2 studies; claimed 40% reduction in grade gap between African and European American students • Bigger effect for lower performing students • No effect for European American students • Conducted additional tests to demonstrate that psychological threat was the mechanism

5. Previous Research • Cohen, Garcia, Apfel, & Master. (2009). Science, 324, 400-403. • Follow-up of 2006 study (but now 3 cohorts and different sample sizes from those reported earlier) • Original study took place at beginning of 7th grade • Follow-up examined effects on performance at end of 8th grade • Booster treatments provided in 7th grade • 385 total subjects • Regression • 3 way interaction among race, condition, prior performance

6. Previous Research • Cohen, Garcia & Apfel, Purdi-Vaughs, Apfel, & Brzustoski. (2009). Science, 324, 400-403. • Race by condition effects continued through 8th grade • Better performance for African American students in TX • No change for European American students In sum, long-term effects of a brief writing intervention on performance, with biggest effect on low achieving African American students

7. Theory of Psychological Threat Worry about reinforcing a negative stereotype about a group one belongs to represents a threat that may cause stress and interfere with performance. Strong empirical evidence of (a) the existence of stigma-based threat, (b) the ease with which threat can be manipulated, and (c) the magnitude of the potential impact on performance Across groups, ages, situations

8. Transactional Developmental Processes Long term effects due to changing trajectories of performance.

9. Durham Study SAMPLE: Population of students at the middle school—622 students, --Quarterly grade data on 585 students (absences and moves) African American 70% (407) Hispanic/Latino 20% (117) White 5% (30) Multi-racial 3% (17) Asian 2% (14) Other

10. Durham Study DESIGN: 3X2 Experimental Random assignment of students to 3 writing prompts Neutral Self-affirming Group-affirming Random assignment of teachers to 2 reading conditions Read essays of all students Don’t read essays of any students

11. Durham Study MEASURES: • Gender: 0=female; 1=male • Race/ethnicity: Examined AA and HL subsets • Teacher condition: 0=did not read essays; 1=did read essays • Child condition: self=1 0; group=0 1; neutral=0 0 • Quarterly grades in Math, Language Arts, and Social Studies (100 point scale) • (pre- and post-EOG scores in math and reading)

12. Durham Study PROCEDURES: Random assignment of students and teachers Preparation of envelopes for students Distribution of envelopes and writing exercise Teachers in reading condition, read essays End of year preparation of grade data linked to study codes (anonymous data provided to researchers)

13. Durham Study HYPOTHESES: Students whose teachers read their essays will have better grades. Students who receive the self-affirming prompt will have better grades than those who receive the neutral prompt. Students who receive the group-affirming prompt will have better grades than those who receive the neutral prompt. Students who receive an affirmation prompt and whose teachers read their essays will have the best grades.

14. Durham Study ANALYSIS: 3-Level Hierarchical Linear Modeling Quarters at level 1 (nested in individual students) Individual students at level 2 Classrooms at level 3 Random intercept and slope (quarters) Tested all 3-way interactions and lower terms To avoid 4-way interactions and to identify within- group effects, analyzed AA and HL separately

15. Results Overview • Four significant treatment effects were found • All involved 3-way interactions • 2 for African American students • 2 for Hispanic Latino students • Generally in hypothesized direction • Language arts and social studies • No effects found on math

16. Interaction for African American Students Among Time, Teacher Condition, and Writing Prompt: Social Studies Grades ……………….

17. Interaction for African American Students Among Time, Gender, and Writing Prompt : Language Arts Grades ? girls

18. Interaction for Hispanic/Latino Students Among Time, Gender, and Writing Prompt: Social Studies Grades

19. Interaction for Hispanic Latino Students Among Time, Teacher Condition, and Writing Prompt: Language Arts Grades

20. Hypotheses Students whose teachers read their essays will have better grades. African American students whose teachers read their essays had better Social Studies grades than students with teachers who did not read their essays, even when they wrote on the neutral prompt. Hispanic/Latino students who wrote on the group prompt AND whose teachers read their essays had the best language arts grades. Otherwise, teacher reading was not associated with better grades.

21. Hypotheses 2. Students who receive the self-affirming prompt will have better grades than those who receive the neutral prompt. Hispanic/Latino boys who wrote on the self-affirming prompt had higher grades in Social Studies than did H/L boys who received the neutral prompt. H/L girls with the self-affirming prompt had higher SS grades after Quarter 2 than boys and all those with the neutral prompt, and a positive grade trajectory. African American students who wrote on the self-affirming prompt AND whose teachers read their essays had better grades than students who had other prompts and whose teachers did not read their essays.

22. Hypotheses 3. Students who receive the group-affirming prompt will have better grades than those who receive the neutral prompt. AA students with the group prompt had higher Social Studies grades than those with the neutral prompt at all time points IF their teachers also read their essays. The same was true for HL students with the group prompt for Language Arts grades. African American Girls with the group prompt had Language Arts grades BELOW those with the neutral prompt.

23. Hypotheses 4. Students who receive an affirmation prompt and whose teachers read their essays will have the best grades. Significant interactions between child and teacher conditions occurred for Hispanic/Latino students and for African American students. --For African American students, Social Studies grades were the highest when the child and teacher conditions were combined (teacher read essay, child wrote on self-affirming prompt). --For Hispanic/Latino students, Language Arts grades were the highest when the child and teacher conditions were combined (teacher read essays, child wrote on group-affirming prompt).

24. Implications Language Arts and Social Studies grades of African American and Hispanic/Latino students appear to be positively affected by a brief writing intervention across middle school grade levels. The intervention affected both intercepts and slopes of quarterly grade trajectories. For African American students, the combination of affirming prompt and teaching reading essays was most effective. For Hispanic/Latinos, the self-affirming prompt was associated with better grades even without teacher reading the essays. Group prompt most effective when combined with teacher reading essay. No effect on Math in current analyses.

25. Implications Classroom factors that can be easily manipulated may have a notable effect on performance. Consistent with stereotype literature Writing intervention per se probably less important that teacher learning more about students, seeing positive side of student, in combination with student experiencing positive sense of self and identity in the learning location

26. Still to Do Initial test with incomplete data shows strong effect of the self affirming prompt on Hispanic Latino EOG scores, controlling for prior year EOG scores. Compare grade levels Examine effects within subject area taught by teaching administering the intervention See if there were effects for Other racial/ethnic groups at the school (closing the gap or improving performance of all?)