National Board Certification: Who Becomes NBPTS Certified and What Are the Consequences for Students

1. 1 National Board Certification:Who Becomes NBPTS Certified and What Are the Consequences for Students? Dan Goldhaber University of Washington and The Urban Institute

2. 2

3. 3 Increase in NBPTS-Certified Teachers

4. 4 Why Should We Care AboutNBPTS Certification? Teachers matter! Unmeasured characteristics - Goldhaber et al. (1999), Rivkin, Hanushek, and Kain (2001), Sanders (1997) Academic Skills - Ehrenberg and Brewer (1994), Strauss and Sawyer (1986), Wayne (2001), Ferguson (1991, 1998) - 1 standard deviation increase in teacher licensure scores raises student scores by .17 standard deviations NBPTS designed to identify �effective� teachers Significant expenditures on NBPTS Rewards associated with NBPTS certification are a significant departure from the single salary schedule Potential long-term influence on teachers� recruitment and career paths

5. 5 Certification Process Two major components of assessment: Portfolio: Classroom practices, teacher reflection & community involvement Videotaped classroom interactions, collected student work (includes assessments, instructional materials), documented work with families, community, colleagues, and professional organizations Assessment center exercises: Content & pedagogical knowledge tests Constructive responses (essays) to challenging classroom situations Must meet performance standard of 275 points Certification process takes 6-12 months and is valid for 10 years Portfolio review: - Due in late spring/early summer, dates vary Assessment center exercises: - Take in summer 2001 with schedule allowing for several weeks for each certificate area - Computer-based testing at Prometric Testing Centers, often located in Sylvan Learning Centers, colleges, universities, and ETS field offices Approximately 60% of the score is appropriated to the portfolio, 40% to the assessment. 40% is for the classroom component, 20% for the community involvement For 2000-2001 certification (applied in Winter 2000-01), score release is scheduled for late November/early December 2001 Points distribution between portfolio and assessment center exercises?Portfolio review: - Due in late spring/early summer, dates vary Assessment center exercises: - Take in summer 2001 with schedule allowing for several weeks for each certificate area - Computer-based testing at Prometric Testing Centers, often located in Sylvan Learning Centers, colleges, universities, and ETS field offices Approximately 60% of the score is appropriated to the portfolio, 40% to the assessment. 40% is for the classroom component, 20% for the community involvement For 2000-2001 certification (applied in Winter 2000-01), score release is scheduled for late November/early December 2001 Points distribution between portfolio and assessment center exercises?

6. 6 Expenditure on NBPTS By 2002, NBPTS had received over $109 million in federal funds $106 million from non-federal sources State and local expenditures Reimbursement of assessment fees Release time and preparation materials One time bonus or annual salary increase

7. 7 State and Local Expenditures on NBPTS(North Carolina Example) The costs are actually substantially higher - More than 1064 are certified in total - The expenditure on12% salary increases may increase over time as teachers stay in the profession and average salary increases Local district example: Union School District provides for an additional $2,000 annually to the teacher salaryThe costs are actually substantially higher - More than 1064 are certified in total - The expenditure on12% salary increases may increase over time as teachers stay in the profession and average salary increases Local district example: Union School District provides for an additional $2,000 annually to the teacher salary

8. 8 Some Key Questions What factors are associated with the decision to seek NBPTS certification? What factors are associated with the success of applicants to NBPTS? Can we judge the effectiveness of NBPTS based on student outcomes Identification of effective teachers NBPTS process as professional development How does NBPTS certification affect teachers� career paths Length of time that teachers remain in the teaching profession Distribution across districts, schools, and students?

9. 9 Key Findings Influences on probability of application Teacher demographics and test scores Districts� financial incentives for successful applicants Influences on probability of certification Major effect of teacher race Strong positive effects of teacher test scores Simulations suggest major reason for increased certification rates is change in applicant pool (stronger applicants) Uneven student outcome results Stronger evidence favoring NBPTS certified teachers for younger students and in math

10. 10 Previous Studies (1): Reasons for Seeking Certification and Perceptions of Effects CPRE case studies Finds applicants in districts were motivated by: Desire to validate their teaching techniques and obtain leadership position, monetary incentive Widespread belief in benefits of NBPTS certification to schools, teachers and students NBPTS surveys Most NBPTS teachers report certification process: Is excellent professional development (80%) Has positively affected teaching practices (91%) Has resulted in positive impacts on student engagement, motivation, and achievement (69%)

11. 11 Previous Studies (2): Student Outcomes Bond study NBPTS teachers outperform non-certified peers in 11 of 13 �key teaching dimensions� Does not use pre- and post-year measures of student performance or adjust for differences among students Stone study NBPTS certified teachers are not particularly effective as measured by TVAAS scores No comparison group or control for student background factors, reliance on 115% as a benchmark for �high performing� Bond study compared 31 successful applicants to 34 unsuccessful applicants Stone study is based on 16 NBPTS certified teachers in TNBond study compared 31 successful applicants to 34 unsuccessful applicants Stone study is based on 16 NBPTS certified teachers in TN

12. 12 Data Sources N. C. Department of Public Instruction Teacher and student records NBPTS/Educational Testing Service NBPTS candidate teacher records Common Core of Data Community and school district characteristics Survey of administrative offices of N.C. school districts Incentives for NBPTS candidates and teachers

13. 13 Why North Carolina Data? Comprehensive accountability system allows for student teacher links Student level (about 1 million obs/year) Background: race, parent education level Subject and year-end test scores: grades 3-12 Teacher level (over 70,000 obs/year) Background: race, age, license, years experience Candidates� scores on NBPTS assessments Strong backing of NBPTS by Gov. Hunt NC has about a quarter of the NBPTS teachers in U.S.

14. 14 Teacher Characteristics

15. 15 School Characteristics

16. 16 District and Community Characteristics

17. 17 Analytic Approach OLS and logit models of the factors associated with Application for NBPTS certification The likelihood of successful certification NBPTS assessment scores Fixed and random effects models to account for potential correlation of error terms across schools/districts & over time Bivariate probit to account for potential correlation of error terms between applicant and certification models

18. 18 Selected Coefficient Estimates

19. 19 Application Model Results Teacher variables Those who are younger, African-American, and/or female are more likely to apply Those fully licensed, and receiving a license from a North Carolina approved teacher training institution more likely to apply Teacher test score and year dummy variables are significant Salary incentive, but not bonus, has an impact Both types of incentives have a statistically significant positive impact on the likelihood that teachers attempt NBPTS certification. Each $1000 increase in the value of the one-time bonus is estimated to increase application by about 1 percentage point, or over 70 percent. The annual raise has a somewhat lessor impact with each $1000 increase in the salary anticipated to increase the probability of application by about 0.4 percentage points, or 29 percent. Both types of incentives have a statistically significant positive impact on the likelihood that teachers attempt NBPTS certification. Each $1000 increase in the value of the one-time bonus is estimated to increase application by about 1 percentage point, or over 70 percent. The annual raise has a somewhat lessor impact with each $1000 increase in the salary anticipated to increase the probability of application by about 0.4 percentage points, or 29 percent.

20. 20 Certification Model Results Those who are younger, female, and/or with Masters degrees are more likely to obtain certification African-Americans far less likely to obtain certification Teacher test score is significant, but no significant difference in slope or intercepts across years Suggests certification is not becoming easier NBPTS assessment model results parallel results from certification models Both types of incentives have a statistically significant positive impact on the likelihood that teachers attempt NBPTS certification. Each $1000 increase in the value of the one-time bonus is estimated to increase application by about 1 percentage point, or over 70 percent. The annual raise has a somewhat lessor impact with each $1000 increase in the salary anticipated to increase the probability of application by about 0.4 percentage points, or 29 percent. Both types of incentives have a statistically significant positive impact on the likelihood that teachers attempt NBPTS certification. Each $1000 increase in the value of the one-time bonus is estimated to increase application by about 1 percentage point, or over 70 percent. The annual raise has a somewhat lessor impact with each $1000 increase in the salary anticipated to increase the probability of application by about 0.4 percentage points, or 29 percent.

21. 21 Simulations Application and certification probabilities, and NBPTS assessment scores over time For a hypothetical teacher with the same characteristics as the 1997 sample mean With changes in teacher test (z-) scores For teachers who switches from an affluent to a disadvantaged school/district Application probabilities with changes in district incentive variables Use these results to simulate costs and benefits

22. 22 Probability of Application(Teacher with 1997 Mean Characteristics)

23. 23 Simulated Application Ratesby Z-Score and Year

24. 24 Simulated Conditional Certification Rates by Z-Score and Year

25. 25 Mean Characteristics of Affluent and Disadvantaged Schools

26. 26 Simulated Application and Certification Rates by School and Teacher Type

27. 27 Simulated Marginal Effects of Local Salary Incentives

28. 28 So is NBPTS IdentifyingEffective Teachers?(Using NC test scores as measure of student gains to measure teacher effectiveness) What are the �Right� Student Outcome Comparisons? Certified versus non-certified teachers Relative effectiveness of NBPTS certified teachers Non-certified applicants versus non-applicants Evidence of teacher �motivation� Non-certified applicants versus certified applicants NBPTS ability to discern teacher effectiveness Certified applicants over time NBPTS process as professional development Multiple grades and subjects

29. 29 Preliminary Evidence: 1997

30. 30 Preliminary Evidence: 1998

31. 31 Preliminary Evidence: 1999

32. 32 Student Achievement Results Relative to students of non-certified applicants in: 18 of 18 cases students of NBPTS teachers had higher end-of-year scores 17 of 18 cases students of NBPTS teachers had higher beginning-of-year scores 9 of 18 cases students of NBPTS teachers had higher growth scores

33. 33 Conclusions Strong correlation (across time) between teacher test scores and application, certification, and NBPTS assessment scores Significant differences by race Clear increase over time in the inclination to apply, but little evidence that the likelihood of certification has gone up over time More academically skilled applicants explain majority of increase in certification rate Uneven student outcome results Stronger evidence favoring NBPTS certified teachers for younger students and in math

34. 34 Next Steps Multivariate analysis of effectiveness of NBPTS assessment process based on student outcomes Identification of NBPTS teachers NBPTS process as professional development Assess impact of NBPTS on teachers� career paths Measure the influence of NBPTS certification on length of time in teaching profession and length of time teaching in particular districts or schools Determine the types of districts, schools, and students are NBPTS-certified teachers most likely to teach Measure the affect of district-specific NBPTS-related incentives on the mobility and distribution of NBPTS-certified teachers within the profession

35. 35 BACKUP SLIDES

36. 36 NBPTS Assessment Model Results Generally parallel results from certification models Interactions between teacher pretest score and year dummies are statistically significant Implies a reduced correlation between pretest scores and NBPTS assessment scores in years following 1997

37. 37 Probability of Certification (Teacher with 1997 Mean Characteristics)

38. 38 NBPTS Assessment Score (Teacher with 1997 Mean Characteristics)

39. 39 Growth in Certification Areas

40. 40 CPRE Study Qualitative study to assess effects of NBPTS certification Conducted interviews with 30 teachers, NBPTS candidates, administrators in 5 districts in 5 different states that vary by level of incentive offered Finds applicants in districts were motivated by: Desire to validate their teaching techniques Obtain leadership position Monetary incentive Strong widespread belief in benefits of NBPTS certification to schools, teachers and students Study by Carolyn Kelley and Steven KimballStudy by Carolyn Kelley and Steven Kimball

41. 41 Bond Study Compares NBPTS certified to non-certified teachers (a total of 65) in 2 states and D.C. Uses teacher surveys, observations and student work to measure 13 areas of teacher effectiveness and 2 areas of student learning Finds NBPTS teachers outperform non-certified peers in 11 of these �key teaching dimensions,� and produce higher achieving students as measured by student work samples Does not use pre- and post-year measures of student performance or adjust for differences among students

42. 42 Stone Study Compares NBPTS-certified teachers to non-NBPTS-certified teachers in Tennessee The study includes only 16 NBPTS-certified teachers Finds that NBPTS certified teachers were not exceptionally successful in improving student achievement The study does not use conventional techniques for demonstrating statistical significance The Value-Added system in Tennessee used here to measure student achievement does not directly account for differences in students� backgrounds

43. 43 NBPTS Surveys Most NBPTS teachers report certification process: Is excellent professional development (80%) Has positively affected teaching practices (91%) Has resulted in positive impacts on student engagement, achievement, and motivation (69%) No comparison group

44. 44 Fixed and Random Effects Application Models Specification tests Random effects models preferred over OLS (Lagrange multiplier test) Reject the null hypothesis of joint significance of school and district fixed-effects being zero Coefficients are quite robust across model specifications Unobserved school and district characteristics important predictors of application, but uncorrelated with included explanatory variables

45. 45

46. 46 Effect Size of Select Variables on Conditional Certification

47. 47 Effect Size of Select Variables on Unconditional Certification

48. 48 NBPTS: Governance & Structure Governed by a 63-member Board of Directors Classroom teachers District administrators and school board members State officials Business and community leaders Union officials Certification Areas Offers certificates in 24 areas Most common in generalist certificates for teachers of multiple subjects (early childhood, middle childhood, and early adolescence)

49. 49 Demographic Information 1997

50. 50 Demographic Information 1998

51. 51 Demographic Information 1999

52. 52 Simulated NBPTS Assessment Scoresby Z-Score and Year