1 / 47

UNIVERSITY OF CENTRAL FLORIDA

UNIVERSITY OF CENTRAL FLORIDA . Mary E. Little, Ph. D. Associate Professor Deb Hahs-Vaughn, Ph. D. Assistant Professor Presentation prepared: CREATE, 2007. Assessing Student Learning from Professional Development in Secondary Content Literacy .

fisk
Download Presentation

UNIVERSITY OF CENTRAL FLORIDA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. UNIVERSITY OF CENTRAL FLORIDA Mary E. Little, Ph. D.Associate ProfessorDeb Hahs-Vaughn, Ph. D.Assistant ProfessorPresentation prepared:CREATE, 2007 Assessing Student Learning from Professional Development in Secondary Content Literacy

  2. Ultimate purpose of professional development and evaluation that takes place in schools is to improve student learning. The purpose of school reform is to impact student learning. All innovation should produce positive results. Fullan, 2005

  3. Session Outcomes • Describe various Content Enhancement Routines to improve content literacy; • Describe planning of professional development and evaluation; • Discuss research questions, methodologies, and results; and • Share instruments, resources, etc.

  4. Why is this Important? “Most systems have enacted accountability policies in the absence of conceptualizing and investing in policies that would increase the capacity of educators to perform in new ways. The system is crucial if we want large-scale, sustainable reform.” (Fullan, 2003b, p. 25).

  5. What are the major policies and research impacting school improvement in Florida, middle level education, and reading? What are the responsibilities of each of the policies, regulations, and procedures? How do each of these impact educators, teachers and students? Once identified, how do each of these policies provide a framework for quality implementation within a system of change (classroom, school, district, and state)? What are the relationships of each within the coordinated and comprehensive framework? How and to whom will these policies be shared and supported? Issues to consider for policy implementation at school level

  6. What instructional practices in reading meet the diverse needs of all students within Florida schools? (Content) Given student performance data, research on reading practices and methods, and disaggregated local data of student performance, how do we select and implement research-based instructional practices in reading?

  7. Philosophical Underpinningsof the Strategic Instruction Model • Most low-achieving students can learn to function independently in classroom settings. • The role of the support-class teacher is to teach low-achieving students strategies that will enable them to be independent learners and performers. • The role of the content teacher is to deliver subject-matter information in a manner that can be understood and remembered by low-achieving students. • Students should have a major voice in decisions about what strategies they are to learn and how fast they are to learn these strategies.

  8. Issues to consider (Process): • WHAT do we KNOW about the reading needs of all of our students, related to the curricular goals? • HOW will each teacher instruct all of the students to improve their reading skills and scores? • HOW will each teacher continue to monitor and adjust instruction to assure continuous improvement for all students?

  9. Effective Professional Development Critical Questions: -What are characteristics of “effective” professional development for continuous improvement that results in improved student achievement. -How will we measure the impact of professional development?

  10. Five Levels of Professional Development Evaluation (Guskey, 2000) • Participants’ Reactions • Participants’ Learning from the Experience • Organization Support & Change • Participants’ Use of New Knowledge & Skill • Results: Student Learning Outcomes

  11. At Each Level, We Must Consider: • What questions are addressed? • How will information be gathered? • What is measured or assessed? • How will the information be used?

  12. Purpose of the Study • To examine changes that occurred at the school, teacher, and student level after implementation of Strategic Instruction Model (SIM) Content Enhancement Routines in middle and high school social studies classes in public schools in a large school district in south Florida.

  13. Description of the School District • A large school district in south Florida (163 schools) expressed their interest in participating in having an external evaluator assist with evaluating the project. • There were approximately 177,000 students enrolled in the district in fiscal year 2005-2006, with about 27,000 or 14.7% receiving services from programs in exceptional education. • This school district employs about 11,600 teachers, 404 of which are National Board Certified. • The average class size is 22 students in middle school and 25 students in high school.

  14. Goal of Reform • Under a recent school district plan, all district schools would include students with disabilities in their academic programs by 2009. Currently, about half of the students with high incidence disabilities (e.g., learning disabilities) spend most of their day in general education classrooms The new plan would increase the numbers of students with disabilities with access to the general education curriculum to 75% over the next few years (Shah, 2006).

  15. Program Development • Established educational goals/objectives for district; developed action plan and implementation team • Reviewed research re: content enhancement reading strategies/routines within secondary content courses • Appropriated budget, resources, and time • Provided initial professional development-almost 700 secondary teachers in social studies • Initiated professional development evaluation plan

  16. Results

  17. Research Question Related to Teachers • Are there average differences in teachers’ knowledge and perceptions of CER for teachers implementing versus non-implementing CER?

  18. Research Questions Related to Students • Are there average differences in students’ American history performance in classrooms where the teacher attended all CER professional development as compared to students in classrooms where the teacher did not attend all CER professional development? • What is the relationship between student characteristics and student gain performance in reading and how is that relationship mediated by the nature of being nested within classrooms where teachers were trained in CER?

  19. Instrumentation • Belief/perception survey • Knowledge assessment of CER • Level of implementation survey • Direct observation rubric (fidelity) • Curriculum-based instrument (social studies curriculum) • State Assessment: Florida Comprehensive Assessment Test (FCAT) reading for grades 8 and 10

  20. Teacher Sample • Seven hundred forty social studies teachers received initial professional development in the social studies curriculum alignment, research-based instructional methods and routines, and related resources at the beginning of the 2005-2006 school year. • The sample consisted of 61 teachers who attended all training and follow-up professional development and who thereby represented teachers in the county who were trained on all components of Content Enhancement Routines.

  21. Teacher Sample • Nearly ¾ of respondents were female (n = 45; 74%). • About 80% (n = 49) were over the age of 35. • About 56% (n = 34) of respondents indicated that "middle school teacher" was most closely aligned with their current position with most of the remaining (n = 25; 41%) classifying their position as “secondary school teacher.” • Over 90% (n = 57) of participants indicated that they were regular education teachers. One respondent was an ESE teacher and one respondent was a gifted teacher. • Nearly 80% (n = 48) of the respondents were white non-Hispanic

  22. Teacher Sample • Nearly 50% (n = 24) of respondents held a master's degree or higher. • The mean number of years employed in an instructional position was 15.92 (SD = 9.13) with a median of 16.0. The minimum number was 2 years and the maximum number was 34 years.

  23. Teacher Sample • About 80% of respondents indicated they were teaching in-field and 5% indicated they were teaching out of field (15% did not reply). • Regarding type of certification, 72% indicated they completed the certification test. Although only 13% (n = 8) indicated they received a four year college degree in education, this number may be underestimated if respondents misunderstood the question and did not assume that "four year college degree in education" also implied that they completed the certification test. • Nearly 35% (n = 21) of respondents indicated that they had an ESE teacher supporting them in the classroom.

  24. Results: Research Question Related to Teachers • Are there average differences in teachers’ knowledge and perceptions of CER for teachers implementing versus non-implementing CER?

  25. Results: Research Question Related to Teachers

  26. Results: Research Question Related to Teachers

  27. Research Questions Related to Students • Are there average differences in students’ American history performance in classrooms where the teacher attended all CER professional development as compared to students in classrooms where the teacher did not attend all CER professional development? • What is the relationship between student characteristics and student gain performance in reading and how is that relationship mediated by the nature of being nested within classrooms where teachers were trained in CER?

  28. Research Question 1: Students • Are there average differences in students’ American history performance in classrooms where the teacher attended all CER professional development as compared to students in classrooms where the teacher did not attend all CER professional development?

  29. Student Sample (Research Question 1) • All students enrolled in American history were required to complete the district exam at the end of the academic year as part of the course requirement. • Data were collected from American history classrooms in which the teacher had completed all training and follow-up on CER (regardless of their observed level of CER implementation) (n = 16) as well as a matched sample of teachers who had not completed CER professional development (n = 15).

  30. Research Question 1: Students

  31. Research Question 1: Students

  32. Research Question 2: Students 2. What is the relationship between student characteristics and student gain performance in reading and how is that relationship mediated by the nature of being nested within classrooms where teachers were trained in CER?

  33. Student Sample (Research Question 2) • There were 566 students who had 2005 and 2006 FCAT SSS reading developmental scale scores (thus allowing the calculation of a gain score) who were instructed by teachers that had attended all the CER training and follow-up (n = 15 teachers), and who had not withdrawn from school by April 2006.

  34. Student Sample (Research Question 2) • Of these students, 100 (18%) were enrolled in American history and 466 (82%) were enrolled in world history. • Approximately 55% of students were female (n = 312). • Over one-half (54%) of students were of a non-White race with most being Black (n = 137, 24%) and Hispanic (n = 134, 24%). • A small proportion of the students were classified as exceptional education students (n = 29; 5%), limited English proficient (n = 63, 11%), or as receiving free or reduced lunch (n = 146, 26%). • Only one student was classified as both exceptional education and limited English proficient. • Seven students (1%) were classified as both exceptional education and receiving free and reduced lunch. • About 3% of students (n = 19) received developmental language instruction. • Nearly 42% of students were enrolled in an intensive reading class (n =237).

  35. Student Sample (Research Question 2) • There were 15 teachers who instructed the students in this sample, and there were 10 schools represented. • Nine of the 15 teachers had been observed in the social studies classroom for implementation of Content Enhancement Routines. Based on observations in the classroom, six of the 15 teachers were CER implementers (40%), three were considered non-CER implementers (20%), and the remaining did not have classroom observation data gathered on them (40%). • Of the 15 teachers represented, approximately two-thirds (n = 10) were female, and nearly one-half (n = 7, 47%) were 45 years or younger. • All but one of the teachers was white (93%). • Nearly one-half of the teachers had completed their bachelor's (47%) or master's degree (47%). • Only two teachers (14%) had taught three years or less. Most teachers had taught 13 or more years (71%). • Twenty percent (n = 3) of teachers stated they were teaching out of field. • Three teachers (20%) indicated that they had an ESE teacher supporting the ESE students in their classrooms. • Over one-half (53%) indicated that their ESE students were making achievement gains in the classroom, and nearly one-half (47%) indicated that their limited English proficient students were making achievement gains in the classroom.

  36. Results: Research Question 2 (Students) • Model 1: Unconditional • The results indicate that 6.17% of the variation in reading gain can be attributed to between-teacher differences (ICC = .0617) with the majority of variation in reading gain being attributed to within teacher/classroom differences (93.83%). • The teacher-level (level 2) variance was statistically significant, c2 (14) = 40.49, p < .001, suggesting that a large amount of variability exists between teachers.

  37. Results: Research Question 2 (Students) • Model 2: Conditional with Level 1 Predictors • A conditional model was estimated next that included the four student-level predictors (exceptional education student status, limited English proficiency status, free or reduced lunch status, and enrollment in intensive reading course) but no predictors at the teacher-level (level 2).

  38. Results: Research Question 2 (Students) • Model 3: Conditional with Level 1 and Level 2 Predictors • The five teacher-level predictors described previously were included in the model next. All teacher-level variables were grand mean centered because they did not have a meaningful zero point. In examining the results from this model, there were no variables that were contributing to variation in reading gain at the teacher-level. Thus the level 2 predictors were removed.

  39. Results: Research Question 2 (Students)

  40. Results: Research Question 2 (Students) • The average student reading gain per teacher was 51.15. • The model for the effect of limited English proficiency indicated that, on average, students with limited English proficiency had an advantage as did students who were enrolled in an intensive reading course. • The model for the effect of free or reduced lunch status indicated that, on average, students that did not receive free or reduced lunch had an advantage. • The pooled within-teacher (or student-level) variance was 30716.47, and the variance among the teacher-level reading gains was 1992.89 suggesting that much remains unexplained after taking into account predictors at level one (student-level). • The proportion of variance in reading gain explained by the student level model is approximately 7.19%.

  41. Limitations • No causal comparisons • Self-report data • Non-random sample (and thus violation of assumption of independence) • Lack of ability to conduct a three-level hierarchical model • Limitations associated with developmental scale scores

  42. Implications for Practice • This research supports current standards for effective professional development related to student outcomes (NSDC, 2002). • Instructional changes and resulting student outcomes can be produced in schools. However, for such reforms to affect instruction in powerful ways, the instructional practices of school reforms must carefully address a core set of characteristics and consistently provide the varying degrees of supports for teacher learning and implementation to impact student learning.

  43. Implications for Practice • This research supports the need for continued research into the impact of innovative programs related to the conditions and context of classroom implementation (Bulgren, 2006; Correnti & Rowan, 2007).

  44. ”Never doubt that a small group of thoughtful committed people can change the world: indeed it is the only thing that ever has.” Margaret Mead

  45. References • Berends, M. Bodilly, S. & Kirby, S. (Eds.) (2002). Study of instructional school sampling design. Ann Arbor: University of Michigan, Institute for Social Research. • Borko, H., Wolf, S., Simone, G., & Uchiyama, K. (2003). Schools in transition: Reform efforts and school capacity in Washington State. Educational Evaluation and Policy Analysis, 25(2), 171-203. • Borman, G., Hewes, G., Overman, L., & Brown, S. (2003). Comprehensive school reform and achievement: A meta-analysis. Review of Educational Research, 73,125-230. • Bulgren, J.A., Lenz, B.K., Marquis, J.G., Deshler, D.D., Schumaker, J.B., Davis, B., & Grossen, B. (2006). The instructional context of inclusive secondary general education classes: Teachers’ instructional roles and practices, curricular demands, and research-based practices and standards. Learning Disabilities: A Contemporary Journal, 4(1), 39-65. • Bulgren, J. A. & Lenz, B. K. (1996). Strategic instruction in the content areas. In D. Deshler, E., S. Ellis, & B. K. Lenz (Eds.) Teaching adolescents with learning disabilities: Strategies an methods (2nd ed., pp. 409-473). Denver: Love Publishing. • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum. • Correnti, R. & Rowan, B. (2007). Opening up the black box: Literacy instruction in schools participating in three comprehensive school reform programs. American Educational Research Journal, 44(2), 298-338. • Cuban, L. (1993). How teachers taught: Constancy and change in American classrooms 1880-1990. New York: Teachers College Press. • Deshler, D. D. (2006). Building a strong literacy foundation for high school excellence. In D. Clark (Ed.). The challenge of high school reform. 21(2), 20-27. Washington, DC: The Aspen Institute. • Deshler, D.D. & Hock, M.F. (2006). Adolescent literacy: Where we are-Where we need to go. University of Kansas Center of Research on Learning: Author. • Deshler, D.D. & Tollefson, J.M. (2006). Strategic interventions: A research-validated instructional model that make adolescent literacy a school-wide priority. The School Administrator, 24-29. • Deshler, D.D. & Lenz, B. K. (2004). The strategic instruction model in secondary classrooms. Lawrence, KS: Edge. • Ellis, E. (1998). FRAMING Routine. Lawrence, KS: Edge Enterprises. • Elmore, R., & Burney, D. (1997). Investing in teacher learning: Staff development and instructional improvement in school district #2, New York City. Philadelphia Consortium for Policy Research in Education and the National Commission on Teaching and America's Future: Author. • Firestone, W.A. & Corbett, H.D. (1988). Planned educational change. In N. J. Boyan (Ed.), Handbook of research on educational administration, 321-344. White Plains, NY: Longman. • Florida Center for Reading Research. (n.d.). Strategic Instruction Model (SIM) and Content Literacy Continuum (CLC). Tallahassee, FL: Author. • Fullan, M., Bertani, A., & Quinn, J. (2004). New lessons for district wide reform [Electronic version]. Educational Leadership, Leading in Tough Times, 61(7), 42-46.

  46. References • Gersten, R., Fuchs, L.S., Williams, J.P., & Baker, S. (2001) Teaching reading comprehension strategies to 71(2), 279-320. • Huberman, A., & Miles, M. (1984). Innovation up close: How school improvement works. New York: Plenum. • Individuals with Disabilities Education Act (IDEA) Amendment. (1997). 105th Congress. • Individuals with Disabilities Education Act (IDEA) Regulations. (2004). (Vol. 48 C.F.R. & 400.500(4)(c)). • Lenz, B. K. & Deshler, D. D. (2004). Teaching content to all: Evidence-based inclusive practices in middle and secondary schools. Boston, MA: Allyn & Bacon.. • Lenz, B. K., Deshler, D.D., & Schumaker, J. B. (1993). [The development and validation of planning routines to enhance the delivery of content to students with handicaps in general education settings]. Unpublished raw data. • Lenz, B. K., Schumaker, J. B., Deshler, D. D., & Bulgren, J.A. (1998). The course organizer routine. Lawrence, KS: Edge Enterprises. • Lenz, B. K. with Bulgren J., A., Schumaker, J. B., Deshler, D. D., & Boudah, D. A. (1994). The unit organizer routine. Lawrence, KS: Edge Enterprises. • Little, M. (2006) Comprehensive school reform based on student results. Forum on Educational Policy. Oxford, UK: Oxford Press. • McLaughlin, M. J. & Thurlow, M. (2003). Educational accountability and students with disabilities: Issues and challenges. Educational Policy, 17(4). 431-450. • McLeskey, J., Hoppey, D., Williamson, P., & Rentz, T. (2004). Is inclusion an illusion? • An examination of national and state trends toward the education of students with learning disabilities in general education classrooms. Learning Disabilities Research and Practice, 19(2), 109-115. • No Child Left Behind Act. (2002). 107th Congress. • Nunnery, J. (1998). Reform ideology and the locus of development problem in educational restructuring. Education and Urban Society, 30(3), 277-295. • Pascopella, A. (2003). The next challenge. District Administrator, 39(6), 24-29. • Raudenbush, S. W. & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage. • Raudenbush, S. W., Bryk, A., & Congdon, R. (2005). Hierarchical Linear and Nonlinear Modeling Software (version 6.01). Chicago, IL: Scientific Software International. • Stringfield, S. & Datnow, A. (1998). Scaling up school restructuring designs in urban schools. Education and Urban Society, 30(3), 269-276. • Swanson, L., Hoskyn, M., & Lee, C. (1999). Interventions for students with learning disabilities: A meta-analysis of treatment outcomes. New York: Guilford Press. • United States Department of Education (2000). To assure a free and appropriate public education of all children with disabilities. Twenty-fourth annual report to Congress on the implementation of the Individuals with Disabilities Education Act. Washington, DC: Office of Special Education Programs.

  47. Contact Information • Mary E. Little, Ph.D.Associate ProfessorUniversity of Central FloridaPO Box 161250Orlando, FL 32816-1250mlittleucf@aol.com • Debbie L. Hahs-Vaughn, Ph.D.Assistant Professor, Educational ResearchUniversity of Central FloridaPO Box 161250Orlando, FL 32816-1250dhahs@mail.ucf.edu

More Related