1 / 36

California English Language Development Test: Developing the Early Literacy Assessment Components

California English Language Development Test: Developing the Early Literacy Assessment Components Panel Presentation at the Tenth Annual Accountability Leadership Institute Burlingame, CA December 8, 2009.

lpardo
Download Presentation

California English Language Development Test: Developing the Early Literacy Assessment Components

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. California English Language Development Test: Developing the Early Literacy Assessment Components Panel Presentation at the Tenth Annual Accountability Leadership Institute Burlingame, CA December 8, 2009

  2. Section 1 –Introductions,Background, and Policy RequirementsLily Roberts, Administrator,California English Language Development Test (CELDT) Office K-1 Early Literacy Assessment

  3. Presentation Overview • Background and policy requirements • Item development cycle • Guiding principles for test development and administration • Field test and lessons learned • Development and technical issues • Discussion • Questions and answers K-1 Early Literacy Assessment

  4. Background • 2007: State legislature authorized development of the Kindergarten and grade one (K-1) early literacy assessment • 2008: Initial item development and expert panel was convened • 2009: Field test was conducted in March • 2009-10: Operational testing in all four domains for all grades (K-12) • 2010: Standard setting will be conducted to recommend performance levels K-1 Early Literacy Assessment

  5. Policy Requirements • Title III, Elementary and Secondary Education Act requires testing in the domains of listening, speaking, reading, and writing in all grades • California Education Code (EC) Section 60810 defines the purposes for the CELDT: • Identify limited English proficient students • Determine the level of English language proficiency • Assess progress acquiring the skills of listening, speaking, reading, and writing K-1 Early Literacy Assessment

  6. Reclassification Criteria • EC Section 313 specifies four criteria for local reclassification decisions, including: • Assessment of English language proficiency • Performance in basic skills • Teacher evaluation • Parental opinion and consultation K-1 Early Literacy Assessment

  7. Section 2 -Initial Item DevelopmentEdynn Sato, WestEd K-1 Early Literacy Assessment

  8. 2008 Item Development • Subject matter experts participated in guiding item development • Conducted two rounds of content and bias reviews • Developed 142 K-1 reading and writing test questions • Twelve reading and writing English Language Development (ELD) standards were identified as important and assessable for K-1 by subject matter experts K-1 Early Literacy Assessment

  9. Item Development Cycle • Test specifications are created based on ELD standards • Guiding principles for item development are established • Item specifications are created to guide item development • Item writers and editors receive training • Items are written and edited • Graphics are created K-1 Early Literacy Assessment

  10. Item Development Cycle (cont.) • Lead editors and content director review and sign-off on items that are ready for CELDT Office and committee reviews • Interdepartmental staff reviews items/graphics • Subject matter experts review items for content • Subject matter experts review items for bias • Necessary edits are made K-1 Early Literacy Assessment

  11. Guidelines for Item Writing • General Guidelines • Typical representations • Reflect a range of cultural and ethnic diversity • Familiar to a range of students • Specific Guidelines • Specific, detailed, and clear • Contextual detail to enhance comprehensibility • People in graphics • Age, gender, ethnicity K-1 Early Literacy Assessment

  12. Specific Questions that Need to be Asked during Content Review • Is the item aligned to the correct ELD Standard? • Does the item test a knowledge or skill that is appropriate for the assigned domain? • Is the item aligned to the correct grade span? • Does the item test for current and worthwhile concepts? • Does the item stem give a full sense of what is being asked? K-1 Early Literacy Assessment

  13. Specific Questions (cont.) • Are the directions clear? • Does the stem avoid wordiness? • Is the key the only correct answer? • Does the item use distractors that relate to the stem in the same way as the key? • Does the item use distractors that are plausible misconceptions and errors? • Does the item contain clues to the correct answer? K-1 Early Literacy Assessment

  14. Specific Questions (cont.) IF THE ITEM CONTAINS GRAPHICS… • Is the graphic required to answer the item? • Does the graphic provide sufficient information needed to answer the item? K-1 Early Literacy Assessment

  15. Item Bias and Sensitivity Review • Ensure that the items avoid stereotype patterns • Ensure that the items avoid language, symbols, words, phrases, or examples that are biased or potentially offensive, inappropriate, or negative toward any group • Ensure that the items acknowledge the multicultural nature of our society • Ensure items do not unfairly advantage or disadvantage any group K-1 Early Literacy Assessment

  16. Different Types of Bias • Bias can refer to age, gender, race, ethnicity, cultural background, social-economic background, religion, disability, or mental condition. • Bias in test items can be present as Content bias or Language bias. K-1 Early Literacy Assessment

  17. Additional development activities: • Cognitive interviews of students • Interviews of teachers • Small-scale pilot of items K-1 Early Literacy Assessment

  18. Section 3 -Field Test and Lessons Learned for Future CELDT EditionsCaroline Fahmy,Educational Data Systems, CELDT Test Contractor K-1 Early Literacy Assessment

  19. Overview: Lessons Learned K-1 Early Literacy • Review of test and lessons learned related to: • Spring 2009 field test • 2009-10 Operational test • Modifications to be incorporated into the 2010-11 operational test K-1 Early Literacy Assessment

  20. 2009 Field Test (FT) • 142 items were delivered to the new CELDT contractor in February • Conducted FT March 16-20, 2009 • Constructed four FT forms with ten reading and ten writing items each • Administered FT in 37 districts to 2,548 K-1 students in 57 schools across California K-1 Early Literacy Assessment

  21. Number of Field Test Participants by Grade K-1 Early Literacy Assessment

  22. 2009-10 Lessons Learned: Test Materials • Format student book to eliminate distractions and enhance ease of administration • Provide all specific, test question information in the student book for ease and speed of administration K-1 Early Literacy Assessment

  23. Page Layout Changes Student Page ExaminerPage 1 2 2 1 23 K-1 Early Literacy Assessment

  24. 2009-10 Lessons Learned:Test Examiner • Include examples on how to pronounce the letter sounds next to each item script • Allow examiner to repeat any question, but no prompting allowed • Provide allowable prompts in the sample practice items in the student test book K-1 Early Literacy Assessment

  25. 2009-10 Lessons Learned:Student • Use formatting to minimize distractions • Use sample item administration as a time to really practice the items • Do not underestimate students’ English language ability K-1 Early Literacy Assessment

  26. 2010-11 Lessons Learned: Test Materials • Student Book and Examiner’s Manuals will include all four domains • Examiner’s Manual will contain all instructions, questions, and pictures • Student Book will only contain: • Bubbles for responses • Areas for written responses K-1 Early Literacy Assessment

  27. 2010-11 Lessons Learned: Test Materials (cont.) • Student Books will be greatly reduced in size • Scoring Guide will be combined with Examiner’s Manual (for all grades) K-1 Early Literacy Assessment

  28. 2010-11 Lessons Learned: Test Examiner • Move instructions from Student Book to Examiner’s Manual • Refine the “Acceptable Responses Include” section (based on actual student responses) • Review the use of pictures in the writing domain K-1 Early Literacy Assessment

  29. 2010-11 Lessons Learned: Student • Writing area - modify open box design to: • Help student know where to write • Help administrator and scorer better interpret upper case and lower case letters • Keep all writing in area clipped for image scoring • Use page formatting to accommodate left-handed students in copying words K-1 Early Literacy Assessment

  30. Section 4 -Technical Considerationsand DiscussionRichard Duran, University of California, Santa BarbaraMember, CELDT Technical Advisory Group K-1 Early Literacy Assessment

  31. K-1 Expert Panel and Technical Advisory Group • Input on development issues • Guiding principles • Alignment to ELD Standards • Input on technical issues • Pilot test • Field test • Operational test K-1 Early Literacy Assessment

  32. Primary Development Issues • Guiding Principles and ELD Standards: Items must: • embody ELD Standards • be based on early literacy theory/research • be meaningful to EL K-1 students • easily administered • scored reliably • Other Concerns K-1 Early Literacy Assessment

  33. Technical Issues • December 2008 pilot test on small K-1 sample of English learner and English-only students • p-values computed • Cognitive interviews conducted • Screened out some items and revised others • Procedures for administration reviewed • March 2009 K-1 field test of four forms - Ten reading and ten writing items per form - Reading items had already been calibrated on common reading scale - Computed basic test statistics, p-values, point biserial correlations K-1 Early Literacy Assessment

  34. Technical Issues (cont.) • July 2009 implemented operational test and a linking study for K-1 writing items with a sample of grade two English learners • January 2010 will conduct a standard setting for K-1 reading and writing performance-level cut scores using a modified Bookmark process K-1 Early Literacy Assessment

  35. Discussion • Achieved alignment of K-1 assessments with state and federal policy and professional testing standards • Contributed to state and national efforts to better ground English language development assessment and practice K-1 Early Literacy Assessment

  36. For More Information Lily Roberts, Ph.D., Administrator CELDT Office Statewide Assessment Division Telephone: 916-319-0784 E-mail: CELDT@cde.ca.gov Web: www.cde.ca.gov/ta/tg/el K-1 Early Literacy Assessment

More Related