The MAAECF: Assessment Procedures, Supporting Data, & Technical Considerations

1. The MAAECF: Assessment Procedures, Supporting Data, & Technical Considerations Stephen N. Elliott Andrew T. Roach Vanderbilt University Georgia State University Presented at the 8th Annual Maryland Conference On Alternate Assessment E & R Assessments 2007

2. Key Questions to be Addressed 1. What does the Mississippi Alternate Assessment of Extended Curriculum Frameworks (MAAECF) measure? 2. How do you conduct an alternate assessment using the MAAECF Rating Scale for students with significant cognitive disabilities? 3. What evidence is there to support the claim that the MAAECF yields reliable and valid results? 4. What are salient technical issues associated with this approach to alternate assessment? E & R Assessments 2007

3. Technical Issues to Consider • Issue #1. Teachers as reliable judges of student performance. • Issue #2. Reliability of teachers’ ratings and the concept of independence. • Issue #3. Use of standards-based IEPs to drive the development and collection of evidence for a meaningful sample of items. • Issue #4. Amount and quality of classroom evidence needed for valid ratings of proficiency. E & R Assessments 2007

4. Validity Evidence Plan E & R Assessments 2007

5. MAAECF : A Comprehensive, Evidence-BasedRating Scale that Uses Teachers’ Judgments to Measure Student Achievement E & R Assessments 2007

6. Assessment of the Mississippi Extended Curriculum Frameworks (MECF) • Extended Content Standards - General statements that describe what students should understand and be able to do in reading/ language arts, mathematics, and science • Competencies and Objectives - Specific statements of expected knowledge and skills necessary to meet a content standard requirement E & R Assessments 2007

7. MECF Organizational Structure E & R Assessments 2007

8. LA Strands and Competencies E & R Assessments 2007

9. MECF Classroom Tasks/Activities to Help Teach and Generate Evidence E & R Assessments 2007

10. Sample MAAECF Items and Rating Scale Format E & R Assessments 2007

11. Essential Features of Evidence-Based Judgment Systems like the MAAECF • Unbiased items. • High-quality evidence about students’ knowledge & skills. • Clear and objective scoring criteria. • Procedures to ensure the reliability of the scores. • Meaningful methods for communicating results of the assessment. • Trained judges – all educators using the MAAECF must attend a workshop + pass a qualification test! E & R Assessments 2007

12. MAAECF Honors “Teachers as Tests” The research-based concept of teachers as tests emphasizes that teachers collect a substantial amount of data about students and when they are provided a structure for collecting evidence and a method for quantifying and reporting this information, it can be used like other highly reliable test results. Key studies on the reliability of teachers’ judgments: • Hoge & Coladarci (1989) • Demary & Elliott (1998) • Hurwitz, Elliott, & Braden (2007) E & R Assessments 2007

13. What Does the MAAECF measure? • The MAAECF (regardless of level) includes more than 100 items representing prerequisite knowledge and skills in core academic areas. Rating scale items are based on the objectives specified in the Mississippi Extended Curriculum Frameworks (MECF). Note that various levels of the MAAECF exist for students in the grade clusters of 3-5, 6-8, and 12. • Teacher rates proficiency using a 4-point scale (0 = Non-Existent to 3 = Accomplished). The same item level rubric is used across all grade clusters. • Results are reported as performance levels (Minimal, Basic, Proficient, or Advanced) in each content area. Definitions and cut scores for these levels vary by grade. The results are on a continuum that is very similar to the general education test’s performance levels. E & R Assessments 2007

14. Item Count Per Scales & Grade Levels E & R Assessments 2007

15. Mississippi Alternate Assessment and Accountability System E & R Assessments 2007

16. MAAECF : Administration & Scoring Procedures E & R Assessments 2007

17. The MAAECF Process Once the IEP Team decides that a student is eligible for an alternate assessment, a 5-step process must be followed: Step 1: Align MAAECF items with IEP goals, objectives, or other learning objectives. Step 2: Collect performance evidence for a sample of items for each strand using Evidence Worksheets. Step 3: Analyze and rate proficiency of all items. E & R Assessments 2007

18. The MAAECF Process(continued) Step 4: Summarize proficiency scores and performance level decisions. A second educator completes the process by reviewing collected evidence, providing ratings on only items with evidence samples, and checking the accuracy of the performance level decisions. Step 5: Report results after the first rater and the individual who completes the reliability check have reached agreement. E & R Assessments 2007

19. Team Work for Reliable Results:Summarizing the Roles of Raters 1 and 2 Rater 1 Step 1 Align Items & Goals Step 2 Collect Evidence for Aligned Items Step 3 Rate All Items Step 4 Summarize Ratings & Proficiency Decisions Step 5 Report Reliable Results Rater 2 Reliability Checker Review Evidence Review & Rate Items with Evidence Double Check Total Scores Make an Independent Proficiency Decision Determine Agreement, Resolve Disagreements E & R Assessments 2007

20. Step 1: Alignment of Items • Determine in which content areas a student will be assessed. • Identify MAAECF items that align with the student’s IEP goals and objectives, or other learning goals. • If there is not at least 1 IEP-aligned item for each competency (4 in LA, 5 in Math, and 4 in Science), then an item must be selected and evidence generated for that item. E & R Assessments 2007

21. Step 2: Collect Evidence of Skills Teachers must collect evidence from 2 different categories for at least 1 item for each competency. • Work Samples • Tests • Observations • Interviews • Video/Photo • Audio Tape E & R Assessments 2007

22. Characteristics of Good Evidence • Recent (collected during the current school year & dated) • Representative (several forms that are typical performances of knowledge and skills with classroom materials and instructional accommodations) • Relevant (evidence that is clearly indicative of an item that is aligned with content standards, and that identifies the relevant item number(s) for the second rater and others) • Reliable (increases the likelihood that 2 or more raters have the same understanding of the item by documenting the support provided to get the response from the student and indicating the overall accuracy of the student’s typical response) E & R Assessments 2007

23. Evidence Collection Worksheets Number of settings is important to determine generalizability of skill Dates are needed to determine if evidence is recent Information about support is important Description of accuracy of the student’s response is essential E & R Assessments 2007

24. Quality Evidence Matters: Collected Evidence Should Meet or Exceed the Standard! E & R Assessments 2007

25. Step 3: Criteria for Rating All Items 0 = NON-EXISTENT (Can’t do currently) 1 = EMERGING (Aware and starting to do) 2 = PROGRESSING(Can do partially and inconsistently) 3 = ACCOMPLISHED (Can do well and consistently) ***Examine descriptions of these Proficiency Rating Levels*** E & R Assessments 2007

26. MAAECF Item Level Proficiency Rating Scale and Descriptive Criteria E & R Assessments 2007

27. Step 4: Overall Proficiency Scores and Performance Continua Students earn an Individualized Proficiency Total Score for each content area. The total scores are used to guide the determination of which of the four Overall Performance Levels best describes the student’s achievement. Minimal Basic Proficient Advanced ***Examine detailed Performance Levels*** E & R Assessments 2007

28. LA Performance Level Descriptor(with Cut Scores for Grades 3, 4, and 5) E & R Assessments 2007

29. MAAECF Item Count & Score Ranges by Performance Level & Grade Clusters E & R Assessments 2007

30. 2nd Rater’s Responsibilities • Review evidence of student’s knowledge and skills and then independently rate items with which they are aligned. This is done on a separate copy of the MAAECF. • Review the 1st rater’s ratings of evidence-based items to determine general degree of agreement and then examine all item ratings to learn more about the student. These ratings provide additional evidence for the 2nd rater. • Based on all of the evidence, independently select the overall Performance Level (Minimal, Basic, Proficient, or Advanced) that best characterizes the student’s current functioning. • Double check 1st rater’s total score for the students in Language Arts, Math, and Science to ensure all items were rated and the total scores are correct. • Compare their overall Performance Level determination to that of the 1st rater and decide if it is reliable using the Inter-Rater Reliability Estimate table of the MAAECF for Language Arts and Math. Settle any disagreements that result in unreliable decisions. E & R Assessments 2007

31. Inter-Rater Agreement Table E & R Assessments 2007

32. E & R Assessments 2007

33. Descriptive Statistics & Key Evidence to Support the Inferences Made From MAAECF Scores About Student Achievement E & R Assessments 2007

34. Descriptive Statistics for MAAECF Math E & R Assessments 2007

35. Frequency Distributions for Math E & R Assessments 2007

36. E & R Assessments 2007

37. Inter-rater Agreement & Evidence Quality Study E & R Assessments 2007

38. Quality of Evidence Samples Matter! E & R Assessments 2007

39. Factor Analytic Evidence to Support Claims about the MAAECF’s Structure E & R Assessments 2007

40. Evidence Based on Relations to Other Variables CAAVES 1% Study E & R Assessments 2007

41. More Evidence Based on Relations to Other Variables CAAVES 1% Study E & R Assessments 2007

42. Evidence based on Consequences of Testing E & R Assessments 2007

43. Item and Scale Revision Plan in Progress for 2008 E & R Assessments 2007

44. MS CAARES Project Enhancements to the MAAECF MS CAARES is a new GSEG Project E & R Assessments 2007

45. Thank You • Contact Information • Stephen N. Elliott • Vanderbilt University • 616-322-2538 • steve.elliott@vanderbilt.edu • Andrew T. Roach • Georgia State University 404-413-8176 aroach@gsu.edu E & R Assessments 2007