RACE TO THE TOP ASSESSMENT PROGRAM PUBLIC & EXPERT INPUT MEETING

1. RACE TO THE TOP ASSESSMENT PROGRAMPUBLIC & EXPERT INPUT MEETING Jamal Abedi University of California, Davis Presented at The Race to the Top Assessment Program January 20, 2010 Washington, DC

2. Q1. How “a through-course summative assessment system” can be developed and implemented with the following characteristics? A system that includes components of assessments delivered periodically throughout the school year If we do this, how should we ask applicants to describe their approaches? What evidence should we request if such summative results are part of an accountability system?

3. A. A system that includes components of assessments delivered periodically throughout the school year Potentials: Provides “value added components” for a more thorough assessments of student performance Multiple measures from different sources with different formats (e.g., performance assessment) could improve the reliability and validity of the summative assessments Accountability system will be based on a more comprehensive picture of what students know and are able to do Provides better assessment opportunities for the special needs students specially for ELLs

4. A. A system that includes components of assessments delivered periodically throughout the school year Challenges: Extra efforts and expenses in developing and implementing the new components Extra time taken from instructions Comparability issues in content, construct, linguistics and psychometrics of the different components within and across the consortia of states Burden on teachers and schools or whoever is responsible for providing data Redundancy of the information

5. B. How should we ask applicants to describe their approaches?(Who is responsible for this? The Department, the applicants? Or in a collaborative effort?) Clearly identify the “through-course” components and their relevance to the assessment and accountability system (new assessments? Interim assessment outcomes? Performance assessments?) Clearly document the value added by the “through-course” components to the assessment and accountability system Design and field test methodologies for creating and interpreting the composite system Identify burden on teachers, schools, districts, consortia of states or whoever responsible for collecting the components and incorporating them into the assessment system Present evidence on problem of redundancy of added criteria

6. C. What evidence should we request if such summative results are part of an accountability system? Validity – including construct, content, consequential, and predictive validity External validity for postsecondary preparedness Reliability – including inter-rater reliability if human scored Fairness Precision across the full performance continuum (e.g. from low to high performers) Comparability across years

7. a. What evidence should we request: Validity, including construct, content, consequential, and predictive validity Construct: Different components measure the same construct; construct-irrelevant sources are controlled; the components would provide added values Content: Evidence of alignment of the components (“end-of-course” and the “through-course”) with the common set of K-12 internationally benchmarked, college and career ready standards Consequential: Evidence in achieving the goal of measuring a common set of standards, examine both intended (determination of student college and career readiness, high school exit decision, college course placement) and unintended consequences (focusing on a limited outcomes, the possibility of teaching to the test) Predictive: High power in predicting college and career readiness

8. b. What evidence should we request: External validity for postsecondary preparednessMeasures or indices of: College and career readiness High school completion College performance Maintaining high performance at college Student attributes for college/career success

9. c. What evidence should we request: Reliability – including inter-rater reliability if human scored Identify sources of systematic error of measurement (bias) and suggest ways to reduce the impact of such sources Test reliability: internal consistency (examining dimensionality), in addition, estimate reliability through either a parallel form or a test-retest approach Evidence of reliability for subgroups, at the minimum, separate reporting for ELLs and students with disabilities Standard error of measurement by subgroups

10. c1. What evidence should we request: Inter-rater reliability Clearly identify factors affecting validity and consistency of scoring open-ended test items such as rater’s background and experience Involve teachers in scoring of open-ended items but at the same time estimate and discuss burden on teachers in getting involved in scoring Provide detailed information on the scoring rubric including a discussion on how sensitive is the rubric to students’ backgrounds Not relying solely on percent of agreement approach, use other approaches that are more robust such as Kappa, intra-class correlation, and William’s Index Examine and report sources of measurement error to inter-rater reliability

11. d. What evidence should we request: Fairness Evidence on examining and controlling for sources of cultural and linguistic biases Evidence on the equal opportunity to learn the content being assessed specially for ELL students Evidence on the appropriateness of accommodations to level the playing field Evidence on the validity and fairness of accommodations

12. e. What evidence should we request: Precision across the full performance continuum Enough test items at different level of difficulty to assess different levels of content knowledge Evidence on the discrimination power across the full range of performance continuum Scoring rubric to be sensitive to student different cultural and linguistic backgrounds Evidence of plans and efforts to make assessments more accessible across the full performance continuum

13. Q2. Evidence on the proposed “system” that will ensure consistent and high level of rigor Evidence of validity from all different aspects including data showing criterion-related validity, e.g, convergent and discriminant validity (MTMM) Evidence of controlling for the impact of construct-irrelevant sources for all students particularly for ELLs and students with disabilities. Evidence on the relevance of assessments to all students with different backgrounds

14. Q3. Computer-based test administration Potentials Capable of incorporating accessibility features for students at risk for example, ELLs and students with disabilities Computer assessment system can include a variety of accommodations that would be difficult to implement in a paper-and-pencil assessment format Examples for ELLs include: English and native language glossaries (pop-up), adaptive based on student’s level of English proficiency, Native language or bilingual version of the assessments, extra time, and dictionaries Examples for SWDs include: Read aloud, flexible schedule, one-on-one testing, audio visual assistance As an example of computer capability for subgroups, adaptive testing for ELLs can be mentioned

15. Q3. Computer-based test administration Challenges: Comparability between the computer-based and paper-and-pencil versions Design and logistical issues A full comparability may undermine the potential capabilities of computer-based testing Issues concerning “Digital Divide” and the impact of student and parent SES on the outcome of computer testing Lack of research

16. 4. Process or approach leading to innovation and improvement over time Evidence of flexibility of the assessment system and its capability in utilizing innovations over time Evidence that incorporating innovations does not alter the construct being measured Plans for studies for carefully examining the impact of innovations on the assessment outcomes before incorporating them into the assessment system The department should have ongoing quality control system for making sure appropriate innovations have been considered

17. 5. Issues that seem to require additional, focus research Research evidence on the value-added methodology should be a major component of a good RFP and should be a part of application Evidence on the comparability and generalizability of performance tasks should be provided by the applicants who are creating such assessments and have access to information on the test content and psychometric properties Studies on how to make assessments more accessible for ELLs and students with disabilities may help consortia of states to be less dependent on accommodations that may not produce valid results Sources of measurement error/construct irrelevant differentially affecting performance of subgroups and ways to control the impact of these sources Effectiveness and validity of accommodations used for ELLs and students with disabilities

18. Other suggestions It is important to incorporate an ongoing quality control procedure (by the Department or its representative) into the process so appropriate corrections can be made Any departure (even minor) from the approved plans should be discussed with the Department and justifications for such changes should be provided The consortia should provide assurance that the decisions on the development and implementation of the assessments are made collectively by all stake holders It is important that people in charge of developing the assessments are quite familiar with research findings in the area of assessments particularly with the recent work on assessments for ELLs and students with disabilities

19. For more information please contact Jamal Abedi at CRESST/UC Davis (530) 754-9150 or jabedi@ucdavis.edu