1 / 23

MCAS Standards Validation: High School Introductory Physics

MCAS Standards Validation: High School Introductory Physics. Sheraton Hotel Braintree, MA September 17-18, 2007. Overview of August Standards Setting. Independent standard-setting for: Biology Chemistry Introductory Physics Technology/Engineering Cut scores recommended for:

Download Presentation

MCAS Standards Validation: High School Introductory Physics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. MCAS Standards Validation:High School Introductory Physics Sheraton Hotel Braintree, MA September 17-18, 2007

  2. Overview of August Standards Setting • Independent standard-setting for: Biology Chemistry Introductory Physics Technology/Engineering • Cut scores recommended for: Failing/Needs Improvement Needs Improvement/Proficient Proficient/Advanced • Each panel composed of: • high school science teachers and administrators • university representatives • community representatives

  3. Post Standard-Setting Analysis • Analyzed each of the recommended cut scores (n=12) across all four content areas • Validated 10 of 12 recommended cut scores • Issues related to 2 of the recommended cuts: • Raw score ranges associated with two cuts for F/NI and NI/P for Introductory Physics were extreme outliers • Some Introductory Physics panelists expressed concern about standard-setting process

  4. Plenary Session • Welcome • Overview of August Standard Setting Meeting • Post Standard Setting Analysis • Standard Setting versus Standards Validation • STE tests: efforts to establish “equivalence” and uses of test results • Recap of Body of Work (BOW) method and role of performance level descriptors • Questions and Answers

  5. Additional Analyses Conducted • Relationship of projected cut scores to psychometric properties of test (TCCs and TIFs) • Relationship of previous student performance in science with that of IP test based on recommended cut scores • Relationship of student work classified as Needs Improvement and Proficient to: • Introductory Physics performance level descriptors • student work classified as Needs Improvement and Proficient for Biology, Chemistry, and T/E

  6. Conclusions Post Standard-Setting • Interpretation of Needs Improvement and Proficient diverged significantly from interpretation made in the other content areas • Considered statistical adjustment as remedy but determined Standards Validation preferable given high stakes use of results • Delay in posting of scaled score/raw score conversion tables for all four content areas until standards validation completed

  7. Remember, we knew this was going to be challenging… • Standard setting for multiple tests–-where roughly “comparable” standards across all tests is the goal—is technically challenging • Other approaches to standard setting considered: • Overlapping content panels • One panel composed of all representatives for all four content areas • Independent panels

  8. General Phases of Standard Setting • Data-collection phase • Policy-making/decision-making phase

  9. Standard Setting vs.Standards Validation • Standard setting • Process of establishing original cut scores • Panelists are not provided initial cut points or focused cut point range • Standards validation • Process of validating cut scores • Panelists are provided initial cut points or a narrowed cut point range

  10. Introductory Physics Standards Validation Cut score established Cut score needed Cut score needed Warning Needs Improvement Proficient Advanced

  11. Standards Validation • Bodies of work to be classified represent a narrowed range of student work based upon: • Standard setting panelists’ recommendations (August) • Psychometric and statistical analyses • Analysis of classification of student work for each of the 12 recommended cut scores (conducted by DOE and Measured Progress content experts) • Pinpointing cut score for Failing/Needs Improvement Needs Improvement/Proficient

  12. Massachusetts High School Competency Determination (CD)Requirements: Class of 2010 • Meet local graduation requirements • Attain scaled score of 220 or higher on high school MCAS tests in ELA, mathematics, and STE (or equivalent on MCAS Alternate Assessment) To earn diploma, a student must:

  13. Efforts to Establish “Equivalence” of High School STE Tests The High School STE tests: • are based on parallel test design and development processes • are based on performance level descriptors that are comparable in scope and rigor • have been designed to have similar psychometric properties

  14. External Validation of Efforts to Establish “Equivalence” of STE Tests • Performance level descriptors used in standard setting have been externally validated by Massachusetts teachers • Technical/psychometric properties analyzed by Dr. Ronald Hambleton, University of Massachusetts, Amherst • Test design and plans for standard setting endorsed by the MCAS National Technical Advisory Committee (TAC)

  15. What is the Body of Work Procedure? Standard Setting method where panelists: • examine student work (actual responses to test questions) and performance level descriptors • make a judgment regarding the performance level to which the student work most closely corresponds.

  16. Body of Work Fundamentals • Examine the student’s responses to multiple-choice questions • Examine the student’s responses to open-response questions • Judge the student’s knowledge and skills demonstrated relative to the PLDs • Panelists do not need to reach consensus on the classifications

  17. Materials Used During Standards Validation • Performance Level Descriptors • General • Content specific • Bodies of Student Work • Responses to constructed-response questions • Multiple-choice summary sheet • Rating Forms

  18. General MCAS Performance Level Descriptors Needs Improvement Students at this level demonstrate partial understanding of subject matter and solve simple problems. Proficient Students at this level demonstrate a solid understanding of challenging subject matter and solve a wide variety of problems. Advanced Students at this level demonstrate a comprehensiveandin-depth understanding of rigorous subject matter, and provide sophisticated solutions to complex problems.

  19. GeneralSTE Performance Level Descriptors

  20. What Next? • Take the Introductory Physics test • Discuss the Performance Level Descriptors • Complete the Item Map • Complete training round • Complete an evaluation form • Complete individual ratings • Receive feedback from first round of ratings • Discuss feedback and provide final ratings • Complete final evaluation form

  21. Ground Rules • Role of facilitator is to “facilitate” and keep process on track • Process solely focused on recommending performance standards (cut scores) for MCAS • MCAS performance level descriptorsare integral to process but are not up for debate • Panelists’ recommendations are vital; however, final cut scores determined by the MDOE • Each panelist must be in attendance for the duration of the process for his/her judgments to be considered • Each panelist must complete evaluation form at the end of the event • Cell phones off, please!

  22. Agenda Monday, September 17 Breakfast 8:00 am – 9:00 am Plenary 9:00 am – 10:30 am Break 10:30 am – 10:45 am Work session 10:45 am –12:00 pm Lunch 12:00 pm – 1:00 pm Work session 1:00 pm – 5:00 pm Tuesday, September 18 Breakfast 8:00 am – 9:00 am Work session 9:00 am – 12:00 pm Lunch 12:00 pm – 12:45 pm Work session 12:45 pm – until completion

  23. Questions?

More Related