standard setting mcas high school science technology engineering ste tests l.
Skip this Video
Loading SlideShow in 5 Seconds..
Standard Setting: MCAS High School Science & Technology/Engineering (STE) Tests PowerPoint Presentation
Download Presentation
Standard Setting: MCAS High School Science & Technology/Engineering (STE) Tests

Loading in 2 Seconds...

play fullscreen
1 / 43

Standard Setting: MCAS High School Science & Technology/Engineering (STE) Tests - PowerPoint PPT Presentation

  • Uploaded on

Standard Setting: MCAS High School Science & Technology/Engineering (STE) Tests. Sheraton Four Points Hotel Norwood, MA August 14-16, 2007. Tuesday, August 14 Overview of Plenary Session. Welcome/Introductions Overview of MCAS Program and the High School STE Tests

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Standard Setting: MCAS High School Science & Technology/Engineering (STE) Tests' - aysha

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
standard setting mcas high school science technology engineering ste tests

Standard Setting: MCAS High School Science & Technology/Engineering (STE) Tests

Sheraton Four Points Hotel

Norwood, MA

August 14-16, 2007

tuesday august 14 overview of plenary session
Tuesday, August 14Overview of Plenary Session
  • Welcome/Introductions
  • Overview of MCAS Program and the High School STE Tests
  • Purpose of 2007 Standard Setting
  • Body of WorkMethod and Procedures
  • Ground Rules for Standard Setting
  • Agenda (Tuesday-Thursday)
department of education
Department of Education
  • Sal Beatini, MCAS Science Test Developer
  • Bob Bickerton, Associate Commissioner
  • Joyce Bowen, Science Specialist
  • Katie Bowler, MCAS Science Test Development Lead
  • Per Christiansen, MCAS Science Test Developer
  • Kevin Dwyer, MCAS History Test Development Lead
  • Jake Foster, State Science Coordinator
  • Mark Johnson, Director of MCAS Test Development
  • Bob Lee, MCAS Chief Analyst
  • Matt O’Connor, Administrator for Administration, Analysis and Reporting
  • Kit Viator, Director of Student Assessment
measured progress
Measured Progress
  • Ann Adjutant, Physics Developer
  • Liz Burton, Psychometrician
  • Lee Butler, Administrative Assistant
  • Lisa Ehrlich, Assistant Vice President
  • Kevin Haley, Manager of Data Analysis
  • Renee Jordan, Service Center Representative
  • Mark Peters, Program Assistant
  • Miechelle Poulin, Program Assistant
  • Michael J. Richards, Program Manager
  • Kevin Sweeney, Assistant Vice President, Research & Analysis
  • David Tong, Assistant Director, MCAS Program Management
  • Karen Whisler, Lead Science Developer
  • Eric Wigode, Director of MCAS Test Development
biology panelists
Biology Panelists

Erin Bengiovanni Biology Teacher Joseph P. Keefe Technical

Carol Bernon Science Department Head Barnstable High School

Sheila Blair Science Dept. Head/Biology Teacher Hull High School

Ronit Carter Educational Consultant In Harmony with Education

Andrew Creamer Biology Teacher Classical High School

Amy Deacon Biology Teacher Pentucket Regional High

Jennifer Dube Science Instructor Greater Lawrence Technical High

Eileen Dufour Biology II Instructor Greater Lowell Technical High

Mary Dulko Biology Teacher Sharon High School

Richard Fardy Biology Teacher Wilmington High School

Martha Hogan Academic Chair, Science Salem High School

Elizabeth Hufnagel Science Teacher Brookline High School

Julie Jonelis Biology Teacher Sabis International Charter

Sarah Juhlin Science Instructor Blackstone Valley Regional

Kelly Kallin Biology Teacher Sharon High School

Aaron Mathieu High School Biology Teacher Acton-Boxborough Regional

Jim Perry Science Chair Mansfield High School

Beth Raccula Science Teacher Walpole High School

Dianne Rees Director of Science Braintree High School

Julie Robertson Upper School Biology Teacher Foxborough Regional Charter

Carla Romney Faculty Member Boston University

Deborah Sakelakos MCAS Program Administrator Bedford High School

Ernestine Struzziero Science Teacher Lynnfield High School

chemistry panelists
Chemistry Panelists

Steven Abdow Chemistry Instructor Durfee High School

Thomas Bartosek Biology/Chemistry Teacher Canton High School

Catherine Botsford Milne Chemistry Teacher Attleboro High School

Patricia Brandl Science Teacher Medford High School

Barbara Brigante Teacher Northeast Metro. Regional

Stacia Brown Science Curriculum Coord. Oakmont Regional HS

Lisa Buron Science Teacher Mansfield High School

Tony DiLuna Science Teacher Woburn Memorial High School

Mary Duane Science Teacher Tantasqua Regional High School

Anthony Fimognari Chemistry Teacher East Longmeadow High School

Catherine Francis Chemistry Teacher North Reading High School

Boriana Georgieva Chemistry Teacher Swampscott High School

Joyce Gleason Educational Consultant

Esther Hines Chemistry Teacher Billerica Memorial High School

Glenn Horner Executive/Scientist DCE Aprilis

Shawn Kenner Chemistry Teacher Sharon High School

Ronee Krashes Teacher Framingham High School

Peter Nassiff Head of Science Department Burlington High School

Diane Perito Chemistry Teacher Malden High School

Nitzan Resnick Director South Area Solomon Schecter Day School

Michael Thompson Chemistry Teacher Amherst Regional High School

Linda Weber Chemistry Teacher Natick High School

Tara Yohan Biology/Chemistry Teacher Amherst Regional High School

introductory physics panelists
Introductory Physics Panelists

Anne Marie Brooks Science Teacher Wareham High School

Darren Burris Math Instructor Boston Collegiate Charter

Norma Chico High School Physics & Chemistry Doherty Memorial

Steven Cogger BMW Technician BMW Peabody

Marilyn Decker Senior Program Director, Science Boston Public Schools

Michael Doherty Teacher Andover High School

Brian Dukeman Science Teacher Whitman Hanson Regional

Erin Marie Flanagan Teacher Wachusett Regional High School

Cynthia Givens Teacher Andover High School

Naila Jirmanus Science Program Coordinator Sabis International Charter

Christine John Physics Teacher Belmont High School

Mon Luke Teacher Bedford High School

Kristin Newton Physics Teacher Cambridge Rindge & Latin

Julie Shea High School Science Teacher Revere High School

Mary Ellen Stephen Physical Science & Chemistry Teacher Plymouth South High School

John Sullivan Retired Physics Teacher Quincy Public Schools

Sue Vincent Teacher Turners Fall High School

Amy Winston Science Department Chair Newton North High School

technology engineering panelists
Technology/Engineering Panelists

Joseph Clement Department Head Beverly High School

Kevin Connolly Technology Teacher Bedford High School

Leigh Estabrooks Inventeams Associate MIT

Thomas Fischer Technology Eng. Teacher Hopedale Jr. Sr. High School

Pamela Glass Math/Science Curriculum Coordinator Westport Community School

Ramiro Gonzalez Engineering & Technology Teacher Boston Arts Academy

William Hankey Engineering Teacher Doherty Memorial High School

Rebecca Lothrop Teacher Clearway School

Charles Pouliot Teacher Lawrence Public Schools

Lee Pulis Curriculum Developer Museum of Science

Bruce Rawley Technology/Engineering Teacher Millbury High School

Anthony Ruscito Technology Teacher Bedford Public Schools

Lynn Russell Biology Teacher New Leadership Charter

Mike Stevens Technology/Engineering Teacher Maynard Public Schools

Victor Valenti Retired Engineer Compaq

Catherine Wolf High School Science Teacher Brookline High School

standard setting facilitators
Standard Setting Facilitators
  • Biology – Lisa Ehrlich
  • Chemistry – Margaret Hill
  • Introductory Physics – Dona Carling
  • Technology/Engineering – Tim Crockett
purpose of mcas program
Purpose of MCAS Program
  • Inform/improve curriculum and instruction
  • Evaluate student, school, and district performance according to Curriculum Framework content standards and MCAS performance standards
  • Determine eligibility for high school diploma (Competency Determination)
massachusetts high school competency determination cd requirements
Massachusetts High School Competency Determination (CD)Requirements
  • Meet local graduation requirements
  • Attain passing standard (scaled score of 220) or higher on high school MCAS tests or equivalent on MCAS Alternate Assessment

To earn diploma, a student must:

selected features of mcas
Selected Features of MCAS
  • Custom developed based on Massachusetts Curriculum Framework content standards and MCAS performance standards
  • 100% of questions used to determine student scores released annually
  • Measures performance of ALL students educated with public funds
  • Results reported according to scaled scores and performance levels
historical background of the mcas tests

Grade 3 Reading, grade 6 Math, and grade 7 ELA tests introduced

Massachusetts Education Reform Law passed

Newly required NCLB tests (grades 3-8) introduced







First MCAS operational tests introduced (ELA, Math, and Science & Technology, grades 4, 8, and 10)

Class of 2003 first graduating class required to earn a CD (ELA and Math)

STE tests in grades 5 and 8 introduced

Class of 2010 first graduating class required to earn a CD in STE

Historical Background of the MCAS Tests

Development of Massachusetts STE Curriculum Framework (CF) Standards and Assessments

1996 First STE Curriculum Framework introduced1998-2000 Integrated HS STE test administered2001-2002 Revised STE Curriculum Framework (major revisions); no STE testing2003 Question tryout for tests in Biology, Chemistry, Introductory Physics, and Technology/Engineering; no results reported2004-2005 Pilot testing of STE tests; item-level results only BOE decides class of 2010 to pass one of STE tests to earn CD2006 High School STE CF revised (minor revisions)2007 Operational test 9th graders may earn CD by passing one of four STE tests (ELA and Math requirements also apply)

mcas ste testing program
MCAS STE Testing Program
  • End-of-course tests offered in grades 9 and 10:
    • Biology
    • Chemistry
    • Introductory Physics
    • Technology/Engineering
  • Students required to pass one of four tests in order to earn CD
high school ste mcas test design
High School STE MCAS Test Design

Each of the four tests includes:

60 total raw score points


MCAS Scaled Scores/Performance Levels






rationale end of course ste test menu
Rationale: End-of-Course STE Test Menu
  • Promote deeper assessment of content than integrated test
  • Support diverse course offerings and scheduling flexibility for high schools
  • Provide students choice of assessment for CD requirement
efforts to establish equivalence of ste tests
Efforts to Establish “Equivalence” of STE Tests

The STE tests:

  • are based on parallel test design and development processes
  • are based on performance level descriptors that are comparable in scope and rigor
  • have been designed to have similar psychometric properties
external validation of efforts to establish equivalence of ste tests
External Validation of Efforts to Establish “Equivalence” of STE Tests
  • Performance level descriptors used in standard setting have been externally validated by Massachusetts teachers
  • Technical/psychometric properties analyzed by Dr. Ronald Hambleton, University of Massachusetts, Amherst
  • Test design and plans for standard setting endorsed by the MCAS National Technical Advisory Committee (TAC)
content standards vs performance standards
Content Standards vs. Performance Standards
  • Content standards = “What”

Describe the knowledge and skills students should acquire in a particular content and grade

  • Performance Standards = “How well”

Describe student work on MCAS tests at the Needs Improvement, Proficient, and Advanced levels

general mcas performance level descriptors
General MCAS Performance Level Descriptors

Needs Improvement

Students at this level demonstrate partial understanding of subject matter and solve simple problems.


Students at this level demonstrate a solid understanding of challenging subject matter and solve a wide variety of problems.


Students at this level demonstrate a comprehensiveandin-depth understanding of rigorous subject matter, and provide sophisticated solutions to complex problems.


High School Science & Technology/ Engineering: To establish cut scores for MCAS performance levels

Cut score needed

Cut score needed

Cut score needed


Needs Improvement



linking performance standards with student work
Linking Performance Standards with Student Work
  • What is standard setting?

Establishment of cut scores to distinguish between performance levels

  • What is your job?

Use the PLDs to evaluate student work and make recommendations for where cut scores should be set

purpose of standard setting
Purpose of Standard Setting
  • Determine cut scores for reporting assessment results
  • Answer the question:
    • How much is enough?
general phases of standard setting
General Phases of Standard Setting
  • Data-collection phase
  • Policy-making/decision-making phase
selected standard setting methods
Selected Standard-Setting Methods
  • Angoff
  • Bookmark
  • Body of Work
choosing a standard setting method
Choosing a Standard-Setting Method
  • Prior usage/history
  • Recommendation/requirement

by policy-making authority

  • Type of assessment

Body of Work method chosen for MCAS tests in

High School Science

& Technology/Engineering

what is the body of work procedure
What is the Body of Work Procedure?

Panelists examine student work (actual responses to test questions) and make a judgment regarding the performance level to which the student work most closely corresponds.

HS STE Standard Setting:

Panelists examine student work

that has not been previously classified

and determine how that work

should be classified.

how to classify student work
How to Classify Student Work

Materials you will need:

  • Performance Level Descriptors
    • General
    • Content specific
  • Bodies of Student Work
    • Responses to constructed-response questions
    • Multiple-choice summary sheet
  • Rating Forms
how to classify student work33
How to Classify Student Work
  • Examine the student’s responses to multiple-choice questions
  • Examine the student’s responses to open-response questions
  • Judge the student’s knowledge and skills demonstrated relative to the PLDs
  • Panelists do not need to reach consensus on the classifications
how to classify student work34
How to Classify Student Work

To help prepare you to do these ratings, you will

spend time becoming familiar with the following:

  • Test
  • Performance level descriptors
    • It is important that all panelists have a clear, common understanding of the PLDs.
  • Bodies of student work
    • Multiple-choice items
    • Constructed-response items
how to classify student work35
How to Classify Student Work
  • You will have the opportunity to discuss your classifications and change them if desired.
  • Don’t worry! We have procedures, materials, and staff to assist you in this process.
what next
What Next?

Break into content-area groups:

  • Take the assessment
  • Discuss the Performance Level Descriptors
  • Complete the Item Map
  • Complete training round
  • Complete individual ratings
  • Receive feedback from first round of ratings
  • Discuss ratings and revise
  • Receive feedback from revised ratings
  • Discuss feedback and provide final ratings
  • Complete an evaluation form
top 8 most misunderstood things about standard setting
Top 8 Most Misunderstood Things about Standard Setting

5. We should use this time to rework HS Science and Tech/Eng. performance level descriptors.

8. Standard setting is a great opportunity to rewrite Curriculum Framework standards.

7. The process is rigged.

6. This is a good time to vent about all the things you hate about MCAS.

top 8 most misunderstood things about standard setting38
Top 8 Most Misunderstood Things about Standard Setting

1. Disagreement is bad.

4. Standard setting is scoring.

3. Only HS Science and Tech/Eng. scholars should be doing this work.

2. Only teachers should be doing this work.

ground rules
Ground Rules
  • Role of facilitator is to “facilitate” and keep process on track
  • Process solely focused on recommending performance standards (cut scores) for MCAS
  • MCAS performance level descriptorsare integral to process but are not up for debate
  • Panelists’ recommendations are vital; however, final cut scores determined by the MDOE
  • Each panelist must be in attendance for the duration of the process for his/her judgments to be considered
  • Each panelist must complete evaluation form at the end of the event
  • Cell phones off, please!

Tuesday, August 14

Plenary 9:00 am – 10:30 am

Break 10:30 am – 10:45 am

Work session 10:45 am –12:00 pm

Lunch 12:00 pm – 1:00 pm

Work session 1:00 pm – 4:00 pm

Wednesday, August 15

Breakfast 7:30 am – 8:30 am

Work session 8:30 am – 12:00 pm

Lunch 12:00 pm – 12:45 pm

Work session 12:45 pm – 4:00 pm


Thursday, August 16

Breakfast 7:30 am – 8:30 am

Work session 8:30 am – 12:00 pm

Lunch 12:00 pm – 12:45 pm

Work session 12:45 pm – Until completion

room assignments
Room Assignments

Physics: 101

Chemistry: 102

Biology: 103

Tech/Eng: 104