assessment workshop creating and evaluating high quality assessments n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Assessment Workshop Creating and Evaluating High Quality Assessments PowerPoint Presentation
Download Presentation
Assessment Workshop Creating and Evaluating High Quality Assessments

Loading in 2 Seconds...

play fullscreen
1 / 75

Assessment Workshop Creating and Evaluating High Quality Assessments - PowerPoint PPT Presentation


  • 108 Views
  • Uploaded on

Assessment Workshop Creating and Evaluating High Quality Assessments. Dr. Deborah Brady. Agenda. Introductions: Overview Break at about 10, lunch at about 12, session ends at about 3:00 Morning presentation (with frequent processing breaks) and afternoon time for beginning to plan

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Assessment Workshop Creating and Evaluating High Quality Assessments' - giza


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
agenda
Agenda
  • Introductions: Overview
      • Break at about 10, lunch at about 12, session ends at about 3:00
      • Morning presentation (with frequent processing breaks) and afternoon time for beginning to plan
  • High quality Assessments (DESE criteria)
    • Tools to evaluate assessments
    • Tools to track all educators’ DDMs
      • Quality Tracking Tool
      • Educator Alignment Tool
  • Measuring Student Growth
      • Direct measures
        • Local alternatives to determine growth
        • Pre-/Post, Holistic Rubrics, Measures over time, Post-test only
        • “Standardization” an alternative, but not required
      • Indirect measures
  • Piloting, preparing for full implementation in SY 2015
  • EXIT SLIPS—questions, priorities, planning for full implementation (to shape the second workshop)
  • My email dbrady3702@msn.com;
slide4
Potential as Transformative ProcessWhen C, I or A is changed….Elmore, Instructional Rounds, and the “task predicts performance”
goals for today
Goals for Today
  • Answer your questions and provide tools/materials to support you
    • ( Use note cards/exit slips)
  • By the end of this session
    • You will understand what needs to be done and explain it to your colleagues
    • You will have tools to begin to do that work in your district
    • You may still have unique questions— Please ask them!
      • dbrady3702@msn.com
the dese requirements
The DESE Requirements

Purpose, timeline, requirements, direct and indirect assessments

district determined measures
District Determined Measures

DEFINITION

TYPES OF MEASURES

Portfolio assessments

Approved commercial assessments

District developed pre and post unit and course assessments

Capstone projects

DDMs are defined as:

“Measures of student learning, growth, and achievement related to the Curriculum Frameworks, that are comparable across grade or subject level district-wide”

the role of ddms dese
The Role of DDMs (DESE)

To provide educators with an opportunity to:

Understand student knowledge and learning patterns more clearly

Broaden the range of what knowledge and skills are assessed and how learning is assessed

Improve educator practice and student learning

Provide educators with feedback about their performance with respect to professional practice and student achievement

Provide evidence of an educators impact on student learning

Bottom Line: This process can be useful

district determined measures regulations
District Determined Measures Regulations

Every educator will need data from at least 2 different measures

Trends must be measured over a course of at least 2 years

One measure must be taken from State-wide testing data such as MCAS if available (grades 4-8 ELA and Math SGP for classroom educators)

One measure must be taken from at least one District Determined Measure which can include Galileo, normed assessments (DRA, MAP, SAT)

timeline for piloting and full implementation
Timeline for Piloting and Full Implementation

2013-2014

District-wide training, development of assessments and piloting

June 2014: Report: All educators in the district have 2 DDMs to be implemented fully in SY2015.

2014-2015

All DDMs are implemented; scores are divided into H-M-and Low and stored locally

2015-2016

Second year data is collected and all educators receive an impact rating that is sent to DESE based on 2 years of data for two DDMs

performance impact ratings
Impact RatingPerformance & Impact Ratings

Performance Rating

Ratings are obtained through data collected from observations, walk-throughs and artifacts

Exemplary

Proficient

Needs Improvement

Unsatisfactory

Ratings are based on trends and patterns in student learning, growth and achievement over a period of at least 2 years Data gathered from DDM’s and State-wide testing

High

Moderate

Low

slide12

Student Impact Rating

Determines Plan Duration for PST

(not future employment)

Impact Rating

on

Student

Performance

Massachusetts Department of Elementary and Secondary Education

what kinds of assessments will work for administrators guidance nurses school psychologists
What kinds of assessments will work for administrators, guidance, nurses, school psychologists?
  • Use School-wide Growth Measures
  • Use MCAS growth measures and extend them to all educators in a school
  • Use “indirect measures” such as dropout rates, attendance, etc., as measures
  • Use Student Learning Objectives (SLOs)
  • Or create measures.
  • A pre- and post- test are generally required to measure growth except with normed assessments
slide14

Indirect Measures

  • Indirect measures of student learning, growth, or achievement provide information about students from means other than student work.
  • These measures may include student record information (e.g., grades, attendance or tardiness records, or other data related to student growth or achievement such as high school graduation or college enrollment rates).
  • To be considered for use as DDMs, a link (relationship) between indirect measures and student growth or achievement must be established.
  • For some educators such as district administrators and guidance counselors, it may be appropriate to use one indirect measure of student learning along with other direct measures;
  • ESE recommends that at least one of the measures used to determine each educator’s student impact rating be a direct measure.
indirect measure examples
Indirect Measure Examples
  • Consider SST Process for a team:
    • High school SST team example
    • Child Study Team example
    • RTI team example
    • High school guidance example
    • Subgroups of students can be studied (School Psychologist group example)
    • Social-emotional growth is appropriate (Autistic/Behavioral Program example)
      • Number of times each student says hello to a non-classroom adult on his or her way to gym or class
      • Number of days (or classes) a student with school anxiety participates
      • Assess level of participation in a class
      • Increase the “in-depth” studies of at risk students
      • Make sure students go through the referral process to decrease the number of students who are unnecessarily assessed
      • Improve applications to college
    • IEP goals can be used as long as they are measuring growth (academic or social-emotional)
table talk 5 minutes
Table Talk (5 minutes)

Using the 6-phase overview, what are your priorities?

assessment quality requirements and definitions from dese
Assessment Quality Requirementsand Definitions from DESE

Alignment, Rigor, Comparability, “Substantial,” Modifications

what are the requirements
What are the requirements?
  • 1. Is the measure aligned to content?
    • Does it assess what is most important for students to learn and be able to do?
    • Does it assess what the educators intend to teach?
    • Bottom Line: “substantial” content of course
      • At least 2 standards
      • ELA: reading/writing
      • Math: Unit exam
      • Not necessarily a “final” exam (unless it’s a high quality exam)
slide20

2. Is the measure informative?

    • Do the results of the measure inform educators about curriculum, instruction, and practice?
    • Does it provide valuable information to educators about their students?
    • Does it provide valuable information to schools and districts about their educators?

Bottom Line: Time to analyze is essential

five considerations dese
Five Considerations (DESE)
  • Measure growth
  • Employ a common administration procedure 
  • Use a common scoring process
  • Translate these assessments to an Impact Rating
  • Assure comparability of assessments (rigor, validity).
c omparability
Comparability
  • Comparable within a grade, subject, or course across schools within a district
    • Identical measures are recommended across a grade, department, course
  • Comparable across grade or subject level district-wide
    • Impact Ratings should have a consistent meaning across educators; therefore, DDMs should not have significantly different levels of rigor
two considerations for local ddms
Two Considerations for Local DDMs,

1. Comparable across schools

  • Example: Teachers with the same job (e.g., all 5th grade teachers)
  • Where possible, measures are identical
    • Easier to compare identical measures
    • Do identical measures provide meaningful information about all students?
  • Exceptions: When might assessments not be identical?
    • Different content (different sections of Algebra I)
    • Differences in untested skills (reading and writing on math test for ELL students)
    • Other accommodations (fewer questions to students who need more time)
    • NOTE: Roster Verification and Group Size will be considerations by DESE
2 comparable across the district
2. Comparable across the District
  • Aligned to your curriculum (comparable content) K-12 in all disciplines
    • Appropriate for your students
    • Aligned to your district’s content
    • Informative, useful to teachers and administrators
  • “Substantial” Assessments (comparable rigor):
    • “Substantial” units with multiple standards and/or concepts assessed.

(DESE began talking about finals/midterms as preferable recently)

See Core Curriculum Objectives (CCOs) on DESE website if you are concerned

http://www.doe.mass.edu/edeval/ddm/example/

    • Quarterly, benchmarks, mid-terms, and common end of year exams
  • NOTE: All of this data stays in your district. Only HML goes to DESE with a MEPID for each educator.
rigor alignment
RigorAlignment

Rigorous

Aligned to District curriculum

Shifted to new expectations

Shifted from MCAS expectations

Consider PARCC

This is a district decision

Gradual increments?

Giant steps?

  • 2011 Massachusetts Frameworks
  • Common Core Shifts
    • Complex texts
    • Complex tasks
    • Writing to text
      • Shift in Persuasive Essay (Formal Argument)
      • Shift in Narrative (More substantial and linked to content)
      • Shift in Informational Text (organization substantiation)
    • Math, Science , History/SS frameworks
writing to text and parcc
Writing to Text and PARCC

The Next Step?

  • The 2011 MA Frameworks Shifts to the Common Core
    • Complex Texts
    • Complex Tasks
    • Multiple Texts
    • Increased Writing

A Giant Step?

Increase in cognitive load

    • Mass Model Units—PBL with Performance-Based Assessments (CEPAs)
    • PARCC assessments require matching multiple texts
two forms to adapt to your standards
Two Forms to Adapt to Your Standards
  • Handout—DDM Proposal form
  • Excel file (on wiki) Simple Excel List
table talk team talk 10 min
Table Talk/Team Talk (10 min)

How will you develop quality assessments?

calculating growth scores
Calculating Growth Scores

Defining growth, measuring growth, calculating growth for a classroom, for a district

slide33

Sample Student GROWTH SCORES from the MCAS

TEACHER GROWTH SCORES are developed from student growth scores

244/ 25 SGP

4503699

230/ 35 SGP

225/ 92 SGP

approaches to measuring student growth
Approaches to Measuring Student Growth
  • Pre-Test/Post Test
  • Repeated Measures
  • Holistic Evaluation
  • Post-Test Only
pre post test
Pre/Post Test
  • Description:
    • The same or similar assessments administered at the beginning and at the end of the course or year
    • Example: Grade 10 ELA writing assessment aligned to College and Career Readiness Standards at beginning and end of year with the passages changed
  • Measuring Growth:
    • Difference between pre- and post-test.
  • Considerations:
    • Do all students have an equal chance of demonstrating growth?
determining growth with pre and post assessments
Determining Growth with Pre- and Post Assessments
  • Cut scores need to be locally determined for local assessments
  • Standardized assessments use “The Body of the Work” protocol which easily translates to local assessments
  • First determine the difference between pre- and post- scores for all students in a grade or course
  • Then determine what Low Moderate and High growth is. (Local cut scores)
  • Top and bottom 10% to begin as a test case
  • Body of the Work check
  • Then all scores are reapportioned to each teacher
  • The MEDIAN score for each teacher determines that teacher’s growth score
table team talk
Table/Team Talk

Discuss the calculations, security, storage, fairness of determining local cut scores.

further measures beyond pre and post tests
Further measures beyond pre- and post- tests

Repeated measures, Holistic Rubrics, Post-Test Only

repeated measures
Repeated Measures
  • Description:
    • Multiple assessments given throughout the year.
    • Example: running records, attendance, mile run
  • Measuring Growth:
    • Graphically
    • Ranging from the sophisticated to simple
  • Considerations:
    • Less pressure on each administration.
    • Authentic Tasks
repeated measures example running record
Repeated Measures Example Running Record

# of errors

Date of Administration

holistic
Holistic
  • Description:
    • Assess growth across student work collected throughout the year.
    • Example: Tennessee Arts Growth Measure System
  • Measuring Growth:
    • Growth Rubric (see example)
  • Considerations:
    • Option for multifaceted performance assessments
    • Rating can be challenging & time consuming
holistic example
Holistic Example

Example taken from Austin, a first grader from Anser Charter School in Boise, Idaho.  Used with permission from Expeditionary Learning. Learn more about this and other examples at http://elschools.org/student-work/butterfly-drafts

post test only
Post-Test Only
  • Description:
    • A single assessment or data that is paired with other information
    • Example: AP exam
  • Measuring Growth, where possible:
    • Use a baseline
    • Assume equal beginning
  • Considerations:
    • May be only option for some indirect measures
    • What is the quality of the baseline information?
post test only a challenge to tabulate growth
Post-Test OnlyA challenge to tabulate growth
  • Portfolios
    • Measuring achievement v. growth
  • Unit Assessments
    • Looking at growth across a series
  • Capstone Projects
    • May be a very strong measure of achievement
tools to support the process
“Tools” to Support the Process
  • For determining what is important (Core Curriculum Objectives)
  • For determining adequacy for use as DDM (Quality Tool)
  • For making sure each educator has 2 DDMs (Excel Sheet)
  • For assessing rigor (Cognitive Complexity Rubric, CEPA Rubric)
slide48

ELA-Literacy — 9 English 9-12https://wested.app.box.com/s/pt3e203fcjfg9z8r02si

Assessment

Hudson High School Portfolio Assessment for English Language Arts and Social Studies

Publisher Website/Sample

Designed to be a measure of student growth over time in high school ELA and social science courses. Student selects work samples to include and uploads them to electronic site. Includes guiding questions for students and scoring criteria. Scoring rubric for portfolio that can be adapted for use in all high school ELA and social science courses. Generalized grading criteria for a portfolio. Could be aligned to a number of CCOs, depending on specification of assignments.

sample ddms local digital portfolio hudson ma
Sample DDMs—Local Digital PortfolioHudson, MA
  • Buy, Borrow, Build
  • Each sample DDM is evaluated

Hudson’s Evaluation: Designed to be a measure of student growth over time in high school ELA and social science courses. Student selects work samples to include and uploads them to electronic site. Includes guiding questions for students and scoring criteria. Scoring rubric for portfolio that can be adapted for use in all high school ELA and social science courses. Generalized grading criteria for a portfolio. Could be aligned to a number of CCOs, depending on specification of assignments.

  • Many are standardized assessments
other tools ma model curricula and rubrics cepas also delaware rubrics for specific text types
Other Tools: MA Model Curricula and Rubrics CEPAs ( Also, Delaware rubrics for specific text types)
slide53

Grade 10 Prose Constructed-Response Item

Use what you have learned from reading “Daedalus and Icarus” by Ovid and “To a Friend Whose Work Has Come to Triumph” by Anne Sexton to write an essay that provides an analysis of how Sexton transforms Daedalus and Icarus.

As a starting point, you may want to consider what is emphasized, absent, or different in the two texts, but feel free to develop your own focus for analysis.

Develop your essay by providing textual evidence from both texts. Be sure to follow the conventions of standard English.

Thus, both comprehension of the 2 texts and the author’s craft are being assessed along with the ability of the student to craft a clear argument with substantiation from two texts.

some examples of direct measures
Some Examples of Direct Measures

Frequently “untested” areas

  • PE—Fitness, Health Concepts
  • Art—Design, color,
  • Music—Self-critique of performance or critique of video of performance
  • SPED: Social-Emotional growth
  • Media/Technology—Research/Search

High School/Middle School

  • Mid-terms
  • Finals
  • Common Exams
  • Portfolios
  • Performances
  • Writing to text

(Reading challenging passage and writing an argument or informational response)

Elementary

  • DRA
  • Running Records
  • Benchmark exams
  • Common exams
table team talk1
Table/Team Talk

Unique areas and possible solutions.

Build on what you have and what you now value

protocols to use locally for inter rater reliability looking at student work
Protocols to Use Locally for Inter-Rater Reliability; Looking at Student Work
  • Developing effective rubrics for large-scale assessment
  • Developing exemplars
  • Calibrating scores
  • Looking at Student Work (LASW)
  • http://Nsfharmony.org/protocol/a_z.html
  • Sample for Developing Rubrics from an assessment
next steps
Next Steps
  • Develop pilot assessments for SY 2014
  • Assess results; use results to help plan for full implementation in 2015
  • Develop a plan for all educators to have two DDMs: MCAS growth, purchased, or local
  • Develop a district process for assessing the quality of assessments (DESE Quality Tool or attachment on last two pages)
  • Develop an internal process for cut scores and determining low, average, and high growth of students
  • Track/organize information for June report: Educators/DDMs
  • Plan for 2015 administration for all educators: Tracking, scheduling, storing year 1 scores, storing year 2 scores
exit slips
Exit Slips
  • Please give me your feedback on this session. It is so helpful.
  • What was helpful?
  • What might improve the presentation?
  • Additional questions you might have. Please include your email so that I can answer you personally
sample ddms

Sample DDMs

Good, Not-so-good, and Problematical

essay prompt from text
Essay Prompt from Text

Read a primary source about Mohammed based on Muhammad’s Wife’s memories of her husband.

Essay: Identify and describe Mohammed’s most admirable quality based on this excerpt. Select someone from your life who has this quality. Identify who they are and describe how they demonstrate this trait.

scoring guides from text
Scoring Guides from Text
  • Lou Vee Air Car built to specs (50 points)
  • Propeller Spins Freely (60 points)
  • Distance car travels
    • 1m 70
    • 2m 80
    • 3m 90
    • 4m 100
  • Best distance (10,8,5)
  • Best car(10,8,5)
  • Best all time distance all classes (+5)
  • 235 points total