Dramatically better assessment systems advice for rttt common assessment rfp
This presentation is the property of its rightful owner.
Sponsored Links
1 / 29

Dramatically Better Assessment Systems: Advice for RTTT “Common Assessment” RFP PowerPoint PPT Presentation


  • 85 Views
  • Uploaded on
  • Presentation posted in: General

Dramatically Better Assessment Systems: Advice for RTTT “Common Assessment” RFP. Brian Gong Center for Assessment Presentation for the Input Meetings Sponsored by the U.S. Department of Education for the “Common Assessment” RFP, “Race to the Top” funding November 17, 2009 Atlanta, GA.

Download Presentation

Dramatically Better Assessment Systems: Advice for RTTT “Common Assessment” RFP

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Dramatically better assessment systems advice for rttt common assessment rfp

Dramatically Better Assessment Systems: Advice for RTTT “Common Assessment” RFP

Brian Gong

Center for Assessment

Presentation for the Input Meetings Sponsored by the U.S. Department of Education for the “Common Assessment” RFP, “Race to the Top” funding

November 17, 2009 Atlanta, GA


My main point

My Main Point

  • The future of assessment in the United States will be shaped by what gets funded in this “Common Assessment” RFP.

  • USED should shape the RFP and fund it with a longer-term view of having in place dramatically better assessment systems in ten years.

    • When USED has to compromise, choose longer-term investments over short-term gains

    • Say very clearly what you want in the RFP

    • Help foster good responses to the RFP

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Personal recommendations

Personal recommendations

  • Hedge bets by funding multiple ways to do multi-state common assessment, especially high school

  • Invest in six “game changers” that could make assessment dramatically better within a decade, but should not be framed as being operationally implemented on the short time schedule (“2012”)

  • Help foster good responses to the RFP and after

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Short term and longer term investments

Short-term and Longer-term Investments

Common Assessment RFP should fund

  • For implementation by 2012, what we already know how to do in large-scale assessment but

    • With new set of content standards

    • With groups of multiple states (difficult to do)

  • For development through 2015, what we do not know how to do well at scale, but which has potential to lead to dramatically better assessment systems

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Implementing a new multi state summative assessment takes years

Implementing a new multi-state summative assessment takes years

2009 2010 2011 2012 2013 2014 2015

50 state systems, NAEP, TIMMS, PISA, PERLS, many LEA systems, NRTs, ACT/SAT, college’s tests, etc.

Fast Implementation of RFP: 2012 (e.g., multi-state assessments with common content standards, “Peer Review” quality of things we know how to do)

Test Specifi-cations; Develop Items; Use specs, reports, equating design,administration agreements, etc. (2009-10)

Fourth operational administra-tion; first graduating high school class, etc. (2014-15)

Pilot Test Items, promulgate high stakes policies, etc. (2010-11)

First operational administra-tion & reporting, etc. (2011-12)

Second operational administra-tion; first report using growth, etc. (2012-13)

Award RFP(s) (9/2009)

And aligning curriculum, instruction, accountability, and supports takes longer.

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Rfp specify specify specify

RFP: Specify, Specify, Specify

  • USED should specify its purpose, theory of action, and how the assessment results will be used so responders know the big picture

  • Specify what is wanted as an deliverable and the set parameters for responders’ creative proposals (e.g., time schedule)

  • Specify the means an outcome should be done if USED really wants a specific means

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Some model systems for 2012

Some Model Systems for 2012

  • Cross-state comparisons

  • Standards-based interpretation

  • Inform better instruction

  • Rapid turn-around

  • Measure growth

  • Measure student performance for teacher/administrator evaluation

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Cross state comparisons 2012

Cross-state Comparisons (2012)

  • Purpose, TOA, Use: Hold students, schools, LEAs, and states accountable to a common performance standard by triggering sanctions

  • Outcome: Statistically robust reports of performance on common metric with no “wiggle room” – stronger than current NAEP mapping studies

  • Means: Same content standards, same test specifications, same performance standards, single assessment across states, same administration procedures, strong equating across years

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Standards based interpretation 2012

Standards-based Interpretation (2012)

  • Purpose, TOA, Use: Promote equity through holding students and schools to common opportunity-to-learn (content standards) and minimal performance standards

  • Outcome: Valid reports of performance related to the designated standards

  • Means: Aligned, grade-level only (?), matrix-sampled (?), high school (?), SWD (?), ELL (?)

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Inform better instruction 2012

Inform Better Instruction (2012)

  • Purpose, TOA, Use: Assess more complex and applied learning (monitor); model and encourage instruction (drive)

  • Outcome: Incrementally better, more valid and reliable measurement of higher-order, complex student performances (?); more widespread “good” instruction (?)

  • Means: Curriculum-embedded assessments (e.g., standardized units, portfolios, graduation projects) (?); curricula with (local) matched assessments (?)

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Rapid turn around 2012

Rapid Turn-around (2012)

  • Purpose, TOA, Use: Promote improvement through rapid feedback to inform actions

  • Outcome: Reports of performance useful to decisions and actions, in appropriate timeframe (distinguish actions that are multi-year or annual monitoring from annual rich content analysis from shorter-term uses, down to course grades and student instructional feedback)

  • Means: Trade-off speed for quality, cost: greater reliance on multiple-choice/machine-scored; trade-off centralized standardization for complex performances, local scoring; ignore administration variations (e.g., missing students)

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Measure growth 2012

Measure Growth (2012)

  • Purpose, TOA, Use: Accountability, program improvement, teacher accountability?

  • Outcome: Report of student progress over time related to what is/could be/should be: grade-level standards (?), own starting point (?), other students (?), program supports (?), “teacher’s contribution” (?); how to use in accountability (?)

  • Means: Out-of-level testing (?), adaptive testing (?), vertical [moderated] scales (?), use math to predict reading for greater reliability (?), pre- post-measures within year (?)

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Teacher administrator evaluation 2012

Teacher/administrator evaluation (2012)

  • Purpose, TOA, Use: Improve teacher quality by providing feedback (?); use in accountability or other high-stakes decisions (?)

  • Outcome: Changes in student performance associated with (attributable to ?) specific teachers, administrators, programs

  • Means: many statistical approaches (check assumptions, limitations) (?); combine with other information (?)

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Personal recommendations1

Personal recommendations

  • Hedge bets on multiple ways to do multi-state common assessment, especially high school

  • Invest in six “game changers” that could make assessment dramatically better within a decade, but should not be framed as being operationally implemented on the short time schedule (“2012”)

  • Help foster good responses to the RFP and after

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Invest in game changers 1

Invest in “Game Changers” - 1

  • Develop technology that provides evidence of more complex knowledge and skills (i.e., more valid)

    • E.g., interactive simulations, non-academic knowledge and skills

  • Only use technology with an evidence-centered design approach to maintain construct relevance, most students

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Invest in game changers 2

Invest in “Game Changers” - 2

  • Develop technology for validity

  • Develop complex performance assessment

    • Specify extended learning and content, real application contexts, student choice

    • Develop credible (local) administration and scoring

    • Include all students (and teachers)

    • Develop means of certifying validity and reliability, and of combining with other evidence

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Invest in game changers 3

Invest in “Game Changers” - 3

  • Develop technology for validity

  • Develop complex performance assessment

  • Develop curricula that specify “what” and “how” of learning, and associated local assessment systems

    • Interim and formative assessments are needed to inform learning directly

    • Real assessment problem is informing “What should be done next?” – cannot solve without curriculum and teacher/administrator expertise

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Invest in game changers 4

Invest in “Game Changers” - 4

  • Develop technology for validity

  • Develop complex performance assessment

  • Develop curricula, local assessment systems

  • Develop new measurement models and technical criteria for assessments of complex knowledge and skills

    • We know current models’ assumptions and limitations; do not impose on innovations! (Example: reliability vs. validity of complex performances; cognitive vs. unidimensional models)

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Invest in game changers 5

Invest in “Game Changers” - 5

  • Develop technology for validity

  • Develop complex performance assessment

  • Develop curricula, comprehensive assessment systems

  • Develop new measurement models and technical criteria

  • Develop better accountability models and support better use of assessment results for program improvement

    • Assessments, assessment use, and instruction are being distorted by our current accountability model

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Invest in game changers 6

Invest in “Game Changers” - 6

  • Develop technology for validity

  • Develop complex performance assessment

  • Develop curricula, local assessment systems

  • Develop new measurement models and technical criteria

  • Develop better models of accountability and program improvement

  • Develop model specifications for a coherent comprehensive assessment system that incorporates above five

    • e.g., NAEP, state,

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Invest in game changers 7

Invest in “Game Changers” - 7

  • Technology for validity

  • Complex performance assessment

  • Curricula & comprehensive assessment systems

  • New measurement models and technical criteria

  • Better accountability models and support for program improvement

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Personal recommendation 2

Personal Recommendation - 2

  • Invest in five assessment “game changers”

  • Hedge bets on multiple ways to do multi-state “2012” common assessment, especially high school

    • Good current models: all MC, mixed MC-CR, computer-based, end-of-course, survey, etc.

    • Interwoven with state policies (e.g., high school exit requirements)

  • Help foster good responses to the RFP and after

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Hedge bets on 2012 assessment

Hedge bets on 2012 assessment

  • End of course AND Grade 11 survey

  • Computer-based AND paper & pencil

  • All multiple choice AND modest short CR AND larger amount and more extensive CR

  • Fund multiple “common content standards”

    • To find out costs and benefits of multi-state common assessments

    • Because no one set of content standards is clearly superior

    • Because no one approach is clearly superior

    • Because reporting on a common score metric is less important

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Rfp portfolio of awards

RFP Portfolio of Awards

  • Multiple (around 8) strong models that represent advances that can be implemented strongly by 2012 and that help get to the longer-term goal

    • Consider strategy: Do not fund strong models that will be adopted even if not funded

  • Multiple (perhaps 12) strong “game changer” awards

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Personal recommendation 3

Personal Recommendation - 3

  • Invest in four assessment “game changers”

  • Hedge bets on multiple ways to do multi-state common assessment, especially high school

  • Help foster good responses to the RFP and after

    • If USED wants certain outcomes of states working together, then promote leadership to make that happen among states, NGOs, test vendors, etc.

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Fostering strong rfp responses

Fostering Strong RFP Responses

  • Provide clear RFP specs and different awards for “2012 implementation” and “game changers”

  • If USED wants states to have vendor partners in their RFP responses, need to indicate that early and facilitate it well (vs. states’ issuing an RFP)

  • USED should think about what states who don’t get RTTT common assessment funds will do

  • USED should think how what it funds will be adopted after RTTT and how that will shape what is available in the future

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Envision intended unintended consequences

Envision Intended & Unintended Consequences

  • What if in 2012 there were five widely used assessments, all aligned to the same common content standards

    • Four were commercially available from current test publishers (like the Achieve/Pearson Algebra 2 end-of-course exam)

    • One was available by joining a consortium (like the WIDA ELP exams)

    • States were purchasing elementary math from one vendor and high school English from another vendor

  • What if there were only one assessment being used? What if there were 46?

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


Envision intended unintended consequences 2

Envision Intended & Unintended Consequences – 2

  • What if in 2012 each commercially available assessment came in five versions:

    • An all multiple-choice, computer-administered short form that took 20 minutes and cost $3/per student

    • An all multiple-choice, computer or paper & pencil form that took 50 minutes and cost $7/per student

    • A computer or p & p version that took 120 minutes, had 40 multiple choice, 8 short constructed response, and 4 extended constructed response items and cost $15/per student

    • A computer of p & p version that took 150 minutes, had 40 multiple choice, 4 extended constructed response, and 2 long constructed response items and cost $60/per student

    • A version that included a standardized test like option 3 and had a curriculum-embedded project and other performance evidence that was centrally audited and cost $200/per student

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


For more information

For more information:

Center for Assessment

www.nciea.org

Brian Gong

[email protected]

Gong – USED Common Assessment RFP Input Mtg – 11/17/09


  • Login