So what what s next l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 52

So What? What’s Next? PowerPoint PPT Presentation


  • 104 Views
  • Uploaded on
  • Presentation posted in: General

So What? What’s Next?. Scott Marion Center for Assessment Wyoming Formative Assessment Institute February 4, 2008. Some Questions. Does formative assessment Improve teaching? Improve learning? If so, how and why? If not, why not? Does interim assessment Improve teaching?

Download Presentation

So What? What’s Next?

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


So what what s next l.jpg

So What?What’s Next?

Scott Marion

Center for Assessment

Wyoming Formative Assessment Institute

February 4, 2008


Some questions l.jpg

Some Questions

  • Does formative assessment

    • Improve teaching?

    • Improve learning?

    • If so, how and why? If not, why not?

  • Does interim assessment

    • Improve teaching?

    • Improve learning?

    • If so, how and why? If not, why not?

Scott Marion. Center for Assessment. WY FAI February 4, 2008


My focus question l.jpg

My Focus Question

  • What is the mechanism for turning an assessment result—broadly speaking—into a useful instructional action?

  • With all of the claims for the instructional benefits for a variety of assessments—value-added, interim/benchmarks, formative—I see little attention to the black box by which teachers and others are supposed turn these results into a useful teachable action

Scott Marion. Center for Assessment. WY FAI February 4, 2008


What s our plan for this morning l.jpg

What’s our plan for this morning?

  • This is all about figuring out what to do once we learn something from the assessments

    • Peaking inside the “black box”?

    • Key attributes of formative assessment with a major focus on learning progressions

    • Analyzing student work with an eye toward “what comes next”

    • Relating this interim and summative score reports

      • Data base decision making approaches

    • Analyzing score reports with an eye toward program evaluation

Scott Marion. Center for Assessment. WY FAI February 4, 2008


What s inside the black box l.jpg

What’s inside the black box?

Valid inferences are required for each transition

Data

Information

Decision(s)

Action(s)

Scott Marion. Center for Assessment. WY FAI February 4, 2008


What do i mean by valid inferences l.jpg

What do I mean by valid inferences?

  • Each transition (e.g., from data to information) is inferential

    • In other words, it is NOT a concrete certainty, which means we can be wrong!

  • We are creating hypotheses about how things are working and what makes sense

  • Stronger compared with weaker hypotheses are grounded in theory (e.g., learning theory) and/or previous research

  • Hypotheses are falsifiable and we must consider plausible alternative explanations for the phenomena we observe

    • Try to protect against a “confirmationist bias” (e.g., trying to support your favorite program)

  • All of these transitions are really hard!!

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Data to information l.jpg

Data to Information

  • On a sheet of paper, jot down your answers to the following:

    • What’s the difference between data and information?

      • What’s the same?

    • What are some criteria that we might use to distinguish between the two concepts?

    • How might we facilitate the transition from data to information?

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Data to information 2 l.jpg

Data to Information-2

  • We are bombarded with data, but we can’t and shouldn’t act on all of it

  • Moving from data to information often involves transformations and reductions

    • For example, some might argue that everything going on in your classroom is a formative assessment opportunity—and it might be—but you can’t attend to all of it

      • Using targeted probes and/or observations will allow you to focus on important areas…and you still will not use all of it

    • Similarly, you probably do not spend a lot of time looking at lists of student raw scores from the state assessment, but look instead at some type of aggregation and disaggregations such as % in category or mean scale scores

      • But, these can also mislead!! That’s why the validity checks are critical

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Information to decisions l.jpg

Information to Decisions

  • Once you have reduced the data to information, the next step is to make decisions about this information

  • What are some of the things we want to consider when making decisions about how and what we might act on regarding this information?

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Information to decisions 2 l.jpg

Information to Decisions-2

  • Some things to consider?

  • Is what the information is telling us something within our control on which to act?

  • Do we know enough about what the information means so that we can make valid decisions?

    • Are we sure about the validity of the information?

    • What is the cost if we are wrong?

      • Replacing a math program is expensive!

      • Providing students with some differential practice can be cheap

  • Have we prioritized our decisions? How?

    • Social justice/equity

    • Resource (time, money, etc) constraints

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Decisions to actions l.jpg

Decisions to Actions

  • Once we decide what needs to be done, do we know how to do it?

  • Once we choose an action, how will we know if it is the right action?

  • Start the cycle again…

Scott Marion. Center for Assessment. WY FAI February 4, 2008


A little diversion already l.jpg

A little Diversion (already?)

  • Dena M. Bravata and her colleagues just published a meta-analysis in JAMA documenting the positive correlation among pedometer usage, increases in physical activity (as much as 1 mile extra walking/day), and decreases in body mass index (BMI)

    • Setting a goal—such as 5,000 or 10,000 steps/day was also a key to increased activity

  • So, is this really a diversion?

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Key attributes of formative assessment l.jpg

Key Attributes of Formative Assessment

  • The Council of Chief State School Officers’ (CCSSO) Formative Assessment work group (SCASS) recently released a draft paper describing six essential attributes of formative assessment...

    • Learning progressions

    • Learning goals

    • Descriptive feedback

    • Embedded in instruction

    • Self and peer assessment

    • Collaboration and culture

  • Based on what I have seen, these should not be new to you

Scott Marion. Center for Assessment. WY FAI February 4, 2008


From fast scass key attributes l.jpg

From FAST SCASS Key Attributes

  • Learning Progressions: The formative assessment process should be grounded in learning progressions that provide a clear understanding of the inter-relationships between major concepts and principles in a discipline and how they develop.

  • Learning Goals: Learning goals and criteria for success should be clearly identified and communicated to students.

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Learning progressions l.jpg

Learning Progressions

  • In the following slides on learning progressions, I draw largely on the work of my colleague, Brian Gong, and to a lesser extent, Margaret Heritage from CRESST.

  • Learning progressions should serve as the basis for formative assessment, but we argue that they should serve as the basis for most other assessments, especially those intended to measure student growth

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Challenge supporting increased student learning l.jpg

Challenge: Supporting increased student learning

  • Assertion 1: Current large-scale assessment and accountability are insufficient to produce the student learning we desire.

    • We can make accountability and large-scale assessment better, but they will always be insufficient because they are too distal to teaching and learning (that’s why you’re here!).

  • Assertion 2: Learning progressions and instructional assessments can support learning and teaching more powerfully (but still are only part of the solution).

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Learning progressions standards and educational reform l.jpg

Learning Progressions, Standards, and Educational Reform

  • Learning progressions are an extension of the standards-assessment model for educational reform.

    • Clear learning targets (content and performance), all students, feedback to guide instruction

  • HOWEVER – Content standards and performance descriptors typically lack the detail, developmental coherence, and skills (reasoning, etc.) needed to present sufficient learning targets and learning progressions

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Content standards are not enough l.jpg

Content standards are not enough

  • We are starting to see limited good examples of state grade-level content standards showing some development of knowledge, skills, or complexity over time

    • NRC Science (2007)

    • New England Common Assessment Program

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Learning progressions multiple related perspectives that are not new l.jpg

Learning Progressions: Multiple, related perspectives that are not new!

  • “Descriptions of the successively more sophisticated ways of thinking about an idea that follow one another as students learn” (Wilson & Bertenthal, 2005)

  • “Descriptions of successively more sophisticated ways of reasoning within a content domain” (Smith et al., in press)

  • “A picture of the path students typically follow as they learn... A description of skills, understandings and knowledge in the sequence in which they typically develop.” (Masters & Forster, 1996)

  • The organization of learning experiences designed to “help students go ahead in different subjects as rapidly as they can”… “to learn how to learn.” “Any subject can be taught effectively in some intellectually honest form to any child at any stage of development.” (Bruner, 1960)

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Some key traditions informing learning progressions l.jpg

Some Key Traditions Informing Learning Progressions

  • Developmental psychology

  • Cognitive psychology

    • Misconception/Naïve theories

    • Expertise

  • Curriculum

    • Structure (Content & Skills)

    • Scope & Sequence

  • Task analysis

  • Instructional technology

  • Assessment

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Developmental psychology features and examples l.jpg

Developmental PsychologyFeatures and examples

  • Features:

    • Inherent

    • Universal

    • Invariant*

  • Examples:

    • Piaget: preoperational, concrete, symbolic

    • ACER: Spelling

    • Applebee: Children’s concept of story

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Curriculum scope sequence features and examples l.jpg

Curriculum: Scope & SequenceFeatures and examples

  • Features:

    • Lays out content knowledge – can be content web rather than linear

  • Examples:

    • AAAS: Atlas of Science Literacy

    • Most state content standards

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Cognitive misconception features and examples l.jpg

Cognitive: MisconceptionFeatures and examples

  • Features:

    • Incorrect schema (concept, procedure)

    • Rule-oriented (“mal-rules”) – well-structured content with a “grammar”

  • Examples:

    • Brown & Burton: “buggy subtraction”

    • Minstral: Inertial motion

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Curriculum structure features and examples l.jpg

Curriculum: StructureFeatures and examples

  • Features:

    • Focuses on “essence” and “structure”

  • Examples:

    • Bruner:

    • NSF “learning progressions” projects

    • Ginsberg: Multiplication deep structure, multiple representations

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Cognitive expertise features and examples l.jpg

Cognitive: ExpertiseFeatures and examples

  • Features:

    • Content plus cognitive skills (habits of mind, heuristic)

    • Plus structure, models, extensive schemata

  • Examples:

    • Wilson: density; Siegler: balance beam

    • Conant: Sea of Air

    • Applebee: Child’s concept of story

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Example 1 aaas atlas of science literacy is there enough detail l.jpg

Example 1: AAAS Atlas of Science Literacy (is there enough detail?)

Characteristics:

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Example 2 a counting and ordering progress map lots of detail and very sequential l.jpg

Example 2: A Counting and Ordering Progress Map (lots of detail and very sequential)

  • FromCurriculum Corporation. (1994). Mathematics Profile for Australian Schools. Carlton: Curriculum Corporation. In Masters, G., & Forster, M. (1997). Developmental Assessment.Victoria, AU: The Australian Council for Educational Research Ltd.

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Slide28 l.jpg

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Slide29 l.jpg

Example 3: Graphic Organizer for Conceptual FlowFrom DiRanna, K. & Topps, J. (2005). What’s the Big Idea? San Francisco: K-12 Alliance/ WestEd.

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Example 4 multiplication l.jpg

Example 4: Multiplication

  • Acquisition – movement from addition to multiplication

    • Multiplication: problem of finding the total quantity of objects contained in a given number of groups with the same number of elements

    • Cognitive challenges:

      • Learner has to know and operate with two different grouping systems (number of groups and number of items in a group) – not like addition or subtraction

      • Operational number systems different than place value system (e.g., 12 is one ten and two ones)

      • Generalization of learned representations (e.g., quantity per set model; area model; number line model)

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Multiplication cont l.jpg

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Multiplication – cont.

  • Multiplication Example (Ginsburg)

Scott Marion. Center for Assessment. WY FAI February 4, 2008


What is intended to develop l.jpg

What is intended to develop

  • Deep content representation

  • Other dimensions

    • Skills; Complexity

    • Independence

    • Generalization, application, reflection = “habits of mind”

  • = Proficiency

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Complexity continua l.jpg

Complexity continua

  • Rote recall to strategic thinking (Webb)

  • Memorize, perform routine procedures, communicate understanding, perform nonroutine problems, conjecture/generalize/prove (Porter & Smithson)

  • Concrete to abstract (Dienes)

  • Global to analytic to deductive (van Hiele)

  • Pre-operational to operational (Piaget & Beth)

  • Concepts to rules to problem-solving (Gagne)

  • Enactive to symbolic (Bruner)

  • External to internal (Vygotsky)

  • Situated to decontextualized (Cole & Griffen; Greeno)

  • Facts/skills to applications to analysis/synthesis/evaluation (Bloom)

  • Naïve interpretations (based on superficial characteristics) to scientific models (focused on key attributes and underlying regularities) (Steen)

  • Application, learning potential, metacognition, beliefs and values, whole (Ginsburg et al.)

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Developing learning progressions and assessments l.jpg

Developing Learning Progressions and Assessments

  • Don’t wait for conclusive research and consensus about “the right way”

    • For instance, researchers might argue whether the “map” type progressions or list type are better, but you need to figure out what works best for your purposes

  • Do understand the conceptual frameworks, and decide what emphasis you’ll make (and why)

  • Do collect as many examples of different approaches as you can

  • Do clarify your purpose(s)

  • Consider revising/extending your state content standards

  • Analyze curricula (frameworks, syllabi, maps)

  • Think about creating developmental rubrics compared with or in addition to achievement level rubrics (I know this is a major shift)

  • Consider supporting teacher benchmarking to develop learning progressions at some level (state, multi-district, district)

Scott Marion. Center for Assessment. WY FAI February 4, 2008


References l.jpg

References

  • AAAS. (2001, 2007). Atlas of Science Literacy (Vols. I and II). Washington, DC: American Association for the Advancement of Science and the National Science Teachers Association.

  • Applebee, Arthur. (1978). The Child’s Concept of Story: Ages Two to Seventeen. Chicago: University of Chicago Press.

  • Bruner, Jerome. (1960). The Process of Education. Cambridge, MA: Harvard University Press.

  • Gong, Brian, Venezky, Richard, & Mioduser, David. (1992). Instructional Assessments: Lever for systemic change in science education classrooms. In Journal of Science Education and Technology, 1 (1), 157-176.

  • Kennedy, Cathleen & Wilson, Mark. (2007). Using progress variables to map intellectual development. Presentation at the MARCES Conference, University of Maryland-College Park.

  • Lesh, Richard, Lamon, Susan J., Gong, Brian, & Post, Thomas R. (1992). Using learning progress maps to improve instructional decision making. In R. Lesh & S.J. Lamon (Eds.), Assessment of Authentic Performance in School Mathematics. Washington, DC: AAAS Press.

  • Masters, Geoff & Forster, Margaret. (1996). Progress Maps. (Part of the Assessment Resource Kit) Melbourne, Australia: The Australian Council for Educational Research.

  • Mariotti, Arleen S. & Homan, Susan P. Linking reading assessment to instruction (4th ed.).

  • National Science Foundation solicitation, http://www.nsf.gov/pubs/2005/nsf05612/nsf05612.htm

  • Smith, Wiser, Anderson, & Krajcek (in press).

  • Wilson, Mark & Bertentahl, Meryl (Eds.). (2005). Systems for State Science Assessment. Board on Testing and Assessment, Center for Education, National Research Council of the National Academies. Washington, DC: National Academies Press.

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Break l.jpg

Break

  • Let’s take a 5 minute break

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Student work materials l.jpg

Student Work-Materials

  • I’ve given you samples of student work from:

    • Wyoming Activities Assessment Consortium

      • Mathematics

      • Language Arts

    • NAEP

      • 4th grade science

      • 8th grade science

  • We’re going to focus on just one aspect of the student work protocol provided to you by Barb and Janet in November…

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Slide38 l.jpg

‘Looking at Student Work’ Form and Protocol

Section #5: Most important section

List strategies that the teacher can do to address the students’ needs (individual or small group instruction).

Learning Needs for ALL Students (whole class instruction): Strategies that could be implemented “whole” group

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Student work instructions l.jpg

Student Work: Instructions

  • At your table, please examine as many of the sets of work as you have time for, but at least two different sets

  • I’m a nice guy, so I left the annotations and comments attached, but you can see—and you learned last time—that making rubric or target-related comments/notes on the student work can be an important source of feedback

  • You can use the annotations, but I think you’ll need more than that, to figure out what you would do next with each of the students. That includes students whose work is satisfactory

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Student work instructions 2 l.jpg

Student Work: Instructions-2

  • Try to think about the section or component of the learning progression that is represented by the particular work sample and where in the progression you would want to help the student move next

    • Avoid the problem that Lorrie Shepard calls the “1000 mini lessons”

  • Write notes for each particular work sample about:

    • What you would do next in terms of feedback to the student (the annotations help with this) and your instructional strategies

    • What additional information do you need or want?

    • Your analytic processes—how are you thinking about this? (this can help your colleagues)

    • What are the implications for assessment design?

  • 45 minutes or so

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Debrief l.jpg

Debrief

  • Math activity

  • Science activities

  • Language arts activity

    • What you would do next in terms of feedback to the student and your instructional strategies

    • What additional information would you need or want?

    • Your analytic processes—how are you thinking about this?

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Debrief cont implications for assessment design l.jpg

Debrief-cont.: Implications for assessment design

  • How can the design of our tasks and rubrics facilitate feedback to teachers and students?

  • How can the design of tasks and rubrics facilitate helping us to figure out what to do next?

  • What are the differences in information received from the short compared with the longer tasks?

Scott Marion. Center for Assessment. WY FAI February 4, 2008


In the moment formative assessment l.jpg

In the moment formative assessment

  • Many argue—and I tend to agree—that true formative assessment occurs “in the moment” such as:

    • questioning in a whole class situation

    • Observing and probing students working on a group project

    • Having students “think aloud” when they are putting math problems on the board

  • How would you apply the kind of thinking and acting we just did with these in the moment activities?

Scott Marion. Center for Assessment. WY FAI February 4, 2008


This is really hard so how do you get smart about this stuff l.jpg

This is really hard, so how do you get smart about this stuff?

  • Disciplinary understanding

  • Intense collaboration

    • Lesson study-type approaches

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Larger scale assessment l.jpg

Larger scale assessment

  • One of the major values of formative assessment is that the information is descriptive…some have argued that it isn’t formative once it is scored

  • Interim and summative assessments almost always come with score reports. Some are good and most are pretty poor.

    • Using these results to inform decisions and actions is often called “data based decision making (DBDM)”

Scott Marion. Center for Assessment. WY FAI February 4, 2008


The black box still applies l.jpg

The black box still applies…

  • We still have to turn data into information, decisions, and actions

  • What kinds of information would you like to be able to get from score reports?

    • Interim assessments

    • Summative assessments

  • Do you think you’ll be able to get this information?

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Skills and knowledge l.jpg

Skills and knowledge

  • Knowledge is power!

  • What kinds of skills do you need to be able to turn assessment reports into useable actions?

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Knowledge and skills l.jpg

Knowledge and skills

  • Some basic statistics—central tendency, variability, correlation

  • The concept of error

    • A issue for all assessments

    • Much larger for subscores

    • Comparisons can be tricky

  • A little bit about test design, construction, and analysis (psychometrics)

  • It is easier to get smarter about these skills because so much as been written

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Examining score reports l.jpg

Examining score reports

  • Let’s try to use the same approach for looking at these score reports as we did with student work

  • What are the actions that you might propose as a result of these scores?

    • What is the level targeted by these proposed actions?

      • Student, classroom, school, district?

    • Why do you think these actions are justified?

    • What are some possible alternative explanations for these results?

Scott Marion. Center for Assessment. WY FAI February 4, 2008


What s common and what s not l.jpg

What’s Common and What’s Not

  • What are the similarities and differences about the thought processes and skills necessary to operate within the black box for classroom formative assessment compared with interim and summative assessments?

Scott Marion. Center for Assessment. WY FAI February 4, 2008


Last thoughts l.jpg

Last thoughts

  • This is hard!

  • Continue to work on learning progressions

  • Don’t be satisfied just administering and scoring assessments—there’s a lot more to learn!

  • If there were simple solutions we would have found them already

  • Continue to operate using a hypothesis testing approach

    • Avoid a confirmationist bias

    • Search for alternative explanations

  • Embrace complexity and uncertainty

Scott Marion. Center for Assessment. WY FAI February 4, 2008


For more information l.jpg

For more information

  • Scott Marion, Center for Assessment

  • [email protected]

  • www.nciea.org

Scott Marion. Center for Assessment. WY FAI February 4, 2008


  • Login