Emily Jolley Let’s Measure Together Project – Constructed Response

1. Emily Jolley Let’s Measure Together Project – Constructed Response Mentor: Dr. Tammy Howard, DPI August 6, 2013 kenanfellows.org dpi.state.nc.us/accountability

2. Kenan Fellows Externship • 5 week externship this summer at the Testing and Accountability office of DPI • Mentor is Dr. Tammy Howard – director of the testing and accountability division for DPI • Attended department meetings, met with test development staff, item writers, content specialists, psychometricians, and just about everyone else involved in testing and accountability.

3. NC DPI Accountability Services Division • The mission of the Accountability Services Division is to promote the academic achievement of all North Carolina public school students and to assist stakeholders in understanding and gauging this achievement against state and national standards.

4. NC DPI Accountability Services Division The major thrust of this mission is three-fold: • the design and development of reliable and valid assessment instruments • the uniform implementation of and access to suitable assessment instruments for all students • the provision of accurate and statistically appropriate reports.

5. EOCs and EOGs • Fulfill requirements of Federal NCLB Act of 2001. • Developed by NCSU Technical Outreach for Public Schools (TOPS) Program as directed by DPI • Questions are written by North Carolina teachers and then edited by TOPS and DPI. • Standardized administration and grading statewide. • Rigorous development process that takes years to complete. • Used to assess student learning and growth.

6. Measures of Student Learning/ Common Exams • Required by Race to the Top Grant. • Developed by NCSU Technical Outreach for Public Schools (TOPS) Program as directed by NC D.P.I. • Questions are written by North Carolina teachers. • Not standardized, considered a local assessment administered and graded by individual Local Education Authorities (LEAs) • Used to assess teachers for Standard 6

7. Item Writing Trainings • Attended testing item writing training for teachers by TOPS. • Focused on breaking down the standards into Revised Bloom’s Taxonomy verbs, standard objects, and supporting items. • After learning how to break down the standards, basic instructions were given about writing items.

8. Item Writing Trainings Revised Bloom’s Knowledge Dimensions: • Factual – basic elements • Conceptual – interrelationships • Procedural – how to do something • Metacognition – awareness of what you know Revised Bloom’s Cognitive Processes: • Remember, Understand, Apply • Analyze, Evaluate, Create

9. Item Writing Trainings For example – breaking down a Civics & Economics standard: CE.C&G.2.7: Analyze contemporary issues and governmental responses at the local, state, and national levels in terms of how they promote the public interest and/or general welfare (eg: taxes, immigration, naturalization, civil rights, economic development, annexation, redistricting, zoning, national security, health care, etc)

10. Most essential standards for high school social studies fall under the understand or analyze range of conceptual knowledge. A few fall under the understand or evaluate range, and a couple standards are procedural knowledge (especially in World History & American History I & II).

11. Why Constructed Response? • Constructed Response questions were included in MSLs for a variety of reasons: • Almost none of the new standards are based on factual knowledge only. Because the new essential standards are more conceptual, having only multiple choice questions is not the best way to assess these standards • Many teachers have been asking for assessments that more closely reflect students’ learning, and constructed response questions help facilitate that request

12. Why Constructed Response? • As teachers, we need to ensure that the written, taught, and tested curriculum are aligned. • We can of course go beyond what the standards require, but we need to make sure we’re teaching and assessing the students at least at the level the standards are written.

13. What about the Common Core State Standards? • We also have literacy standards for history/social studies. These are not technically tested by MSLs, but we should still be teaching them with fidelity to help our students become better readers and writers. A sample reading standard: • CCSS.ELA-Literacy.RH.9-10.8: Assess the extent to which the reasoning and evidence in a text support the author’s claims.

14. Almost all of the Common Core State Standards for literacy in high school social studies fall under the apply, analyze, or evaluate ranges of procedural knowledge, with a few falling under the create range of procedural knowledge (especially the writing strand).

15. Essential Standards Breakdown • Now, let’s practice breaking down our own standards into this basic format. Use the template from my site to start breaking down your standards: • The unpacking documents can also be found on my site. It may be more useful (along with the examples in the standards) when figuring out what content to teach.

16. Quick Assessment • When you’ve broken down some of your standards, take the quick survey on my site to assess your comfort with this process. • The survey has some standards from each course. It also has a few of the released constructed response questions from last fall. You should categorize each by RBT cognitive process and knowledge dimension. • I highly recommend breaking down all the standards from your course this way (and looking through the unpacking document!)

17. Constructed Response Criteria • Now we’re ready to write! • You need a few things to begin writing a constructed response question. Basic format: • Stimulus or Context Statement (optional): chart, map, picture, graph, text, quote, intro, or scenario • Prompt or Task Statement: directive to students • Requirements Statement: response specifics • Scoring Criteria: rubric, range 0-4 points

18. C&E Fall 2012 CR Released Example: • Stimulus/Context Statement: Throughout its history, American democracy has upheld and strengthened citizens’ individual rights. • Prompt or Task Statement: Take a position that argues for or against the above statement. • Requirements Statement: Provide at least two specific historic examples (e.g., legislation, Supreme Court cases, executive orders, or other actions by the federal government) that support your position. • Scoring Criteria: • Score 0 No response or the response does not address the prompt. • Score 1 Fulfills only 1 of 3 requirements of a level 3 performance • Score 2 Fulfills 2 of 3 requirements of a level 3 performance • Score 3 Takes a position that argues for or against the statement; provides one historic example that supports the position; provides a second historic example that supports the position

19. C&E Fall 2012 CR Released Example: • Throughout its history, American democracy has upheld and strengthened citizens’ individual rights. • Take a position that argues for or against the above statement. Provide at least two specific historic examples (e.g., legislation, Supreme Court cases, executive orders, or other actions by the federal government) that support your position. • Standard Alignment: • CE.C&G.2.6 Evaluate the authority federal, state, and local governments have over individuals’ rights and privileges (e.g., Bill of Rights, Delegated Powers, Reserved Powers, Concurrent Powers, Pardons, Writ of habeas corpus, Judicial Process, states’ rights, Patriot Act, etc.). • CE.C&G.3.8 Evaluate the rights of individuals in terms of how well those rights have been upheld by democratic government in the United States.

20. Constructed Response Tips • Start with the objective(s) you want to measure. Most constructed response questions will satisfy more than one objective. • You must use the same RBT level that the standard requires. (You also must use the same content… obviously!) • You should adhere to the same basic scoring criteria whether you are creating a 2, 3, or 4 point question. Remember, students don’t see this criteria on the MSL, so you should teach them how to pick out points (there’s a PowerPoint about this on my site). • Review the released questions and use the RBT question stems if you need help getting started!

21. Constructed Response • Choose at least 2 objectives from your course and write your own constructed response using the template provided. • After writing your constructed response, please trade it with a partner. You should break down each other’s CR and determine if it fits the criteria and matches the content and RBT levels of the standard.

22. Public Google Spreadsheet • Please load the constructed response question you created today into the Google spreadsheet found on my site: https://docs.google.com/spreadsheet/ccc?key=0AtoXZ1jIf_QadDJ5N3RDTC0zZlhsSnVpOHpJSEE4Wmc&usp=sharing • This is a public spreadsheet that anyone can access. I will update it throughout the year with constructed response questions I write (mostly for C&E), and I hope you will do the same. • Please share with everyone!

23. Other Resources • TOPS suggested these resources for writing constructed response questions and understanding concept-based instruction: • Lynn Erickson - Concept-Based Curriculum and Instruction • LorinAnderson & David Krathwohl- A Taxonomy of Learning, Teaching, and Assessing • Robert Mager - Measuring Instructional Results • RBT Resources (Section 5): http://tpri.wikispaces.com/space/content 

24. Contact Info • Please contact me if you have any questions: Emily Jolley Millbrook High School ejolley@wcpss.net

25. Preparing Students for the Real World Takes Teachers Who Learn in the Real World BECOME A KENAN FELLOW • See first-hand how researchers and industry experts use STEM skills in real-world settings • Apply that knowledge to better prepare students for the jobs of tomorrow • Enrich your career through professional development opportunities • Serve as an ambassador for educational excellence in your school, community and across the state Find out more: kenanfellows.org

26. Kenan Fellows Program Timeline

27. Kenan Fellows Program Stay Connected facebook.com/KenanFellows @KenanFellows Kenan Fellows Program Group pinterest.com/kenanfellows youtube.com/user/KenanFellows