1 / 54

TIMSS 2007

TIMSS 2007. Ina V.S. Mullis and Michael O. Martin 46 th IEA General Assembly Helsinki, 10-13 October, 2005. TIMSS 2007. General Progress Overview Detailed Look at Mathematics and Science Assessment Development. TIMSS 2007. Mathematics and Science Grades 4 and 8

chuck
Download Presentation

TIMSS 2007

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. TIMSS 2007 Ina V.S. Mullis and Michael O. Martin 46th IEA General Assembly Helsinki, 10-13 October, 2005

  2. TIMSS 2007 • General Progress Overview • Detailed Look at Mathematics and Science Assessment Development

  3. TIMSS 2007 • Mathematics and Science • Grades 4 and 8 • Trends 1995, 1999, 2003, 2007 • Baseline for New Countries

  4. Algeria Armenia Australia Austria Bahrain Bosnia Botswana Bulgaria Ontario, Canada Quebec, Canada Chinese Taipei Colombia Czech Republic Denmark Djibouti Egypt El Salvador England Georgia Ghana Honduras Qatar Romania Russian Federation Saudi Arabia Scotland Serbia Singapore Slovak Republic Slovenia South Africa Basque Ctry, Spain Sweden Syria Thailand Tunisia Turkey Ukraine United States Uzbekistan Yemen Countries Participating Hong Kong, SAR Indonesia Iran Israel Italy Japan Jordan Korea Kuwait Latvia Lebanon Lithuania Malaysia Malta Mongolia Morocco Netherlands New Zealand Norway Oman Palestinian Auth.

  5. Since Last General Assembly, Have Been Working on Assessment Development • Published TIMSS 2007 Assessment Frameworks • Developed Field-test Achievement Items and Scoring Guides • Developed Field-test Questionnaires • Drafted Procedural Manuals

  6. TIMSS 2007 SMIRC Mathematics Experts Kiril Bankov, Bulgaria Hanako Senuma, Japan Khattab Mohammad Abu Lebdeh, Jordan Robert Garden, New Zealand Liv Sissel Gronmo, Norway Mary Lindquist, United States

  7. TIMSS 2007 SMIRC Science Experts Ahmed Mohammed Rafea, Bahrain Chen-yung Lin, Chinese Taipei Jophus Anamuah-Mensah, Ghana Gabriela Noveanu, Romania Galina Kovaleva, Russian Federation Jackie Heaton, Scotland Audrey Champagne, United States

  8. TIMSS 2007 QIRC Sue Thomson, Australia Serge Baillergeon, Quebec Solaiman El-Khodary El-Sheikh, Egypt Martina Meelissen, The Netherlands Boey Kok Leong, Singapore Barbara Japelj, Slovenia Vijay Reddy, South Africa Patrick Gonzales, United States

  9. TIMSS 2007 Assessment Specialists • Mathematics Coordinator • Graham Ruddock, NFER • Science Coordinator • Christine O’Sullivan, K-12 Consulting • Special “Task Force” Consultants in Mathematics and Science • Intensive Week-long Working Sessions

  10. TIMSS 2007 Assessment Specialists • Staff at TIMSS & PIRLS ISC • Content/Measurement Experience • Production/Graphics Experience • NRCs – National Research Coordinators • Wealth of Knowledge and Experience

  11. Improvements in Mathematics and Science Frameworks • Increased Validity Through Better Curricular Alignment at 4th and 8th Grades • Specifically Targeted Content Domains and Assessment Topics • Modified and Refined Cognitive Domains • Knowing, Applying, and Reasoning

  12. Mathematics Content Domains

  13. Science Content Domains

  14. Enhanced Design • More Response Time per Item • Total Student Time Unchanged • Simplified Block-Booklet Assignment • Fewer Blocks of Items per Booklet • Two Blocks per Subject per Booklet • 14 Booklets in Total • Bridge Study for Trend Countries

  15. Developed Field-test Achievement Items and Scoring Guides for NRC Review • Mathematics Grade 4 (200) • Mathematics Grade 8 (240) • Science Grade 4 (200) • Science Grade 8 (240) Total = 880 Items

  16. Developed Field-test Questionnaires for NRC Review • Student (Grades 4 and 8) • Teacher (Grade 4, Grade 8 Mathematics, Grade 8 Science) • School (Grades 4 and 8) • Curriculum (Mathematics and Science, Grades 4 and 8)

  17. Planned TIMSS 2007 Encyclopedia Mathematics and Science Education in Primary and Lower Secondary Grades • Curriculum • Instruction • Teachers and Teacher Education • Examinations and Assessments

  18. Drafted Procedural Manuals • Sampling Manual and Software • To Assist Countries with School and In-school Sampling • Operations Manual • New Modular Approach

  19. TIMSS 2007Developing the Mathematics and Science Assessments

  20. Overview of Development Process • Framework • Drafting Items and Scoring Guides • Reviewing Items and Scoring Guides • Conducting the Field Test • Finalizing Items and Scoring Guides

  21. Frameworks Establish Extent of Development Effort • Time/Number of Items • What Type of Items • Multiple-choice • Constructed-response • Specific Topics Within Content and Cognitive Domains

  22. Frameworks (Cont.) Initial Development Steps TIMSS 2007 • Reviewed and Updated Mathematics and Science Frameworks from TIMSS 2003 • Reviewed and Updated Design

  23. Updating TIMSS 2007 Frameworks • Met with Coordinators to Identify Areas and Proposals for Updating • Discussed Proposals at NRC Meeting • February 2005, Cairo • Prepared and Distributed Questionnaire to Countries • Collected and Analyzed Questionnaire Results

  24. Updating TIMSS 2007 Frameworks (Cont.) • Drafted Updated Frameworks • SMIRC Reviewed and Revised Updated Frameworks – April 2005, London • Production of Updated Frameworks • NRCs Reviewed Updated Frameworks via Web and Discussed at Next Meeting • May 2005, Amsterdam

  25. Updating TIMSS 2007 Frameworks (Cont.) • Production of Final Frameworks • Published TIMSS 2007 Assessment Frameworks – Sept. 2005 • Web and Print • Distributed TIMSS 2007 Assessment Frameworks

  26. Establishing Scope of TIMSS 2007 Item-writing Effort • Updated TIMSS 2007 Design Has Same Number of Item Blocks as TIMSS 2003 • Released Half the Item Blocks in 2003 • 7 Math, 7 Science Blocks Each Grade

  27. Establishing Scope of TIMSS 2007 Item-writing Effort (Cont.) • Number of Released Items • Mathematics: 80 Grade 4; 99 Grade 8 • Science: 76 Grade 4; 95 Grade 8 • Need to Replace the Released Items from 2003 • Field Test About Twice Required Number

  28. Establishing Scope of TIMSS 2007 Item-writing Effort (Cont.) Needed to Actually Field Test • Mathematics • Grade 4: 150; Grade 8: 200 • Science • Grade 4: 150; Grade 8: 200 • Total: 700 Items • Total Allowing for Review Attrition: 900

  29. Establishing Scope of TIMSS 2007 Item-writing Effort (Cont.) Items Need to Meet Target Percentages Established in Mathematics and Science Frameworks – e.g., 8th – 30% Algebra • 3 Topic Areas in Algebra • Patterns, Expressions, Equations • 12 Objectives Across Topic Areas • Less Unreleased Trend Items

  30. Drafting TIMSS 2007 Field-test Items and Scoring Guides Planned NRC Item-writing Meeting – May 2005, Amsterdam • Determined Item-writing Goals for Each Content Domain in Each Subject – e.g., 48 Items 8th Grade Geometry • Asked NRCs Preferred Area – e.g., Subject Area, Grade, and Specialty for Science (Physics, Biology, etc.)

  31. Drafting TIMSS 2007 Field-test Items and Scoring Guides (Cont.) Planned NRC Item-writing Meeting – May 2005, Amsterdam • Configured NRCs into 27 Teams – About Three Persons per Team • Assigned Each Team Specific Topic Areas of Framework – e.g., Fractions and Decimals, Forces and Motion

  32. Drafting TIMSS 2007 Field-test Items and Scoring Guides (Cont.) • Prepared TIMSS 2007 Item-writing Guidelines for Multiple-choice and Constructed-response Items (20 pages) • Distributed Item-writing Materials to NRCs • Sent Guidelines • Sent Assignments • Reminded NRCs to Bring Laptops and Item-writing Resources – e.g., Popular Textbooks

  33. Drafting TIMSS 2007 Field-test Items and Scoring Guides (Cont.) • Managed NRC Item-writing Process • About 80 Persons • Two Days • Enormous Amounts of Technology • Yield = 760 Draft Items and Scoring Guides VERY IMPRESSIVE!

  34. Reviewing the TIMSS 2007 Field-test Items and Scoring Guides • Coordinator and ISC Reviewed • OK, Revise, Delete • Drafted Items to Fill Gaps • Production – Graphics/Artwork • InDesign Production Softare • Comissioned Artwork – e.g., Ocean Organisms • Iterative Internal Review/Revision • Task Force Reviewed and Wrote Items to Fill Gaps

  35. Reviewing the TIMSS 2007 Field-test Items and Scoring Guides (Cont.) • Production – Graphics/Artwork for Revised and New Items • Entered Items and Scoring Guides into Database • SMIRC Reviewed and Wrote Items to Fill Gaps – August 2005, Kyoto • Production – Graphics/Artwork for Revised and New Items

  36. Reviewing the TIMSS 2007 Field-test Items and Scoring Guides (Cont.) • Posted Field-test Items and Scoring Guides for NRC Review • Grade 4: Mathematics – 200, Science – 200 • Grade 8: Mathematics – 240, Science – 240 • Total: 880 (Contains Extra Items) This Is Happening Now!

  37. Reviewing the TIMSS 2007 Field-test Items and Scoring Guides (Cont.) • NRC Meeting to Review Field-test Items and Scoring Guides – November 2005, Ljubljana • Production – Graphics/Artwork for Revised Items • Assemble Items into Blocks • Post Field-test Blocks/Booklets for Countries – December 2005

  38. Developing the TIMSS 2007 Field-test Scoring Training Materials • Since Field-test Twice Number Needed, This Is Substantial Effort • 500 Scoring Guides (200 at 4th, 300 at 8th) • Conduct Pilot in English-speaking Countries to Obtain Sample Student Responses – January 2006 • Scan Student Responses for Review

  39. Developing the TIMSS 2007 Field-test Scoring Training Materials (Cont.) • Update the Scoring Training Manual • Task Force Meeting to Select Sample Responses to: • Illustrate Categories in Scoring Guides • Create Training Packets • 500 x 10 = 5,000 Responses • Provide Sets of Practice Papers • 500 x 15 = 7,500 Responses

  40. Developing the TIMSS 2007 Field-test Scoring Training Materials (Cont.) • Merge Selected Responses with Database Information • Distribute Scoring Training Materials via Web • NRC Meeting to Conduct Scoring Training – March 2006, Malta

  41. Conducting the TIMSS 2007 Field Test • Determine Number of Blocks/Booklets to Be Field Tested • 28 Blocks Total Each Grade • 14 Mathematics Blocks • 14 Science Blocks • 7 Booklets Each Grade (4 Blocks per Booklet)

  42. Conducting the TIMSS 2007 Field Test (Cont.) • Determine Number of Students to Be Field Tested in Each Country • 200 per Booklet = 1,400 per Grade • Field-test Sampling Integrated with Main TIMSS 2007 Sampling Plan • Every Participating Country Required to Take Part in Field Test

  43. Analyzing the TIMSS 2007 Field-test Data • Diagnostic Statistics • Difficulty • Discrimination • Scoring Reliability • Item-by-Country Interaction • For Each Country and Language Groups Within Country

  44. Reviewing the TIMSS 2007 Field-test Results • TIMSS & PIRLS ISC and Coordinators Review Results – Check Analysis and Note Any Issues • SMIRC Review Results, Select/Revise Items and Blocks for TIMSS 2007 Data Collection – July 2006 • NRCs Review Selected/Revised Blocks and Approve Final Item Selection – Aug. 2006

  45. Distributing the TIMSS 2007 Blocks/Booklets • Revise InDesign Versions • Assemble Final Blocks • Assemble Final InDesign Booklet Templates • Distribute Final Mathematics and Science Assessments to Countries • Web and CD

  46. Conducting the TIMSS 2007 Scoring Training • Scan Selected Student Responses from Field test • Task Force Meeting to Update Sample Responses for: • Scoring Guides • Training Packets • Practice Papers

  47. Conducting the TIMSS 2007 Scoring Training (Cont.) • Merge Selected Responses with Database Information • Distribute Scoring Training Materials via Web • NRC Meeting to Conduct Scoring Training – March 2007 Note: A Separate Scoring Training Is Conducted Earlier for Southern Hemisphere – Nov. 2006

  48. Now, Let’s Recap…Overview of Major Steps Process for Developing the TIMSS 2007 Mathematics and Science Items

  49. Overview of TIMSS 2007 Item Development Process • Reviewed and Updated TIMSS 2003 Frameworks and Assessment Design • Established Time/Number of Items Needed for Assessment and Field Test • Drafted Field-test Items and Scoring Guides • Reviewed Items and Produced InDesign Version with Artwork • ITERATED Review, Revise, Update Until Done

  50. Overview of TIMSS 2007 Item Development Process (Cont.) • Entered Items and Scoring Guides into Item Database • Tracks Items by Type and Framework Classifications • Assembled Items into Blocks (e.g., 20-minute sets) • Distributed Field-test Blocks/Booklets to Countries • Translation/Production and Verification

More Related