advancing assessment of quantitative and scientific reasoning n.
Skip this Video
Loading SlideShow in 5 Seconds..
Advancing Assessment of Quantitative and Scientific Reasoning PowerPoint Presentation
Download Presentation
Advancing Assessment of Quantitative and Scientific Reasoning

Loading in 2 Seconds...

play fullscreen
1 / 21

Advancing Assessment of Quantitative and Scientific Reasoning - PowerPoint PPT Presentation

  • Uploaded on

Advancing Assessment of Quantitative and Scientific Reasoning. Donna L. Sundre Amy D. Thelk Center for Assessment and Research Studies (CARS) James Madison University Overview of talk. Current NSF Research project History of the test instrument

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Advancing Assessment of Quantitative and Scientific Reasoning' - anakin

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
advancing assessment of quantitative and scientific reasoning

Advancing Assessment of Quantitative and Scientific Reasoning

Donna L. Sundre

Amy D. Thelk

Center for Assessment and Research Studies (CARS)

James Madison University

overview of talk
Overview of talk
  • Current NSF Research project
  • History of the test instrument
  • Phase I: Results from JMU
  • Phase II: Future directions
  • Results from some of our partners:
  • Michigan State
  • Truman State
  • Virginia State
current nsf project
Current NSF Project
  • 3-year grant funded by National Science Foundation: “Advancing assessment of scientific and quantitative reasoning”
  • Hersh & Benjamin (2002) listed four barriers to assessing general education learning outcomes:
    • confusion;
    • definitional drift;
    • lack of adequate measures, and
    • misconception that general education cannot be measured
  • This project addresses all of these concerns with special emphasis on the dearth of adequate measures
objective of nsf project
Objective of NSF project
  • Exploring the psychometric quality and generalizability of JMU’s Quantitative and Scientific Reasoning instruments to institutions with diverse missions and serving diverse populations.
partner institutions
Partner Institutions
  • Virginia State University: State-supported; Historically Black institution
  • Michigan State University: State-supported; Research institution
  • Truman State University: State-supported; Midwestern liberal arts institution
  • St. Mary’s University (Texas): Independent; Roman-Catholic; Hispanic Serving institution
project phases
Project phases
  • Phase I: First Faculty institute (conducted July 2007 at JMU); followed by data collection, identification of barriers, and reporting of results
  • Phase II: Validity studies (to be developed and discussed during second faculty institute, July 2008), dissemination of findings and institutional reports
history of the instrument
History of the instrument
  • Natural World test, developed at JMU, currently in 9th version
  • Successfully used for assessment of General Education program effectiveness in scientific and quantitative reasoning
  • Generates two subscores: SR and QR
  • Summary of results since 2001
  • Table of Results -- 5 Test Versions.doc
adaptation of an instrument
Adaptation of an instrument
  • JMU instrument has been carefully scrutinized for over 10 years
  • The QR and SR is currently administered at over 25 institutions across the nation
  • NSF decided to fund this CCLI project to further study procedures for adoption and adaptation of instruments and assessment models
step 1 mapping items to objectives
Step 1: Mapping Items to Objectives
  • Relating test items to stated objectives for each institution
    • In the past back translation method was used (Dawis, 1987) ..\..\JMU\NSF Grant\Truman\Blank ObjectiveGrid_truman.doc
    • Participants at the NSF Faculty Institute used a new content alignment method that was reported on at NCME (Miller, Setzer, Sundre & Zeng, 2007)
    • Forms were custom made for each institution

Example Content Alignment form.doc

early content validity evidence
Early content validity evidence
  • Results strongly support generalizability of test items
    • Truman State: 100% of items mapped to their objectives
    • Michigan State: 98% (1 item not mapped)
    • Virginia State: 97% (2 items unmapped)
    • St. Mary’s: 92% (5 items not mapped)
  • Mapping of items alone is not sufficient
  • Balance across objectives must be obtained
  • Teams then created additional items to cover identified gaps in content coverage
    • 14 for MSU; 11 for St. Mary’s; 10 for Truman State; 4 for VSU
step 2 data collection and analysis
Step 2: Data Collection and Analysis
  • During Fall 2007 semester, test was administered to students at 3 of the 4 partner institutions
  • Spring 2008 – data collection from students at sophomore level or above
  • Results so far
    • Means not given: This activity is not intended to promote comparison of students across institutions
    • At this stage, reliabilities provide the most compelling generalizability evidence; of course, the upcoming validity studies will be informative
research at jmu
Research at JMU
  • Standard Setting to aid in interpretation
  • Validity evidence: Instrument aligns with curriculum
standard setting
Standard Setting
  • Used Angoff Method to set standards
  • Our process was informal, unique
  • Results look meaningful but we’ll reevaluate as we collect more data in upcoming administrations
phase ii studies
Phase II studies
  • Samples of Upcoming Studies:
  • Correlational Studies: Is there a relationship between scores on the QR/SR and other standardized tests? … and other academic indicators?
  • Comparison of means or models: Is there a variation in the level of student achievement based upon demographic variables? Is there a relationship between scores on the QR/SR and declared majors? Can this instrument be used as a predictor for success and/or retention for specific majors?
  • Qualitative Research: Will institutional differences be reflected in the results of a qualitative interview that accompanies the administration of QRSR?
  • Dawis, R. (1987). Scale construction. Journal of Counseling Psychology, 34, 481-489.
  • Hersh, R. H., & Benjamin, R. (2002). Assessing selected liberal education outcomes: A new approach. Peer Review, 4 (2/3), 11-15.
  • Miller, B. J., Setzer, C., Sundre, D. L., & Zeng, X. (2007, April). Content validity: A comparison of two methods.  Paper presentation to the National Council on Measurement in Education. Chicago, IL.

Any Questions?

Up next: Michigan State University