Advancing assessment of quantitative and scientific reasoning
This presentation is the property of its rightful owner.
Sponsored Links
1 / 21

Advancing Assessment of Quantitative and Scientific Reasoning PowerPoint PPT Presentation


  • 50 Views
  • Uploaded on
  • Presentation posted in: General

Advancing Assessment of Quantitative and Scientific Reasoning. Donna L. Sundre Amy D. Thelk Center for Assessment and Research Studies (CARS) James Madison University www.jmu.edu/assessment/. Overview of talk. Current NSF Research project History of the test instrument

Download Presentation

Advancing Assessment of Quantitative and Scientific Reasoning

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Advancing assessment of quantitative and scientific reasoning

Advancing Assessment of Quantitative and Scientific Reasoning

Donna L. Sundre

Amy D. Thelk

Center for Assessment and Research Studies (CARS)

James Madison University

www.jmu.edu/assessment/


Overview of talk

Overview of talk

  • Current NSF Research project

  • History of the test instrument

  • Phase I: Results from JMU

  • Phase II: Future directions

  • Results from some of our partners:

  • Michigan State

  • Truman State

  • Virginia State


Current nsf project

Current NSF Project

  • 3-year grant funded by National Science Foundation: “Advancing assessment of scientific and quantitative reasoning”

  • Hersh & Benjamin (2002) listed four barriers to assessing general education learning outcomes:

    • confusion;

    • definitional drift;

    • lack of adequate measures, and

    • misconception that general education cannot be measured

  • This project addresses all of these concerns with special emphasis on the dearth of adequate measures


Objective of nsf project

Objective of NSF project

  • Exploring the psychometric quality and generalizability of JMU’s Quantitative and Scientific Reasoning instruments to institutions with diverse missions and serving diverse populations.


Partner institutions

Partner Institutions

  • Virginia State University: State-supported; Historically Black institution

  • Michigan State University: State-supported; Research institution

  • Truman State University: State-supported; Midwestern liberal arts institution

  • St. Mary’s University (Texas): Independent; Roman-Catholic; Hispanic Serving institution


Project phases

Project phases

  • Phase I: First Faculty institute (conducted July 2007 at JMU); followed by data collection, identification of barriers, and reporting of results

  • Phase II: Validity studies (to be developed and discussed during second faculty institute, July 2008), dissemination of findings and institutional reports


History of the instrument

History of the instrument

  • Natural World test, developed at JMU, currently in 9th version

  • Successfully used for assessment of General Education program effectiveness in scientific and quantitative reasoning

  • Generates two subscores: SR and QR

  • Summary of results since 2001

  • Table of Results -- 5 Test Versions.doc


Adaptation of an instrument

Adaptation of an instrument

  • JMU instrument has been carefully scrutinized for over 10 years

  • The QR and SR is currently administered at over 25 institutions across the nation

  • NSF decided to fund this CCLI project to further study procedures for adoption and adaptation of instruments and assessment models


Evaluating the generalizability of the instrument

Evaluating the generalizability of the instrument


Step 1 mapping items to objectives

Step 1: Mapping Items to Objectives

  • Relating test items to stated objectives for each institution

    • In the past back translation method was used (Dawis, 1987) ..\..\JMU\NSF Grant\Truman\Blank ObjectiveGrid_truman.doc

    • Participants at the NSF Faculty Institute used a new content alignment method that was reported on at NCME (Miller, Setzer, Sundre & Zeng, 2007)

    • Forms were custom made for each institution

      Example Content Alignment form.doc


Early content validity evidence

Early content validity evidence

  • Results strongly support generalizability of test items

    • Truman State: 100% of items mapped to their objectives

    • Michigan State: 98% (1 item not mapped)

    • Virginia State: 97% (2 items unmapped)

    • St. Mary’s: 92% (5 items not mapped)

  • Mapping of items alone is not sufficient

  • Balance across objectives must be obtained

  • Teams then created additional items to cover identified gaps in content coverage

    • 14 for MSU; 11 for St. Mary’s; 10 for Truman State; 4 for VSU


Step 2 data collection and analysis

Step 2: Data Collection and Analysis

  • During Fall 2007 semester, test was administered to students at 3 of the 4 partner institutions

  • Spring 2008 – data collection from students at sophomore level or above

  • Results so far

    • Means not given: This activity is not intended to promote comparison of students across institutions

    • At this stage, reliabilities provide the most compelling generalizability evidence; of course, the upcoming validity studies will be informative


Research at jmu

Research at JMU

  • Standard Setting to aid in interpretation

  • Validity evidence: Instrument aligns with curriculum


Standard setting

Standard Setting

  • Used Angoff Method to set standards

  • Our process was informal, unique

  • Results look meaningful but we’ll reevaluate as we collect more data in upcoming administrations


Faculty objective standards

Faculty Objective Standards


Validity evidence for instrument and curriculum at jmu

Validity evidence for instrument and curriculum at JMU


Validity evidence for instrument and curriculum at jmu 2

Validity evidence for instrument and curriculum at JMU -- 2


Phase ii studies

Phase II studies

  • Samples of Upcoming Studies:

  • Correlational Studies: Is there a relationship between scores on the QR/SR and other standardized tests? … and other academic indicators?

  • Comparison of means or models: Is there a variation in the level of student achievement based upon demographic variables? Is there a relationship between scores on the QR/SR and declared majors? Can this instrument be used as a predictor for success and/or retention for specific majors?

  • Qualitative Research: Will institutional differences be reflected in the results of a qualitative interview that accompanies the administration of QRSR?


References

References

  • Dawis, R. (1987). Scale construction. Journal of Counseling Psychology, 34, 481-489.

  • Hersh, R. H., & Benjamin, R. (2002). Assessing selected liberal education outcomes: A new approach. Peer Review, 4 (2/3), 11-15.

  • Miller, B. J., Setzer, C., Sundre, D. L., & Zeng, X. (2007, April). Content validity: A comparison of two methods.  Paper presentation to the National Council on Measurement in Education. Chicago, IL.


Advancing assessment of quantitative and scientific reasoning

Any Questions?

Up next: Michigan State University


  • Login