making computerized adaptive testing diagnostic tools for schools n.
Skip this Video
Loading SlideShow in 5 Seconds..
Making Computerized Adaptive Testing Diagnostic Tools for Schools PowerPoint Presentation
Download Presentation
Making Computerized Adaptive Testing Diagnostic Tools for Schools

Loading in 2 Seconds...

play fullscreen
1 / 55

Making Computerized Adaptive Testing Diagnostic Tools for Schools - PowerPoint PPT Presentation

  • Uploaded on

Making Computerized Adaptive Testing Diagnostic Tools for Schools. Hua-Hua Chang University of Illinois at Urbana-Champaign October 17, 2010. What is Adaptive Testing?. O riginally called tailored tests (Lord, 1970)

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Making Computerized Adaptive Testing Diagnostic Tools for Schools' - meagan

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
making computerized adaptive testing diagnostic tools for schools

Making Computerized Adaptive Testing Diagnostic Tools for Schools

Hua-Hua Chang

University of Illinois at Urbana-Champaign

October 17, 2010

what is adaptive testing
What is Adaptive Testing?
  • Originally called tailored tests (Lord, 1970)
    • Examinee are measured most effectively if items are neither too difficult nor too easy.
  • Θ: latent trait. Heuristically,
    • if the answer is correct, the next item should be more difficult;
    • If the answer is incorrect, the next item should be easier.
  • How adaptive test works?
    • An item pool, known item properties (such as difficulty level, discrimination level,..
    • Algorithm, computer, and network
    • The core is the item selection algorithm
    • Need mathematicians help to design algorithm
sequential design robbins monro process 1955
Sequential Design &Robbins-Monro Process (1955)
  • Numerous Refinements:
  • Engineering (Goodwin, Ramadge and Caines, 1981; Kumar, 1985)
  • Biomedical science (Finney, 1978)
  • Education (Lord, 1970)
the maximum information criterion mic
The Maximum Information Criterion (MIC)
  • Lord’s (1980) MIC method, the most commonly used method.
  • MIC would select items with high discrimination
  • There have been many other methods

In 2D Case: item whose volume is max should be selected

Item inf surface

Information volume

Theta region

from theoretical development to large scale operation
From Theoretical Development to Large Scale Operation
  • Issues to be addressed:
  • Should CAT only use the best items?
    • It is common only 50% items are used
  • Is CAT more secure than paper/pencil test?
    • How to improve CAT test security?
  • How to control non-statistical constraints?
  • How to get diagnostic information?
  • How to make CAT affordable to many schools?
two nsf grants and loads of papers
Two NSF Grants and loads of Papers
  • Constraint-weighted design
    • Cheng, Chang, & Yi (2006), Cheng & Chang, (2008), Cheng, Chang Douglas, & Guo (2009), etc.
  • Establish theoretical foundation
    • Chang & Ying (2009)
  • Test Security
    • Chang & Zhang (2001), Zhang & Chang (2010)
  • Cognitive diagnostic CAT
    • McGlohen & Chang (2008)
  • Multi-dimensional CAT
    • Wang & Chang (in press), Wang & Chang (accepted for publication)
  • Large scale k-12 Applications in China
    • Liu, Yu, Wang, Ding & Chang (2010)
cat transformative research
CAT & Transformative Research
  • National Science Board (2007)unanimously approved a motion to enhance support of transformative research at the NSF.
    • All proposals received after Jan 5, 2008, will be reviewed against the criterion.
      • revolutionizing entire disciplines;
      • creating entirely new fields; or
      • disrupting accepted theories and perspectives
  • Many CAT researches are transformative!
new developments
New Developments
  • Measuring Patient-Reported Outcomes
    • Conventional measures of disease such as lab results do not fully capture information about chronic diseases and how treatments affect patients.
    • CAT can be used to assess patients subjective experiences such as symptom severity, social well-being, and perceived level of health.
  • K-12 Applications
    • Large State testing
    • Teaching/learning, within School application
    • Diagnostic purpose
    • Web-based learning
challenges in nclb testing
Challenges in NCLB Testing
  • Many items are too difficult to students
    • 70% math items may be too difficult
      • The influence of this kind of test taking experience on low-achieving students is not well-understood (e.g., Roderick & Engle, 2001, Ryan & Ryan, 2005; Ryan, Ryan, Arbuthnot, & Samuels, 2007).
  • Test security of NCLB
      • The # of security violations in P&P based NCLB testing in on the rise.
      • Documented cases of such incidents have been uncovered in numerous states including New York, Texas, California, Illinois, and Massachusetts. (Jacob & Levitt, 2003, and Texas Education Agency, 2007).
cat has glowing future in the k 12 context
CAT Has Glowing Future in the K-12 Context.
  • Why not use benchmark testing?
    • Adaptive Testing can do better.
  • Quellmalz & Pellegrino (2009):
    • more than 27 states currently have operational or pilot versions of online tests, including Oregon, North Carolina, Utah, Idaho, Kansas, Wyoming, and Maryland.
    • The landscape of educational assessment is changing rapidly with the growth of computer-administered tests.
how to get diagnostic information
How to get diagnostic information?
  • Post-hoc approach (non-adaptive)
    • perform CD after students completed CAT
  • Adaptive approach
    • Select the next item which provides the max info about the student’s strength and weakness
    • Need a model, item selection algorithm
    • Psychometric theory
    • Simulation study
    • Field test

Cognitive Diagnosis

Provide examinees with more information than just a single score.

  • How? By considering the different attributes measured by the test.
  • An attribute is a “task, subtask, cognitive process, or skill” assessed by the test, such as addition or reading comprehension.

What should be reported to examinees?

Traditional Testing:



A set of scores:

One for each attribute.

(K is the total # of attributes.)

A single score


Why is this beneficial?

Feedback from an exam can be more individualized to a student’s specific

strengths and weaknesses.

Julia R.

Halle B.


The Item-Attribute Relationship

Which items measure which attributes is represented by the Q-matrix:

cognitive diagnostic models
Cognitive Diagnostic Models


  • Many models have been proposed
  • DINA model
  • Fusion model (Stout’s group)



the dina model
The DINA Model

Student i

Item j

how to adaptively select items
How to adaptively select items?
  • No direct analogy to “match theta with b-parameter”
    • Regular CAT, b-parameter with
  • Now is a vector, called latent class

The KL information Approach (Xu, Chang, & Douglass, 2004)

  • Let’s assume
  • The likelihood test is the most powerful test
  • Intuitively

the j-th item selected should make



Taking expected value assume is true

Select item j to make the following as large as possible


j: item, c: attribute pattern

4 possible patterns

Interim estimate for an examinee

Item bank


Consider two attributes and four candidate items


Change the slipping/guessing parameters of the items

  • The magnitude of the non-zero values depends on the item slipping and guessing parameters

Change the interim estimate to


The positions of the zero KL cells changed for item 2 & 3

To explain the last table in the previous slide
    • “0” means this item provides no information to discriminate the interim estimate with another possible attribute pattern.
    • The magnitude of the non-zero values depends on the item slipping and guessing parameters
    • Which cell is zero depends on the q-vector and the examinee’s interim estimate. If for a particular item (e.g., item 4 in this demo), q-vector contains all zeros, all cells will be zero.

Response data

Students’ latent class

Item parameters

new tests vs existing tests
New Tests vs. Existing Tests
  • Existing Exams
    • Analyze the responses from an existing large-scale assessment from a Cognitive Diagnosis framework.
    • Examine the results across various methods of constructing a Q-matrix.
  • New Exams
    • Identify Attributes and Content validity structure
    • Writing items according to cognitive specifications
    • Pre-testing
    • Q-matrix validation
application 1 existing dataset
Application 1: existing dataset
  • A simple random sample of 2000 examinees who took the
    • Grade 3 TAAS from Spring 2002
    • Grade 11 TEKS from Spring 2003
  • The Math & Reading portion of each test was analyzed by using the Fusion Model
  • Item selection methods
    • Kullback-Libler (KL)
    • Shannon Entropy (SHE), and etc.
  • Reference, e.g.,
    • McGlohen & Chang (2008)
    • Download from
    • Or, google Hua-Hua Chang
taxes 3 rd grade reading assessment 6 attributes application 1
Taxes 3rd grade reading assessment 6 attributes (Application 1)
  • The student will determine the meaning of words in a variety of written texts.
  • The student will identify supporting ideas in a variety of written texts.
  • The student will summarize a variety of written texts.
  • The student will perceive relationships and recognize outcomes in a variety of written texts.
  • The student will analyze information in a variety of written texts in order to make inferences and generalizations.
  • The student will recognize points of view, propaganda, and/or statements of fact and opinion in a variety of written texts.
building cat driven assessment and diagnosis to improve student learning chang ryan ies proposal
Building CAT-Driven Assessment and Diagnosis to Improve StudentLearningChang & Ryan (IES Proposal)
  • Develop the technical foundations for a CAT system to meet NCLB accountability and to inform teaching and learning.
  • In alignment with race to the top (RTTT) priorities, the proposed CAT will include
    • individualized diagnostic information to provide teachers, schools, and states with more-precise information about student achievement levels along with valuable formative feedback to inform instructional planning.
new technologies schools can use existing pcs or macs
New Technologies--- Schools can use existing PCs or MACs
  • Client/Server Architecture (CS)
    • CAT software has to be installed on each client computer ( large workload)
    • only applicable to Local Area Network (LAN)
  • Browser/Server Architecture (BS)
    • database is still on the server
    • nearly all the tasks concerning development, maintenance and upgrade, are carried out on the server.  
    • based on the Wide Area Network (WAN)
  • Advantages of BS
    • Low maintenance, no network programming
application 2 level ii english proficiency test
Application 2:Level II English Proficiency Test
  • Pretest and Calibration of Item bank
    • Pretest
      • 38,662 students from 78 schools, 12 counties participated
    • Analyzing pretest data
      • Estimated the parameters of DINA model
      • Estimated the parameters of 3PLM model
      • Calibrate attributes of item again
      • If it fits well then stop, otherwise revise q-matrix and got 3
    • Assembling the item bank with item parameters and specifications.
distribution of the students in pretest
Distribution of the students in pretest

Red: Field Test Sampling Area

Yellow and red: Current Implementation


linking design
Linking Design

Eg, this block has 10 anchor items,

The locations of the anchor items in each booklet are the same (as they appear in anchor test).


item writing
Item Writing
  • About 40 Excellent Teachers in Beijing
  • Process
    • Psychometric Training
    • Identify Attributes
    • Writing Items
    • Constructing Q-matrix
    • Pre-testing and check FITTING
    • Revise Q-matrix until fitting is ok; go to 5 if not
    • stop
item selection strategy
Item Selection Strategy
  • Shannon Entropy (SHE) procedure was applied to select next items
    • SHE (Tatsuoka, 2002, Xu, Chang, & Douglas, 2004, McGlohen & Chang, 2008)
  • Dual Information (McGlohen & Chang, 2004 and 2008) Cheng and Chang, 2007)
Parameters Estimation
    • The knowledge state of examinee is estimated sequentially.
    • The Maximum posterior estimation (MAPE) method was used in the system.
    • The ability is estimated at the end of the test.
monte carlo simulation studies
Monte Carlo simulation Studies
  • Item selection rule
    • Content constraints (same test structure as Pretest)
      • Listening Dialog (item1-item10), the next items is selected within remaining Listening Dialog items in the item bank.
      • Short Talks (item11-item12), two items for a piece of speech is selected within the short-talk items in the item bank.
      • Grammar and Vocabulary (item17-item32), the next items is selected within remaining Grammar and Vocabulary items in the item bank.
      • Reading Comprehension (item33-item40), the next items is selected within remaining Reading Comprehension items in the item bank.
  • Item selection strategy
    • the item was selected according to Shannon entropy procedure
classification accuracy evaluation criteria
Classification Accuracy & Evaluation Criteria
  • Evaluation criteria
    • Rate of pattern match (RPM)
    • Rate of marginal match (RMM)
    • average test information
field test
Field Test
  • SHE with content constraints
  • The adaptive test was web-based, consisting of 36 items and lasting for 40 minutes.
  • Number of Participants: 584
    • 5th and 6th grade, from 8 schools in Beijing, China
validity study
Validity Study
  • Evaluating the consistency of
    • CD-CAT system results with an existing English achievement test
      • a group of students took two exams
    • CD-CAT system results with Teachers’ evaluation outcomes.
cd scores vs scores of an achievement test
CD scores vs. scores of an achievement test

The Consistence between levels and # of mastered attributes

cd cat results vs teachers assessment
CD-CAT Results vs. Teachers' Assessment

Comparison of a CD scores with teachers’ assessment

Participants from three classes:

91 6-grade students and 3 teachers were recruited to evaluate the diagnostic reports. one rural school and two urban schools.


Students’ diagnostic reports were presented to three teachers, they were asked to evaluate the accuracy of this report.

validity study cd vs teachers
Validity Study: CD vs. Teachers

Evaluation on the CD-CAT feedback reports by teachers

  • Large scale field tests will take place in Shanghai and Dalian in the near future.
  • CD-CAT can be implemented effectively and economically.
  • Though the DINA model was used, the results can be generalized to many other IRT and Cognitive Diagnostic Models!
  • The method for on-line calibrating of pre-test items has been developed. In the future, paper/pencil based pretesting is not needed.
  • CAT is revolutionarily changing the way we address challenges in assessment and learning.
  • In June 2010 the IES proposal was revised and resubmitted.
  • Any good example of LARGE-SCALE CD-CAT?