Test Validity – Revisited Again. Virginia Association of Test Directors Conference, Richmond, VA October 29, 2008, 1:30 – 2:30. David Mott Tests for Higher Standards ROS works, LLC. The title of this talk is deliberately repetitive. It’s like: “The Department of Redundancy Department”,
Virginia Association of Test Directors Conference, Richmond, VA October 29, 2008, 1:30 – 2:30
Tests for Higher Standards
“The Department of Redundancy Department”,
“Déjà vu, all over again”,
Because validity is seen as the most important quality that a test or assessment can have.
Discussions about test validity have been going on for years. One simple prepositional phrase needs to be added to make the ongoing discussion meaningful — “FOR . . .” Test scores can never be either valid or invalid unless the space after FOR is filled in. This is a presentation and a discussion of what needs to go after the FOR, within the world of AYP and NCLB, and why it matters. Data will be presented and there will be a test.
Began as a psychologist
Including Supervisor of Test Development
Testing Supervisor in charge of VSAP
This is the FOR in the abstract.
I will call this (modestly), “Mott’s Law of Validities.”
Concurrent validity refers to the degree to which the test scores correlate with other measures of the same underlying thing that are measured at the same time. For example, this would mean that the scores on tests administered to students should correlate with their grades. That scores on one reading test compare with scores on another reading test.
It can have a useful function for us.
Predictive validity is concurrent validity with a sense of time. How does one score predict another in the future. How will this bench-mark test predict scores on the SOL test?
This is what divisions ask my company and others to supply when they ask us for “the validity evidence” for our tests.
Problems, with this?
Convergent validity refers to the degree to which a measure is correlated with many other measures that it is theoretically predicted to correlate with. This is sort of concurrent validity on steroids.
Reading ability is the best example of this:
DRP units or Lexiles.
We have long had a hint of this because all decent reading tests correlate pretty highly with each other.
Face validity: If it looks like a duck, and walks like a duck, and quacks like a duck — It’s a . . .
It’s a non-statistical type of validity
Content validity is systematic examination of test content to determine whether it covers a good sample of the domain intended to be measured by a test. For standards-related tests: Does the test cover the all desired standards. Does it cover them evenly or in the proportion it should. Does it’s coverage not include other standards? How about matching the cognitive levels?
Diagnostic validity is the ability of a test to discriminate accurately between the skills, abilities, etc. an individual has and can do versus those not attained. The overall score is not really important, except as context.
Okay, I made up this term, but it is akin to what Jim Popham has called “instructional sensitivity”. His point is that many of our tests don’t have it.
Why is diagnostic validity of importance?
Because it is what most testing on the formative end of the spectrum is used for.
This type is not well described statistically. There are no well-known statistics used to measure it.
And there is always questions about how specific we should be or can be in our diagnoses:
Total Test Score
Some common reasons for testing:
Arguments about what is formative vs. summative go on and on.
However, most of the research which shows formative assessment is highly effective in learning is based on assessment that is deeply integrated into learning. These are techniques used by teachers that are used in the midst of teaching. The teacher and the student are in a very short feedback loop.
“ . . . evidence gathered from dozens of studies conducted around the world consistently reveals a half to a full standard deviation gain in student achievement attributable to the careful management of the classroom assessment process, with the largest gains accruing for struggling learners.” (Black and Wiliam, 1998; Hattie and Timperley, 2007).
This really does not mean benchmark tests, mid-semester exams, end-of-course tests and the like. It may not even mean teacher made pop quizzes and so on.
It does mean tests in which the student finds out what he does and doesn’t know and can use the information, together with his teacher, to drive his own learning.
This is usually not the stated purpose for assessment, but both the good things and the bad things that come out of testing often relate to this function.
It is the “stakes” part of the high-stakes vs. low-stakes dichotomy.
High-stakes for whom, is a good question to ask.
Stiggins make a number of points about motivation in his Manifesto.
I say we because I mean me, my company, you, your division, your teachers, all of us.
Let’s look at this last, because this started me off on this tack.
Try A different method of scoring:
Tell the students that they get 1 point for every answer they get right, 1/3 of a point for every answer they omit, and 0 points for every answer they get wrong. This removes the incentive for guessing and it lets you know explicitly that students need help.
Discussion . . .
So it comes full circle. Assessment is about learning. Assessment, used correctly, is really all FOR learning.
The test that follows this session is really a test of us.
Stiggins, R. (2008). Assessment Manifesto: A call for the Development of Balanced Assessment Systems. Portland, OR: ETS Assessment Training Institute. [www.ets.org/ati]
Popham, W. J. (2008). All about Assessment: A Misunderstood Grail. Educational Leadership, 66(1), 82-83.
Popham, W. J. (2008). Transformative Assessment. Alexandria, VA: Association for Supervision and Curriculum Development.
Wise, S. L. and DeMars, C. E. at the Center for Assessment & Research Studies (CARS) in the JMU School of Education — Response-time studies.
5310 Markel Road, Suite 104
Richmond, VA 23230