Reliability or Validity. Reliability gets more attention: Easier to understand Easier to measure More formulas (like stats!) Base for validity. Need for validity. Does test measure what it claims? Can test be used to make decisions?. Validity.
Reliability gets more attention:
Reliability is a necessary, but not a sufficient condition for validity.
“A test is valid to the extent that inferences made from it are appropriate, meaningful, and useful”
Standards for Educational and Psychological Testing, 1999
“looks good to me”!!!!!!!
“How adequately a test samples behaviors representative of the universe of behaviors the test was designed to measure.”
Agreement among raters if item is:
“A theoretical intangible”
“An informed, scientific idea”
-- how well the test measures that construct
How well the test can be used to help in decision making about a particular criterion.
“The relationship between performance on the test and on some other criterion.”
Correlation between test score and score on criterion measure.
A) Concurrent validity
B) Predictive validity
“Validity is not a property of the test, but rather the meaning of the scores.”
Relevance and utility
Dr. Heidi considers using the Scranton Depression Inventory to help identify severity of depression and especially to distinguish depression from anxiety. What evidence should Dr. Heidi use to determine if the test does what she hopes it will do?
The newly published Diagnostic Wonder Test promises to identify children with a mathematics learning disability. How will we know whether the test does so or is simply a slickly packaged general ability test?
Ivy College uses the Western Admissions Test (WAT) to select applicants who should be successful in their studies. What type of evidence should we seek to determine if the WAT satisfies its purpose?
Mike is reviewing a narrative report of his scores on the Nifty Personality Questionnaire (NPQ). The report says he is exceptionally introverted and unusually curious about the world around him. Can Mike have any confidence in these statements or should they be dismissed as equivalent to palm readings at the county fair?
A school system wants to use an achievement battery that will measure the extent to which students are learning the curriculum specified by the school. How should the school system proceed in reviewing the available achievement tests?
Super sun computers needs to hire three new employees. They have decided to administer the Computer Skills Assessment (CSA) to their applicants and use the results as the basis of their decision. How can they determine if that measure is a good fit for their hiring practice?