reliability or validity l.
Skip this Video
Loading SlideShow in 5 Seconds..
Reliability or Validity PowerPoint Presentation
Download Presentation
Reliability or Validity

Loading in 2 Seconds...

play fullscreen
1 / 33

Reliability or Validity - PowerPoint PPT Presentation

  • Uploaded on

Reliability or Validity. Reliability gets more attention: Easier to understand Easier to measure More formulas (like stats!) Base for validity. Need for validity. Does test measure what it claims? Can test be used to make decisions?. Validity.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Reliability or Validity

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
reliability or validity
Reliability or Validity

Reliability gets more attention:

  • Easier to understand
  • Easier to measure
  • More formulas (like stats!)
  • Base for validity
need for validity
Need for validity
  • Does test measure what it claims?
  • Can test be used to make decisions?

Reliability is a necessary, but not a sufficient condition for validity.

validity a definition
Validity: a definition

“A test is valid to the extent that inferences made from it are appropriate, meaningful, and useful”

Standards for Educational and Psychological Testing, 1999

face validity
“Face Validity”

“looks good to me”!!!!!!!

trinitarian view of validity
Trinitarian view of Validity
  • Content (meaning)
  • Construct (meaning)
  • Criterion (use)
1 content validity
1) Content Validity

“How adequately a test samples behaviors representative of the universe of behaviors the test was designed to measure.”

determining content validity
Determining Content Validity
  • describe the domain
  • specify areas to be measured
  • compare test to domain
content validity ratio cvr
Content Validity Ratio (CVR)

Agreement among raters if item is:

  • Essential
  • Useful but not essential
  • Not necessary
2 construct validity
2) Construct validity

“A theoretical intangible”

“An informed, scientific idea”

-- how well the test measures that construct

determining construct validity
Determining Construct validity
  • behaviors related to constructs
  • related/unrelated constructs
  • identify relationships
  • multi trait/multi method
multitrait multimethod matrix
Multitrait-Multimethod Matrix
  • Correlate scores from 2 (or more tests)
  • Correlate scores obtained from 2 (or more) methods
evidence of construct validity
Evidence of Construct Validity
  • Upholds theoretical predictions
    • Changes (?) over time, gender, training
  • Homogeneity of questions
    • (internal consistency, factor or item analysis)
  • Convergent/discriminant
    • Multitrait-multimethod matrix
decision making
Decision Making

How well the test can be used to help in decision making about a particular criterion.

decision theory
Decision Theory
  • Base rate
  • Hit rate
  • Miss rate
  • False positive
  • False negative
3 criterion validity
3) Criterion Validity

“The relationship between performance on the test and on some other criterion.”

validity coefficient
Validity coefficient

Correlation between test score and score on criterion measure.

two ways to establish criterion validity
Two ways to establish Criterion Validity

A) Concurrent validity

B) Predictive validity

determining concurrent validity
Determining Concurrent validity
  • Assess individuals on construct
  • Administer test to lo/hi on construct
  • Correlate test scores to prior identification
  • Use test later to make decisions
determining predictive validity
Determining Predictive validity
  • Give test to group of people
  • Follow up group
  • Assess later
  • Review test scores
  • If correlate with behavior later can use later to make decisions
incremental validity
Incremental validity
  • Value of including more than one predictor
  • Based on multiple regression
  • What is added to prediction not present with previous measures?
expectancy data
Expectancy data
  • Taylor-Russell Table
  • Naylor-Shine Tables
  • Too vague, outdated, biased
unified validity messick
Unified Validity - Messick

“Validity is not a property of the test, but rather the meaning of the scores.”

Value implications

Relevance and utility

unitarian considerations
Unitarian considerations
  • Content
  • Construct
  • Criterion
  • Consequences
threats to validity
Threats to validity
  • Construct underrepresentation (too narrow)
  • Construct-irrelevant variance (too broad)

construct-irrelevant difficulty

construct-irrelevant easiness

example 1
Example 1

Dr. Heidi considers using the Scranton Depression Inventory to help identify severity of depression and especially to distinguish depression from anxiety. What evidence should Dr. Heidi use to determine if the test does what she hopes it will do?

example 2
Example 2

The newly published Diagnostic Wonder Test promises to identify children with a mathematics learning disability. How will we know whether the test does so or is simply a slickly packaged general ability test?

example 3
Example 3

Ivy College uses the Western Admissions Test (WTA) to select applicants who should be successful in their studies. What type of evidence should we seek to determine if the WAT satisfies its purpose?

example 4
Example 4

Mike is reviewing a narrative report of his scores on the Nifty Personality Questionnaire (NPQ). The report says he is exceptionally introverted and unusually curious about the world around him. Can Mike have any confidence in these statements or should they be dismissed as equivalent to palm readings at the county fair?

example 5
Example 5

A school system wants to use an achievement battery that will measure the extent to which students are learning the curriculum specified by the school. How should the school system proceed in reviewing the available achievement tests?

project homework question
Project homework question
  • What content or construct is your measure assessing? (explain your answer)
  • What do you think congruent and discriminate constructs would be to the one in your measure?
  • How would you determine the content or construct validity of your measure?
  • How would you determine the criterion validity of your measure?
  • Why would you use those approaches?
project homework question32
Project homework question
  • Select astandardized instrument from MMY to use as a comparison for your measure?
  • Copy the relevant data.
  • Why did you select that instrument?
  • How would you use it to help standardize your measure?