Test Construction and Measurement - PowerPoint PPT Presentation

Test Construction and Measurement

• Researcher gave students the Diagnostic Inventory Blank
• Hobbies, reading interests, secret hopes and ambitions
• Then gave students typed descriptions of their personalities
• Asked students to rate how well personality sketch described them

• You have a need for other people to like and admire you
• You have a tendency to be critical of yourself
• While you have some personality weaknesses, you are generally able to compensate for them

• You pride yourself as an independent thinker and do not accept others’ statements without proof
• At times you are extraverted and sociable, while at other times you are introverted and reserved

• Almost all students were very impressed with how well DIB described them
• Rated DIB as very accurate personality test

• Every student was given exactly the same personality description

• Beware of the Barnum effect
• Tendency of people to see vague, universal statements as descriptive of themselves

• Real psychological measurement is a complicated and difficult process

• Correlation
• Steps in constructing a psychological test
• Reliability and validity
• Factor analysis

• Focuses on relationships among variables
• Changes in one variable are associated with changes in another variable

• Number which expresses the direction and strength of the relationship between 2 variables
• Ranges from -1 to 1
• Index of the degree to which scores on one measure can be used to predict scores on a 2nd measure

• Indicated by + or - sign (slope)
• Positive correlation
• as one variable goes up, so does the other
• Negative correlation
• As one variable goes up, the other goes down

• Indicated by absolute value
• perfect positive relationship = 1
• perfect negative relationship = -1
• no relationship = 0

• Percent of variance in “measure A” that can be accounted for “measure B”
• square correlation coefficient and multiply by 100
• Correlation of .50 means we can account for 25% of variance

• Correlation just tells you that 2 variables are related
• Can’t make causal interpretations

• Possible interpretations
• Spending lots of time on internet causes depression
• Being depressed causes you to spend lots of time on internet
• Some third variable, such as living by one’s self, causes both

Steps in Psychological Test Construction

• It is difficult, but not impossible, to construct a meaningful psychological test

1. Decide what to measure

• Identify construct
• Idea that helps us makes sense of world around us
• Not directly observable
• Examples: intelligence, extraversion, racism, pessimism, creativity

2. Develop a set of items/questions

• Search literature
• Get experts or lay people to tell us what construct means to them

3. Get sample of people to answer items

• From population you want to use test for

4. Evaluate each item

• Correlate each item with mean of whole set
• Correlate each item with item directly assessing self-reported racism
• drop bad items

5. Select a set of items for further study

• Want normal distribution
• Drop high YES and high NO items

6. Assess reliability of entire test

• Consistency of measurement
• 3 major types

1. Inter-rater:

• Extent to which different people scoring same test get same result
• Correlate set of tests scored by one rater with same set of tests scored by different rater

2. Test-retest:

• Extent to which people get same results if take test again
• Subjects take test twice. Correlate set of time 1 scores with time 2 scores

3. Internal consistency:

• Split-half: correlation between one half of test and other half
• Coefficient alpha: average of all possible split-half reliabilities

• Expect r = .80 or better

• Clarity of items
• Motivation of test taker
• Number of items

7. Assess validity of entire test

• Extent to which test measures what it is supposed to measure
• Face validity not sufficient
• Do series of validity studies

1. Criterion validity

• Correlation between test and concrete, directly observable criterion
• Example: correlate self-report of weight with actual weight on scale

2. Content validity

• Adequate coverage of target domain
• Example: test of chapters 1-4 which only covers chapter 2 and 3 lacks content validity

3. Convergent validity

• Agreement among alternative measures of same construct
• Example: correlation between ACT and SAT

4. Discriminant validity

• Lack of correlation between tests that are intended to measure different constructs
• Example: expect low correlation between ACT and test of aggression

Response tendency

• Assign numbers to items for reasons that have little to do with the construct the item is intended to measure

• Extremity tendency
• Use end of scales
• Acquiescence tendency
• Agree with questions
• Social desirability
• Answer in a way that makes you look good

• Statistical technique that examines pattern of correlations among multiple tests or items
• Tests or items that correlate strongly with one another are considered to represent a common, underlying factor

• Each item has a factor loading: correlation between item and factor
• Marker variable
• item that has high factor loading (correlation) with given factor
• closely related to meaning of factor
• Blend
• item that loads moderately high on more than one factor
• not a pure measure of factor, related to two or more factors