1 / 29

Measurement and Scaling

Measurement and Scaling. Farzin Madjidi, Ed.D. Pepperdine University Graduate School of Education and Psychology. Variables. Independent Precedes, influences or predicts results Dependent Affected by or predicted by the Independent Variable Extraneous

alvaro
Download Presentation

Measurement and Scaling

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Measurement and Scaling Farzin Madjidi, Ed.D. Pepperdine University Graduate School of Education and Psychology

  2. Variables • Independent • Precedes, influences or predicts results • Dependent • Affected by or predicted by the Independent Variable • Extraneous • Affected by the D.V., but not controlled or measured. Causes error

  3. Variables • Confounding • An extraneous variable that varies systematically (has a relationship) with the I.V. • Intervening • Unobservable trait that influences behavior (e.g., effect of new intervention on self-esteem may be affected by the motivation level of subjects)

  4. Variables • Control • Used to eliminate the effect of extraneous variables • Organismic • Aka, measured, or assigned • Characteristics of the subjects that cannot be manipulated

  5. Levels of Measurements Four levels of Measurements • Nominal • Measures categories • Ordinal • Categories + rank and order • Interval • Equal distance between any two consecutive measures • Ratio • Intervals + meaningful zeros

  6. Categories of Scales • Categorical (ratings) • Score without comparison - 1 to 5 scales • Comparative (ranking) • Score by comparing - Smartest • Preference • Subjective - which do you prefer • Non-preference • Objective - which solution is less costly

  7. Categories of Scales • Unidimensional • Involves only one aspect of the measurement • Measurement by one construct • Multi-dimensional • Involves several aspects of a measurement • Uses several dimensions to measure a single construct

  8. Types of Scales • Likert/Summated Rating Scales • Semantic Differential Scales • Magnitude Scaling • Thruston Scales • Guttman Scales

  9. Likert Scales • A very popular rating scale • Measures the feelings/degree of agreement of the respondents • Ideally, 4 to 7 points • Examples of 5-point surveys • Agreement SD D ND/NA A SA • Satisfaction SD D ND/NS S SS • Quality VP P Average G VG

  10. Summative Ratings • A number of items collectively measure one construct (Job Satisfaction) • A number of items collectively measure a dimension of a construct and a collection of dimensions will measure the construct (Self-esteem)

  11. Summative Likert Scales • Must contain multiple items • Each individual item must measure something that has an underlying, quantitative measurement continuum • There can be no right/wrong answers as opposed to multiple-choice questions • Items must be statements to which the respondent assigns a rating • Cannot be used to measure knowledge or ability, but familiarity

  12. Semantic Differential Scales • Uses a set of scale anchored by their extreme responses using words of opposite meaning. • Example: Dark ___ ___ ___ ___ ___ Light Short ___ ___ ___ ___ ___ Tall Evil ___ ___ ___ ___ ___ Good • Four to seven categories are ideal

  13. Magnitude Scaling • Attempts to measure constructs along a numerical, ratio level scale • Respondent is given an item with a pre-assigned numerical value attached to it to establish a “norm” • The respondent is asked to rate other items with numerical values as a proportion of the “norm” • Very powerful if reliability is established

  14. Thurston Scales • Thurston Scales • Items are formed • Panel of experts assigns values from 1 to 11 to each item • Mean or median scores are calculated for each item • Select statements evenly spread across the scale

  15. Thurston Scales • Example: Please check the item that best describes your level of willingness to try new tasks • I seldom feel willing to take on new tasks (1.7) • I will occasionally try new tasks (3.6) • I look forward to new tasks (6.9) • I am excited to try new tasks (9.8)

  16. Guttman Scales • Also known as Scalograms • Both the respondents and items are ranked • Cutting points are determined (Goodenough-Edwards technique) • Coefficient of Reproducibility (CReg) - a measure of goodness of fit between the observed and predicted ideal response patterns • Keep items with CReg of 0.90 or higher

  17. Scale Construction • Define Constructs • Conceptual/theoretical basis from the literature • Are their sub-scales (dimensions) to the scale • Multiple item sub-scales • Principle of Parsimony • Simplest explanation among a number of equally valid explanations must be used

  18. Item Construction • Agreement items • Write declarative statements • Death penalty should be abolished • I like to listen to classical music • Frequency items (how often) • I like to read • Evaluation items • How well did your team play • How well does the police serve your community

  19. Item Writing • Mutually exclusive and collectively exhaustive items • Use positively and negatively phrased questions • Avoid colloquialism, expressions and jargon • Avoid the use of negatives to reverse the wording of an item • Don’t use: I am not satisfied with my job • Use: I hate my job! • Be brief, focused, and clear • Use simple, unbiased questions

  20. Sources of Error • Social desirability • Giving politically correct answers • Response sets • All yes, or all no responses • Acquiescence • Telling you what you want to hear • Personal bias • Wants to send a message

  21. Sources of Error • Response order • Recency - Respondent stops reading once s/he gets to the response s/he likes • Primacy - Remember better the initial choices • Fatigue • Item order • Answers to later items may be affected by earlier items (simple, factual items first) • Respondent may not know how to answer earlier questions

  22. Assessing Instruments • Three issues to consider • Validity: Does the instrument measure what its supposed to measure • Reliability: Does it consistently repeat the same measurement • Practicality: Is this a practical instrument

  23. Types of Validity • Face validity • Does the instrument, on its face, appear to measure what it is supposed to measure • Content validity • Degree to which the content of the items adequately represent the universe of all relevant items under study • Generally arrived at through a panel of experts

  24. Types of Validity • Criterion related • Degree to which the predictor is adequate in capturing the relevant aspects of criterion • Uses Correlation analysis • Concurrent validity • Criterion data is available at the same time as predictor score- requires high correlation between the two • Predictive validity • Criterion is measured after the passage of time • Retrospective look at the validity of the measurement • Known-groups

  25. Types of Validity • Construct Validity • Measures what accounts for the variance • Attempts to identify the underlying constructs • Techniques used: • Correlation of proposed test with other existing tests • Factor analysis • Multi-trait-multimethod analysis • Convergent validity - Calls for high correlation between the different measures of the same construct • Discriminant validity - Calls for low correlation between sub-scales within a construct

  26. Types of Reliability • Stability • Test-retest: Same test is administered twice to the same subjects over a short interval (3 weeks to 6 months) • Look for high correlation between the test and retest • Situational factors must be minimized

  27. Types of Reliability • Equivalence • Degree to which alternative forms of the same measure produce same or similar results • Give parallel forms of the same test to the same group with a short delay to avoid fatigue • Look for high correlation between the scores of the two forms of the test • Inter-rater reliability

  28. Types of Reliability • Internal Consistency • Degree to which instrument items are homogeneous and reflect the same underlying constructs • Split-half testing where the test is split into two halves that contain the same types of questions • Uses Cronbach’s alpha to determine internal consistency. Only one administration of the test is required • Kuder-Richardson (KR20) for items with right and wrong answers

  29. Practicality • Is the survey economical • Cost of producing and administering the survey • Time requirement • Common sense! • Convenience • Adequacy of instructions • Easy to administer • Can the measurement be interpreted by others • Scoring keys • Evidence of validity and reliability • Established norms

More Related