Educational research
Download
1 / 37

Educational Research - PowerPoint PPT Presentation


  • 113 Views
  • Uploaded on

Educational Research. Chapter 5 Selecting Measuring Instruments Gay, Mills, and Airasian. Topics Discussed in this Chapter. Data collection Measuring instruments Terminology Interpreting data Types of instruments Technical issues Validity Reliability Selection of a test.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Educational Research' - azuka


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Educational research l.jpg

Educational Research

Chapter 5

Selecting Measuring Instruments

Gay, Mills, and Airasian


Topics discussed in this chapter l.jpg
Topics Discussed in this Chapter

  • Data collection

  • Measuring instruments

    • Terminology

    • Interpreting data

    • Types of instruments

  • Technical issues

    • Validity

    • Reliability

  • Selection of a test


Data collection l.jpg
Data Collection

  • Scientific inquiry requires the collection, analysis, and interpretation of data

    • Data – the pieces of information that are collected to examine the research topic

    • Issues related to the collection of this information are the focus of this chapter


Data collection4 l.jpg
Data Collection

  • Terminology related to data

    • Constructs – abstractions that cannot be observed directly but are helpful when trying to explain behavior

      • Intelligence

      • Teacher effectiveness

      • Self concept

Obj. 1.1 & 1.2


Data collection5 l.jpg
Data Collection

  • Data terminology (continued)

    • Operational definition – the ways by which constructs are observed and measured

      • Weschler IQ test

      • Virgilio Teacher Effectiveness Inventory

      • Tennessee Self-Concept Scale

    • Variable – a construct that has been operationalized and has two or more values

Obj. 1.1 & 1.2


Data collection6 l.jpg
Data Collection

  • Measurement scales

    • Nominal – categories

      • Gender, ethnicity, etc.

    • Ordinal – ordered categories

      • Rank in class, order of finish, etc.

    • Interval – equal intervals

      • Test scores, attitude scores, etc.

    • Ratio – absolute zero

      • Time, height, weight, etc.

Obj. 2.1


Data collection7 l.jpg
Data Collection

  • Types of variables

    • Categorical or quantitative

      • Categorical variables reflect nominal scales and measure the presence of different qualities (e.g., gender, ethnicity, etc.)

      • Quantitative variables reflect ordinal, interval, or ratio scales and measure different quantities of a variable (e.g., test scores, self-esteem scores, etc.)

Obj. 2.2


Data collection8 l.jpg
Data Collection

  • Types of variables

    • Independent or dependent

      • Independent variables are purported causes

      • Dependent variables are purported effects

      • Two instructional strategies, co-operative groups and traditional lectures, were used during a three week social studies unit. Students’ exam scores were analyzed for differences between the groups.

        • The independent variable is the instructional approach (of which there are two levels)

        • The dependent variable is the students’ achievement

Obj. 2.3


Measurement instruments l.jpg
Measurement Instruments

  • Important terms

    • Instrument – a tool used to collect data

    • Test – a formal, systematic procedure for gathering information

    • Assessment – the general process of collecting, synthesizing, and interpreting information

    • Measurement – the process of quantifying or scoring a subject’s performance

Obj. 3.1 & 3.2


Measurement instruments10 l.jpg
Measurement Instruments

  • Important terms (continued)

    • Cognitive tests – examining subjects’ thoughts and thought processes

    • Affective tests – examining subjects’ feelings, interests, attitudes, beliefs, etc.

    • Standardized tests – tests that are administered, scored, and interpreted in a consistent manner

Obj. 3.1


Measurement instruments11 l.jpg
Measurement Instruments

  • Important terms (continued)

    • Selected response item format – respondents select answers from a set of alternatives

      • Multiple choice

      • True-false

      • Matching

    • Supply response item format – respondents construct answers

      • Short answer

      • Completion

      • Essay

Obj. 3.3 & 11.3


Measurement instruments12 l.jpg
Measurement Instruments

  • Important terms (continued)

    • Individual tests – tests administered on an individual basis

    • Group tests – tests administered to a group of subjects at the same time

    • Performance assessments – assessments that focus on processes or products that have been created

Obj. 3.6


Measurement instruments13 l.jpg
Measurement Instruments

  • Interpreting data

    • Raw scores – the actual score made on a test

    • Standard scores – statistical transformations of raw scores

      • Percentiles (0.00 – 99.9)

      • Stanines (1 – 9)

      • Normal Curve Equivalents (0.00 – 99.99)

Obj. 3.4


Measurement instruments14 l.jpg
Measurement Instruments

  • Interpreting data (continued)

    • Norm-referenced – scores are interpreted relative to the scores of others taking the test

    • Criterion-referenced – scores are interpreted relative to a predetermined level of performance

    • Self-referenced – scores are interpreted relative to changes over time

Obj. 3.5


Measurement instruments15 l.jpg
Measurement Instruments

  • Types of instruments

    • Cognitive – measuring intellectual processes such as thinking, memorizing, problem solving, analyzing, or reasoning

    • Achievement – measuring what students already know

    • Aptitude – measuring general mental ability, usually for predicting future performance

Obj. 4.1 & 4.2


Measurement instruments16 l.jpg
Measurement Instruments

  • Types of instruments (continued)

    • Affective – assessing individuals’ feelings, values, attitudes, beliefs, etc.

      • Typical affective characteristics of interest

        • Values – deeply held beliefs about ideas, persons, or objects

        • Attitudes – dispositions that are favorable or unfavorable toward things

        • Interests – inclinations to seek out or participate in particular activities, objects, ideas, etc.

        • Personality – characteristics that represent a person’s typical behaviors

Obj. 4.1 & 4.5


Measurement instruments17 l.jpg
Measurement Instruments

  • Types of instruments (continued)

    • Affective (continued)

      • Scales used for responding to items on affective tests

        • Likert

          • Positive or negative statements to which subjects respond on scales such as strongly disagree, disagree, neutral, agree, or strongly agree

        • Semantic differential

          • Bipolar adjectives (i.e., two opposite adjectives) with a scale between each adjective

          • Dislike: ___ ___ ___ ___ ___ :Like

        • Rating scales – rankings based on how a subject would rate the trait of interest

Obj. 5.1


Measurement instruments18 l.jpg
Measurement Instruments

  • Types of instruments (continued)

    • Affective (continued)

      • Scales used for responding to items on affective tests (continued)

        • Thurstone – statements related to the trait of interest to which subjects agree or disagree

        • Guttman – statements representing a uni-dimensional trait

Obj. 5.1


Measurement instruments19 l.jpg
Measurement Instruments

  • Issues for cognitive, aptitude, or affective tests

    • Problems inherent in the use of self-report measures

      • Bias – distortions of a respondent’s performance or responses based on ethnicity, race, gender, language, etc.

      • Responses to affective test items

        • Socially acceptable responses

        • Accuracy of responses

        • Response sets

    • Alternatives include the use of projective tests

Obj. 4.3, 4.4


Technical issues l.jpg
Technical Issues

  • Two concerns

    • Validity

    • Reliability


Technical issues21 l.jpg
Technical Issues

  • Validity – extent to which interpretations made from a test score are appropriate

    • Characteristics

      • The most important technical characteristic

      • Situation specific

      • Does not refer to the instrument but to the interpretations of scores on the instrument

      • Best thought of in terms of degree

Obj. 6.1 & 7.1


Technical issues22 l.jpg
Technical Issues

  • Validity (continued)

    • Four types

      • Content – to what extent does the test measure what it is supposed to measure

        • Item validity

        • Sampling validity

        • Determined by expert judgment

Obj. 7.1 & 7.2


Technical issues23 l.jpg
Technical Issues

  • Validity (continued)

    • Criterion-related

      • Predictive – to what extent does the test predict a future performance

      • Concurrent - to what extent does the test predict a performance measured at the same time

      • Estimated by correlations between two tests

    • Construct – the extent to which a test measures the construct it represents

      • Underlying difficulty defining constructs

      • Estimated in many ways

Obj. 7.1, 7.3, & 7.4


Technical issues24 l.jpg
Technical Issues

  • Validity (continued)

    • Consequential – to what extent are the consequences that occur from the test harmful

      • Estimated by empirical and expert judgment

  • Factors affecting validity

    • Unclear test directions

    • Confusing and ambiguous test items

    • Vocabulary that is too difficult for test takers

Obj. 7.1, 7.5, & 7.7


Technical issues25 l.jpg
Technical Issues

  • Factors affecting validity (continued)

    • Overly difficult and complex sentence structure

    • Inconsistent and subjective scoring

    • Untaught items

    • Failure to follow standardized administration procedures

    • Cheating by the participants or someone teaching to the test items

Obj. 7.7


Technical issues26 l.jpg
Technical Issues

  • Reliability – the degree to which a test consistently measures whatever it is measuring

    • Characteristics

      • Expressed as a coefficient ranging from 0 to 1

      • A necessary but not sufficient characteristic of a test

Obj. 6.1, 8.1, & 8.7


Technical issues27 l.jpg
Technical Issues

  • Reliability (continued)

    • Six reliability coefficients

      • Stability – consistency over time with the same instrument

        • Test – retest

        • Estimated by a correlation between the two administrations of the same test

      • Equivalence – consistency with two parallel tests administered at the same time

        • Parallel forms

        • Estimated by a correlation between the parallel tests

Obj. 8.1, 8.2, 8.3, & 8.7


Technical issues28 l.jpg
Technical Issues

  • Reliability (continued)

    • Six reliability coefficients (continued)

      • Equivalence and stability – consistency over time with parallel forms of the test

        • Combines attributes of stability and equivalence

        • Estimated by a correlation between the parallel forms

      • Internal consistency – artificially splitting the test into halves

        • Several coefficients – split halves, KR 20, KR 21, Cronbach alpha

        • All coefficients provide estimates ranging from 0 to 1

Obj. 8.1, 8.4, 8.5, & 8.7


Technical issues29 l.jpg
Technical Issues

  • Reliability (continued)

    • Six reliability coefficients

      • Scorer/rater – consistency of observations between raters

        • Inter-judge – two observers

        • Intra-judge – one judge over two occasions

        • Estimated by percent agreement between observations

Obj. 8.1, 8.6, & 8.7


Technical issues30 l.jpg
Technical Issues

  • Reliability (continued)

    • Six reliability coefficients (continued)

      • Standard error of measurement (SEM) – an estimate of how much difference there is between a person’s obtained score and his or her true score

        • Function of the variation of the test and the reliability coefficient (e.g., KR 20, Cronbach alpha, etc.)

        • Estimated by specifying an interval rather than a point estimate of a person’s score

Obj. 8.1, 8.7, & 9.1


Selection of a test l.jpg
Selection of a Test

  • Sources of test information

    • Mental Measurement Yearbooks (MMY)

      • The reviews in MMY are most easily accessed through your university library and the services to which they subscribe (e.g., EBSCO)

      • Provides factual information on all known tests

      • Provides objective test reviews

      • Comprehensive bibliography for specific tests

      • Indices: titles, acronyms, subject, publishers, developers

      • Buros Institute

Obj. 10.1 & 12.1


Selection of a test32 l.jpg
Selection of a Test

  • Sources (continued)

    • Tests in Print

      • Tests in Print is a subsidiary of the Buros Institute

      • The reviews in it are most easily accessed through your university library and the services to which they subscribe (e.g., EBSCO)

      • Bibliography of all known commercially produced tests currently available

      • Very useful to determine availability

      • Tests in Print

Obj. 10.1 & 12.1


Selection of a test33 l.jpg
Selection of a Test

  • Sources (continued)

    • ETS Test Collection

      • Published and unpublished tests

      • Includes test title, author, publication date, target population, publisher, and description of purpose

      • Annotated bibliographies on achievement, aptitude, attitude and interests, personality, sensory motor, special populations, vocational/occupational, and miscellaneous

      • ETS Test Collection

Obj. 10.1 &12.1


Selection of a test34 l.jpg
Selection of a Test

  • Sources (continued)

    • Professional journals

    • Test publishers and distributors

  • Issues to consider when selecting tests

    • Psychometric properties

      • Validity

      • Reliability

      • Length of test

      • Scoring and score interpretation

Obj. 10.1, 11.1, & 12.1


Selection of a test35 l.jpg
Selection of a Test

  • Issues to consider when selecting tests

    • Non-psychometric issues

      • Cost

      • Administrative time

      • Objections to content by parents or others

      • Duplication of testing

Obj. 11.1


Selection of a test36 l.jpg
Selection of a Test

  • Designing your own tests

    • Get help from others with experience in developing tests

    • Item writing guidelines

      • Avoid ambiguous and confusing wording and sentence structure

      • Use appropriate vocabulary

      • Write items that have only one correct answer

      • Give information about the nature of the desired answer

      • Do not provide clues to the correct answer

      • See Writing Multiple Choice Items

Obj. 11.2


Selection of a test37 l.jpg
Selection of a Test

  • Test administration guidelines

    • Plan ahead

    • Be certain that there is consistency across testing sessions

    • Be familiar with any and all procedures necessary to administer a test

Obj. 11.4