1 / 63

Testing 05

Testing 05. Reliability. Errors & Reliability. Errors in the test cause unreliability. The fewer the errors, the more reliable the test Sources of errors: Obvious: poor health, fatigue, lack of interest Less obvious: facets discussed in Fig. 5.3. Reliability & Validity.

basil-nash
Download Presentation

Testing 05

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Testing 05 Reliability

  2. Errors & Reliability • Errors in the test cause unreliability. • The fewer the errors, the more reliable the test • Sources of errors: • Obvious: poor health, fatigue, lack of interest • Less obvious: facets discussed in Fig. 5.3

  3. Reliability & Validity • Reliability is a necessary condition for validity. • Reliability & validity are complementary aspects of the measurement. • Reliability: How much of the performance is due to measurement errors, or to factors other than the language ability we want to measure. • Validity: How much of the performance is due to the language ability we want to measure.

  4. Reliability Measurement • Reliability measurement includes: logical analysis and empirical research, i.e. identify sources of errors and estimate the magnitude of their effects on the scores.

  5. Logical Analysis • Example of identification of source of errors: • Topic in an oral interview: business negotiation • Source of error: if we want to measure the test taker’s ability of general topics. • Indicator of the ability: if we want to the test taker’s ability of business English.

  6. Empirical Research • Procedures are usually complex. • Three kinds of theories • Classical true score theory (CTS) • Generalizability theory (G-Theory) • Item Response Theory (IRT)

  7. Factors on Test Scores • Characteristics of factors • general vs. specific • lasting vs. temporary • systematic vs. unsystematic

  8. Factors that affect language test scores

  9. Variance & Standard Deviation • s: standard deviation of the sample • σ: standard deviation of the population • s2: variance of the sample • σ2: variance of the population • s=√∑(X-Xˉ)2/n-1 • where • X: individual score • Xˉ: mean score • n: number of students

  10. Correlation Coefficient (相关系数) • Covariance (COV): two variables, X and Y, vary together. • COV(X,Y)=1/(n-1)∑(Xi－Xˉ)(Yi－Yˉ) • Correlation Coefficient (Pearson Product-moment Correlation Coefficient 皮尔逊积差相关系数) • r(x,y)=COV(x,y)/sxsy • r(x,y)= 1/(n-1)∑(Xi－Xˉ)(Yi－Yˉ)/ sxsy

  11. Correlation Coefficient • Where • n: number of items • Xi: individual score of the first half • Xˉ: mean of the scores in the first half • Yi: individual score of the second half • Yˉ: mean of the scores of the second half • sx: standard deviation of the first half • sy: standard deviation of the second half

  12. Calculation of Correlation Coefficient • Manually • Manual + Excel • Excel

  13. Classical True Score Theory • also referred to as the classical reliability theory because its major task is to estimate the reliability of the observed scores of a test. That is, it attempts to estimate the strength of the relationship between the observed score and the true score. • sometimes referred to as the true score theory because its theoretical derivations are based on a mathematical model known as the true score model

  14. Assumptions in CTS • Assumption 1: The observed score consists of the true score and the error score, i..e. x=xt+xe • Assumption 2: Error scores are unsystematic, random and uncorrelated to the true score, i.e. s2=st2+se2

  15. Parallel Test • Two tests are parallel if xˉ=x’ˉ sx2=sx’ˉ2 rxy=rx’y

  16. Correlation Between Parallel Tests • If the observed scores on two parallel tests are highly correlated, the effects of the error scores are minimal. • Reliability is the correlation between the observed scores of two parallel tests. • The definition is the basis for all estimates of reliability within CTS theory. • Condition: the observed scores on the two tests are experimentally independent.

  17. Error Score Estimation and Measurement • Relations between reliability, true score and error score: • The higher the portion of the true score, the higher the correlation of the two parallel tests. (True scores are systematic) • The higher the portion of the error score, the lower the correlation of the two parallel tests. (Error scores are random)

  18. Error Score Estimation and Measurement • rxx’=st2/se2 • (st2+se2)/sx2=1 • se2/ sx2=1- st2/ sx2 • st2/ sx2= rxx’ • se2/ sx2=1- rxx’ • se2=(1- rxx’)/ sx2

  19. Approaches to Estimate Reliability • Three approaches based on different sources of errors. • Internal consistency: source of errors from within the test and scoring procedure • Stability: How consistent test scores are over time. • Equivalence: Scores on alternative forms of tests are equivalent.

  20. Internal Consistency • Dichotomous Split-half reliability estimates The Spearman-Brown split-half estimate The Guttman split-half estimate Kuder-Richardson reliability coefficients • Non-dichotomous Coefficient alpha Rater consistency

  21. Split-half Reliability Estimates • Split the test into two halves which have equal means and variances (equivalence) and are independent of each other (independence). • 1.divide the test into the first and second halves. • 2.random halves • 3.odd-even method

  22. Spearman-Brown Reliability Estimate • rxx’=2rhh’/(1+rhh’) • where: • rhh’: correlation between the two halves of the test • Procedure: • 1.Divide the test into two equal halves • 2.Calculate the correlation coefficient between the two halves • 3.Calculate the Spearman-Brown reliability estimate

  23. Guttman Split-Half Estimate • rxx’=2(1－(sh12+sh22)/sx2) • where • sh12: variance of the first half • sh22: variance of the second half • sx2: variance of the total scores

  24. Kuder-Richardson Formula 20 • rxx’=k/(k－1)(1－∑pq/sx2) • where • k: number of items on the test • p: proportion of the correct answers, i.e. correct answers/total answers (difficulty) • q: proportion of the incorrect answers, i.e. 1-p • sx2: total test score variance

  25. Kuder-Richardson Formula 21 • rxx’=(ksx2－xˉ(k－xˉ))/(k－1)sx2 • where • k: number of items on the test • sx2: total test score variance • xˉ: mean score

  26. Coefficient alpha • α=k/(k－1)(1－∑si2/sx2) • where • k: number of items on the test • ∑si2 : sum of the variances of the different parts of the test • sx2: variance of the test scores

  27. Assumption Effect if assumption is violated Estimate Equivalence Independence Equivalence Independence Spearman-Brown + + underestimate overestimate Guttman － + overestimate K-C + + underestimate overestimate Coefficientα － － － － Comparison of Estimates: Assumptions

  28. Summary: Estimate Procedure • Spearman-Brown • 1. split • 2. variances of each half • 3. correlation coefficient of each half • 4. reliability coefficient

  29. Summary: Estimate Procedure • Guttman • 1. split • 2. variances of each half • 3. variance of the whole test • 4. reliability coefficient

  30. Summary: Estimate Procedure • K-C 20 • 1. number of questions • 2. proportion of correct answers of each question • 3. proportion of incorrect answers of each question • 4. sum of the product of p and q • 5. variance of the whole test • 6. reliability coefficient

  31. Summary: Estimate Procedure • K-C 21 • 1. number of questions • 2. mean of the test • 3. variance of the test • 4. reliability coefficient

  32. Summary: Estimate Procedure • Coefficientα • 1. number of the parts of the test • 2. mean of each part • 3. variance of each part • 4. sum of variances of all parts • 5. mean of the test • 6. variance of the test • 7. reliability coefficient

  33. Rater Consistency • Intra-rater • Inter-rater

  34. Intra-rater Reliability • Rate each paper twice. Condition: the two ratings must be independent of each other. • Two ways of estimating: • Spearman-Brown: Take each rating as a split half and compute the reliability coefficient.

  35. Intra-rater Reliability • Conditions: the two ratings must have the similar means and variances to ensure the equivalence of the two ratings • Coefficient alpha: Take two ratings as two parts of a test. • α=(k/(k－1))(1－(sx12+sx22)/sx1+x22)

  36. Intra-rater Reliability • where • k: number of ratings • sx12: variance of the first rating • sx22: variance of the second rating • sx1+x22: variance of the summed ratings • Since k=2, the formula can be reduced to the Guttman Reliability Coefficient Formula.

  37. Inter-rater Reliability • If there are only two raters, use split-half estimates to obtain the reliability coefficient. • Or Grade Correlation Coefficient: • rxx’=1－6∑D2/(n(n2－1)) • where • D: difference between the grades of the two ratings

  38. Inter-rater Reliability • n: number of the test takers • See testing 05-2 sheet 5 for example • Note: the same grade should be shared. • If there are more than two raters, use Coefficient alpha estimate

  39. Stability (test-retest reliability) • Administer the test twice to a group of individuals and compute the correlation between the two set of scores. The correlation can then be interpreted as an indicator of how stable the scores are over time. • Learning effects and practice effects must be taken into account.

  40. Equivalence (parallel forms reliability) • Use alternative forms of a given test. Compute and compare the means and standard deviations of for each of the two forms to determine their equivalence. The correlation between the two sets can be interpreted as an indicator of the equivalence of the two tests or an estimate of the reliability of either one.

  41. GENERALIZABILITY THEORY

  42. GENERALIZABILITY THEORY • Generalizability theory (G-theory) is a framework of factorial design and the analysis of variance. It constitutes a theory and set of procedures for specifying and estimating the relative effects of different factors on observed test scores, and thus provides a means for relating the uses or interpretations to the way test users specify and interpret different factors as either abilities or sources of error.

  43. GENERALIZABILITY THEORY • G-theory treats a given measure or score as a sample from a hypothetical universe of possible measures, i.e. on the basis of an individual's performance on a test we generalize to his performance in other contexts. • Reliability = generalizability • The way we define a given universe of measures will depend upon the universe of generalization

  44. Application of G-theory • Two stages: • G-study • D-study

  45. G-study • consider the uses that will be made of the test scores, investigate the sources of variance that are of concern or interest.On the basis of this generalizability study, the test developer obtains estimates of the relative sizes of the different sources of variance ('variance components').

  46. D-study • When the results of the G-study are satisfactory, the test developer administers the test under operational conditions, and uses G-theory procedures to estimate the magnitude of the variance components. These estimates provide information that can inform the interpretation and use of the test scores.

  47. Significance of G-theory • The application of G-Theory thus enables test developers and test users to specify the different sources of variance that are of concern for a given test use, to estimate the relative importance of these different sources simultaneously, and to employ these estimates in the interpretation and use of test scores.

  48. Universes Of Generalization And Universe Of Measures • universe of generalization, a domain of uses or abilities (or both) • the universe of possible measures: types of test scores we would be willing to accept as indicators of the ability to be measured for the purpose intended.

  49. Populations of Persons • In addition to defining the universe of possible measures, we must define the group, or population of persons about whom we are going to make decisions or inferences.

More Related