The Mastery Level Judgment Consistency Rate of a Rasch Model Based Standard Setting Method for Classroom Achievement Te

1. The Mastery Level Judgment Consistency Rate of a Rasch Model Based Standard Setting Method for Classroom Achievement Tests

2. 2 Table of Presentation Introduction Standard Setting Methods Research Questions Methodology Research Procedure Results Summary and Discussion

3. 3 Introduction Criterion-referenced Test - based on an appropriate standard (criterion) - guarantees what examinee has achieved and what he or she is able to do actually

4. 4 Introduction However in Korea - There have not been many researches related to standard setting method (Kim, Baek & Chae, 1998; Yang, 2000). - In order to promote the implementation of criterion-referenced tests, a variety of studies has to be conducted. Such as,

5. 5 Introduction Currently at middle schools in Korea - raw score itself is used as a standard ex) 80% mastery level = raw score 80 points out of 100 points - very simple to set up mastery level of tests - however, also easy to be changed according to the difficulty level of the test, so low judgment consistency rate

6. 6 Introduction An appropriate standard setting method in this study - judge student�s achievement level consistently regardless of the difficulty level of tests

7. 7 Standard Setting Methods - 1

8. 8 Standard Setting Methods - 1

9. 9 Standard Setting Methods - 2 Angoff Method - based on judge�s judgment about each item of the test 1) judge participated in judge training 2) to decide how many (out of 100) prescribed minimally mastery level examinees would likely answer example items correctly 3) to set up the first standard from a summation and averaging of those predictions 4) to repeat the second step with referenced to the first result, and then the second standard is set up 5) after iterative ratings of total three times, the final standard is established * The adequacy of a standard ? whether judges accurately conceptualize the minimal mastery level of candidates.

10. 10 Standard Setting Methods - 3 Rasch Method - based on Rasch model - based on judge�s judgment about each item of the test - recently developed method (Stone, 2001) ? necessary to examine usability at classroom achievement tests

11. 11 Standard setting method - 1 Procedure of Rasch Method 1) judge participated in judge training 2) to determine whether each item reflects essential content which minimally competent students should necessarily solve. ? �essential item group� 3) to set up a judge�s baseline criterion point = mean item difficulty of the essential group 4) to reflect the mastery level, mastery level which transformed into a logit measure (i.e. 80%=1.4logit) was added to a base line A judge�s final standard = a baseline criterion point + mastery level Final standard of the test = the average of each judge�s final standard

12. 12 Research Questions Is there significant difference in the mastery level judgment consistency rate between Rasch method and raw-score method? Is there significant difference in the mastery level judgment consistency rate between Rasch method and Angoff method?

13. 13 Methodology

14. 14 Methodology Two Science Achievement Tests (�Test 1� and �Test 2�) - developed with different difficulty levels (�test 1� was intended to be easier that �test 2� by and large) - based on Korean National Science Curriculum for 8th grade - composed of 25 multiple-choice items respectively - total test scores ranged between 0 and 100 respectively - Crobach�s alpha ? �Test 1� : 0.84 , �Test 2� : 0.82

15. 15 Methodology Item # 12 in �Test 1� (An example) 12. Which is the right match-up between the hormone that is released from the thyroid gland and its effect? ? Estrogen - promotes female secondary sex characteristics ? Thyroid Hormone - regulates metabolic rate ? Vasopressin - causes kidney to withhold water ? Epinephrine - regulates metabolic rate ? Insulin - facilitates to secrete Pancreatic Enzymes

16. 16 Methodology Item # 4 in �Test 2� (An example) * The below picture shows the endocrine glands in the human body. Answer the next question.

17. 17 Methodology Unidimensionality Check - � Test 1�

18. 18 Methodology Unidimensionality Check - �Test 2�

19. 19 Research Procedure

20. 20

21. 21 Results Descriptive Statistics

22. 22 Results Established Standards by Three Methods

23. 23 Results

24. 24 Results

25. 25 Results

26. 26 Results

27. 27 Results Grouped Students According to the Standards 1) Raw-score Method

28. 28 Results 2) Angoff Method

29. 29 Results 3) Rasch Method

30. 30 Results Investigated the Mastery Level Consistency Rate

31. 31 Results

32. 32 Summary and Discussion Summary - This study shows that the mastery level judgment consistency rate of Rasch method for classroom achievement tests is relatively higher than raw-score method�s rate and as high as Angoff method�s rate. - Moreover, there are additional advantages of Rasch method. ex) �Item-person Map� ? the difficulty of items and the distribution of persons can be represented visually.

33. 33 Summary and Discussion

34. 34 Summary and Discussion Discussion - Rasch method may supplement or replace both raw-score method and Angoff method for classroom achievement tests. - However, there are some restrictions because of the limited number of students and judges involved in the study and restricted items. - For further study, ? more examinees and judges are required. ? the comparison studies with other established standard setting methods, such as Ebel method, Jeager method and Nedelsky method as well as Angoff method and raw-score method should be conducted.

35. 35