Chapter 16

1. Evaluating Knowledge Chapter 16

2. Chapter 16 Outline • Purposes of Knowledge Testing • Levels of Knowledge • Types of Knowledge Tests • Construction • Administration and Scoring • Analysis and Revision • Questionnaires

3. Purposes of Knowledge Testing • Assigning a grade (summative evaluation) • Measuring progress (formative evaluation) • Providing feedback to students or participants • Motivating students or participants • Documenting effectiveness • Assessing instructional effectiveness

4. Levels of Knowledge • Understanding can range from superficial to thorough. • Bloom’s taxonomy of educational objectives • Knowledge • Comprehension • Application • Analysis • Synthesis • Evaluation

5. Educational Testing Service • Uses three levels of knowledge for Education Field Tests: • Knowledge • Comprehension • Application

6. Characteristics of a GoodKnowledge Test • Fair • Covers the course content • Uses appropriate test format • Clear and understandable (no tricks) • Reliable and valid

7. Students’ Pet Peeves • Trick questions • Poorly-written items • No opportunity for feedback • Too easy/too difficult • floor/ceiling effect • Testing outside the content boundaries • Too much rote memorization of facts with limited relevance

8. Types of Knowledge Tests • Essay vs. Objective • Mastery vs. Discrimination

9. Essay Test • People answer each item with whatever information they choose and write answers in sentences. • Quick to construct, but time consuming to score.

10. Objective Test • Potential answers are provided with each test item. • true-false • multiple choice • matching • Time consuming to construct, but quick to score.

11. Mastery Test • Used to determine whether students have mastered the material. • Items test information everyone is expected to know. • Formative evaluation with criterion-referenced standards (e.g., graded pass-fail). • Performance standards tend to be high. • e.g., 80% to 90% correct for passing.

12. Discrimination Test • Purpose is to differentiate among students. • Summative evaluation with norm-referenced standards. • Items are written to discriminate among different knowledge levels. • Items should come from higher levels of Bloom’s taxonomy. • Because discrimination tests are more difficult than mastery tests, performance standards should be lower.

13. Construction ofKnowledge Tests • Advantages of instructor-made tests: • tend to cover material stressed • terminology is consistent with teaching • tend not to include irrelevant material • tend not to omit important material

14. Steps in Constructing a Knowledge Test • Construct a table of specifications • define content area • Decide the nature of the test • choose types of item to be used • Construct the test items • Determine test format and administrative details

15. Table of Specifications - the Blueprint • Content validity evidence • What is the content domain? • Relevance (what does/does not belong in the content domain?) • Representativeness (what is the appropriate balance of content?) • Determined by importance, or time spent teaching that content.

16. Example Table of Specifications

17. Types of Test Items Closed-ended (“objective”) • correct answer is provided • more difficult to construct, easier to grade • do not test partial knowledge • may reward guessing Types include: • true/false • multiple choice • matching

18. Types of Items Open-ended (“subjective”) • student provides correct answer • no knowledge, no score • rewards partial knowledge (may seem fairer) • easier to construct, more difficult to grade Types include: • short answer • fill in the blank • essay

19. Item Construction Guidelines • Determine the knowledge, understanding, abilities • Determine the most appropriate format

20. Multiple choice items • Terms: stem, foil, distractor • Use complete statement, question, or instruction in stem • Test only one concept at a time • Provide plausible distractors • Numerical answers should be in order, and not overlap each other • Avoid “compound” answers, unless the material dictates it

21. Multiple choice items • Avoid negatives, and especially double (and triple!) negatives • Ensure that stem and foils are all grammatically compatible, similar in length and “parallel” in content • Keep to factually or logically supportable correct answers (avoid opinion) • Could a good student answer without the foils? • Avoid “irrelevant cues” – e.g., always, never, common terms between stem and correct answer

22. Common Errors • All distractors are not plausible. • Item is ambiguous. • Wording the correct response more precisely than the distractors. • Specific determiners. • Irrelevant clues. • Grammatical clues

23. Common Errors • Look at the following multiple-choice items • Can you figure out the answer? • What is wrong with the item? • How could each item be improved?

24. Question 1: In Cornwall, a tourist or holidaymaker is called an: • rubberneck • emmet • pastie • tin mine

25. Question 1: In Cornwall, a tourist or holidaymaker is called an: • rubberneck • emmet* • pastie • tin mine

26. Question 1: In Cornwall, what term is used to describe a tourist or holidaymaker? • rubberneck • emmet • pastie • tin mine

27. Question 2: What is the best way to get from Buckingham Palace to Elephant and Castle? • in the Queen’s carriage • with an umbrella • in a straight line • via Piccadilly Circus

28. Question 2: What is the best way to get from Buckingham Palace to Elephant and Castle? • in the Queen’s carriage • with an umbrella • in a straight line • via Piccadilly Circus*

29. Question 2: What is the best way to get from Buckingham Palace to Elephant and Castle? • in the Queen’s carriage* • with an umbrella* • in a straight line* • via Piccadilly Circus*

30. Question 2: Of the following routes, which one is the most direct route from Buckingham Palace to Elephant and Castle on the London Underground? • via Sloane Square • via Marble Arch • via Paddington Station • via Piccadilly Circus

31. Question 3: Use of ID cards has led to a decrease in crowd violence at which one of the following venues? • Tesco’s supermarkets • Elton John concerts • Soccer matches • The Natural History Museum

32. Question 3: Use of ID cards has led to a decrease in crowd violence at which one of the following venues? • Tesco’s supermarkets • Elton John concerts • Soccer matches* • The Natural History Museum

33. Question 3: Use of ID cards has led to a decrease in crowd violence at which one of the following professional sporting venues? • Rugby matches • Cricket matches • Soccer matches* • Tennis matches

34. Question 4: Which famous British knight of the realm starred in the movie “Bridge Over the River Kwai”? • Duke of Wellington • Lord Louis Mountbatten • Earl Grey • Sir Alec Guinness

35. Question 4: Which famous British knight of the realm starred in the movie “Bridge Over the River Kwai”? • Duke of Wellington • Lord Louis Mountbatten • Earl Grey • Sir Alec Guinness*

36. Question 4: Which famous British knight of the realm starred in the movie “Bridge Over the River Kwai”? • Sir Laurence Olivier • Sir John Gielgud • Sir Anthony Hopkins • Sir Alec Guinness*

37. Question 5: At what age can children leave the British school system? • 15 years • 16 years • 17 years • 18 years

38. Question 5: At what age can children leave the British school system? • 15 years* • 16 years • 17 years • 18 years

39. Question 5: At what age can children leave the British school system? • 15 years* • 16 years* • 17 years* • 18 years*

40. Question 5: What is the minimum age at which children are legally allowed to leave the British school system? • 15 years* • 16 years • 17 years • 18 years

41. Question 6: What is the nickname for the professional basketball team of Birmingham (England)? • Rangers • Bullets • Celtics • Gunners

42. Question 6: What is the nickname for the professional basketball team of Birmingham (England)? • Rangers • Bullets* • Celtics • Gunners

43. Question 6: In which British city is the Bullets professional basketball team located? • Birmingham • Bradford • Bolton • Barnsley

44. Administration and Scoring • Provide comfortable, distraction-free testing site. • Consider test security: • all students should face same direction • students should sit far enough apart to inhibit cheating • use alternate forms of a test when necessary

45. Before the Test • Prepare students. • Eliminate test-wise advantage: • grammatical clues • specific determiners • Proofread before photocopying test.

46. During the Test • Help students pace themselves. • Answer individual questions carefully. • Control cheating: • Negates reliability and validity of test. • Negative attitudes toward those who cheat. • Negative attitudes toward teacher who does not control cheating. • Negative attitude toward testing in general. • Control the environment.

47. After the Test • Correct and report test scores as quickly as possible. • Report scores anonymously. • Do not misuse or misinterpret test scores.

48. Analysis and Revision • Overall Difficulty • Mean • Variability • Standard deviation. Larger S, more the test discriminates. • Reliability • KR20 or KR2

49. Analysis and Revision • Difficulty of each item. • Discrimination or validity of each item. • Quality of each response. • In a multiple choice test, some students should select each response.

50. Kuder-Richardson (KR) • Estimate of internal consistency reliability by determining how all items on a test relate to the total test. • KR formulas 20 and 21 are typically used to estimate Rxx of knowledge tests. • Used with dichotomous items (scored as right or wrong). • KR20 = coefficient alpha