Kirkpatrick Model

1. Kirkpatrick Model • Reaction • Learning • Behavior • Results (includes ROI)

2. Bloom/s Taxonomy • Evaluation – make & justify value judgments • Synthesis – combine & create new whole • Analysis – breakdown & identify relationships • Application – use learning in new situations • Comprehension – understand & restate • Knowledge/Recall -- memorization

3. Constructing Measures of Cognitive Achievement • Measures of Cognitive Achievement • Measures of Cognitive Achievement • Objective Measures • True-False • Multiple Choice • Matching • Constructed Response • Short Answer • Essay

4. Using True-False Questions • Good news • Easy to construct • Easy to score • Students familiar with them • Save time on testing • Bad news • Easy to misinterpret • Hard to measure above facts & details

5. Overall Limitations • Tends to be unreliable • Guessing problem • Encourage poor study habits • Difficult to use with opinionated material

6. Suggestions for Constructing True-False Items • Make the critical part obvious • Avoid trick questions • Avoid determining characteristics (give-away-words) - long statements, words like always, never, all, no, none, etc. • Avoid direct quotes from readings Avoid double negatives (It is never true that not all cooks burn the biscuits) • Provide random distributions of T-F

7. Multiple Choice Questions • Usually has a stem plus the correct answer • disguised by distractions or foils • Can be used in the measurement of affective, psychomotor, or cognitive learning • Can be very good at assessing higher levels of cognitive learning

8. Basic Formats • Best Answer • Correct Answer • Association • Analogy • Incomplete Sentence

9. Things You Can Measure with M/C • Applications of principles to new situations • Discrimination between correct & incorrect • Ability to explain, give reason for, or understand why

10. Advantages of M/C • Can measure a wider variety of behavior than other objective type items • Objectively scored • More reliable than T-F • Corrects somewhat for guessing

11. Disadvantages of M/C • Difficult & time consuming to construct • Requires good verbal skills • Require lots of space on page • Limits student response

12. How to Write Good M/C Items • Make the distracters plausible • Keep alternatives of equal length • Use at least 4 alternatives • Keep alternatives parallel in content • Avoid revealing clues (length, parallelism, verbs, & nouns) • Avoid patterns in keying responses

13. Matching Items – MC Variation • Simple type (A--B) • Compound type (A--B--C) • Multiple use of Alternatives (More than one match for each) • Balanced - alternatives match # s. Unbalanced - more alternatives than items • Verbal-pictorial

14. Uses of Matching Items • Problems -- Solutions • Words -- Meanings • Observe -- Identify • Cause -- Effect • Questions -- Answers • Situations -- Actions • Words – Symbols • Dates -- Events

15. Guide to Constructing Matching Items • Indicate the basis for the match • Make short items the alternatives & long items the stems • List alternatives in some logical order (e.g. alphabetical) • Should have from 5 to 15 items • Should be on a single page • Use only one correct match

16. Advantages of Matching • Efficient use of space & time • Good measure of learning associations • Good measure of cause & effect • Very objective in scoring • Easily scored • Relatively free from guessing • Can be used for higher level learning

17. Limitations of Matching Items • Difficult to write good matching items • Should avoid clues to the correct answer • Careful (& time-consuming) test construction required

18. Short Answer (Completion Item) • Recall or supply omitted key words, dates, figures, symbols • Good for assessing low level cognitive knowledge

19. Types of Short Answer Questions • Sentence completion • Paragraph completion • Drawing or picture completion • Formula completion • Simple recall • Object identification

20. Advantages of Completion tems • Demand accurate information • Measure retention of facts • Easy to construct • Good for who, what, where, when • Good for application of certain knowledge • High discrimination value • Minimal guessing

21. Disadvantages of Completion Items • Emphasizes verbal memorization • Students may “block” on known material • Rarely objective • Difficulty to measure higher order cognition

22. Guide for Constructing Completion Items • Don’t copy straight from the text • Place blanks near the end of the sentence • Avoid unnecessary clues • Have only 1 correct response • Limit things to be recalled to 1 or 2 concepts • Leave enough room for responses • Should be no relationship between the desired response and the size of the blank

23. Essay Items • Restricted Response- limits response by including specific delimiters • EX: Discuss 4 factors that have contributed to the current economic state of Illinois • Extended Response -- poses a very broad question or problem (i.e. Comp Questions) • EX: List, describe, and defend your selection of three most critical problems in training within the United States. Describe and defend the action(s) you would recommend as national policy for each problem, and describe probable results from each action

24. Advantages of Essay Items • Good for measuring complex cognitive behavior (analysis, synthesis, problem solving) • May help improve writing skills • Easy to construct • Measure creative thinking & problem solving

25. Limitations • Legibility • Subjective scoring • Takes a long time to score • Not good for large groups

26. Guide for ConstructingEssay Items • Ask for the response you want • Use only when objective items won’t work • Include time limits • Indicate weight given to each item • Prepare the scoring key before the test is written