Training Kirkpatrick’s Four Levels of Evaluation Kelly Arthur Richard Gage-Little Dale Munson

1. Training Kirkpatrick’s Four Levels of EvaluationKelly ArthurRichard Gage-LittleDale Munson Evaluation Strategies for Instructional Designers

2. Lost Keys

3. Where should we be searching for the keys to good evaluation results?

4. Beginning to see the light… 2000 ASTD State of the Industry Report 100 95% 80 60 40 37% 20 13% 3% 0 Reaction Learning Behavior Results Kirkpatrick’s four levels

5. Billions Wasted About $5.6 billion to $16.8 billion is wasted annually on ineffective training programs that focus on so-called soft skills, based on estimates from a Rutgers University study. USA Today

6. Higher Standard of Accountability • In the “new economy”, training and development initiatives are being held to a higher level of financial accountability. • Every dollar spent on training should produce measurable business results – increases in key business metrics.

7. Results of Evaluation Survey How important do you think evaluations are in the instructional design process? • “Very important, otherwise you don’t know if your training met your objectives” • “Evaluations are integral, but often overlooked” • “Crucial, they’re the only way to know if we are reaching our goals” • “Without evaluations, we have no clue what what the core of the problem is”

8. Results of Evaluation Survey Have you completed an evaluation and how did you feel about it? • “I never found them very useful to me personally” • “…felt they were a waste of my time” • “…dreading it” • “…seems ineffective since the next class is presented the same way” • “A little annoyed if I was busy”

9. What is Evaluation? Evaluationhelps document whether a program is accomplishing its goals or not. It identifies program weaknesses and strengths and the areas of the program that need revision. • Asking questions • Collecting answers • Making decisions based on those answers

10. What is Assessment? Assessment is the systematic gathering of information about “things” that are to be evaluated. • Examples: Behaviors, attitudes, performance • Identifies needs or gaps • Determines whether or not instruction or training is needed

11. Two Types of Evaluation Formative Evaluation • Occurs during development phase • Focuses on process • Used to help instructional designers improve the quality of instruction Summative Evaluation • Occurs near, at, or after the end of a project • Focus is on the effectiveness • Used to help instructional designers validate their training

12. Five Evaluation Models • Hierarchy of Evaluation (Kirkpatrick) • Objectives Approach (Tyler) • Goal-Free (Scriven) • CIPP (Stufflebeam) • Naturalistic (Guba)

13. Who is Donald Kirkpatrick? • Professor at University of Wisconsin • Initial articles published in 1959-1960 • American Society of Training Directors, ASTD • Most widely recognized and used model in the training industry

14. Phases of Evaluation in Training 1 2 3 4 Pre-Testing During training event After training In the workplace Training Work Place 1. Reaction 3. Behavior Kirkpatrick’s four levels 2. Learning 4. Results

15. Hierarchy of Evaluation • Four Levels of Evaluation • Level 1 – Reaction (participant satisfaction) • Level 2 – Learning (knowledge, mastery) • Level 3 – Behavior (transference of skills) • Level 4 – Results (community impact) • Fifth level was recently “added” for return on investment (ROI) but this was not in Kirkpatrick’s original model.

16. Measures reaction Checks participants response to: Session or course Trainer’s presentation style or content Quality of training materials Example: Questionnaires or surveys Level One Evaluation

17. Example at Sun • How satisfied were you with the instructor’s technical expertise? • How satisfied were you with the labs/and or exercises provided? • How satisfied were you with the accuracy of the course materials?

18. Level Two Evaluation • Measures learning • Checks participants response to: • Knowledge and factual information • Skills • Attitudes • Interpretation of information • Example: Pre and Post-Testing

19. Measures behavior Checks participants ability to: Carry out tasks more effectively after the course Provide better care to clients Be more knowledgeable/skillful in job performance Examples: Pre and post tests, surveys and interviews, self-assessments Establishes a baseline of skills/knowledge Level Three Evaluation

20. Level Four Evaluation • Measures results • Provides information about: • Bottom line final results • Impact on the organization • Cost vs. Benefit • Example: Reports • Proof vs. Evidence

21. Example at Sun • Service call reduction – external customers • Reduction in the amount of time spent per service call – internal support engineers

22. Hierarchy of Evaluation

23. Pre-Test Answers • Reaction: Survey, questionnaires • Learning: Case studies, tests and quizzes • Behavior: Pre and post tests, interviews, self-assessments • Results: Reports • Formative evaluation • Summative evaluation

24. In-Class Activity

25. In-Class Activity • Your boss, Mr. Grinch, has decided to downsize his company, Hateful Things. The training program that you’ve been conducting on “How to ruin Christmas” is currently under review. You’ve been asked to justify and qualify why this training and your job are important to his company. How will you do this?

26. Remember The “Keys” To Evaluation Reaction Learning Behavior Results 1 2 3 4

27. References • Kirkpatrick, Donald. “Another Look at Evaluating Training Programs.” ASTD, Alexandria, VA. 1998. • Kirkpatrick, Donald. “Evaluating Training Programs.” Berrett-Koehler Publishers, San Francisco, CA. 1998.