Exercise Unit 1 Jennifer Maddrell R561: Evaluation & Change in the Instructional Development Process Indiana Unive

1. Exercise Unit 1 Jennifer Maddrell R561: Evaluation & Change in the Instructional Development Process Indiana University Professor Knuth May 29, 2006

2. Branching OutTo a new Training Evaluation Plan

3. Evaluation is a means to: Brinkerhoff & Dressler: • Increase the value of the training investment Kaufman & Watkins: • Compare results with intentions Morrison, Kemp & Ross: • Make judgments about the worth or success of the training program Phillips: • Capture the contribution of human resource development • Determine customer satisfaction Van Tiem, Moseley & Dessinger: • Judge the results of performance • Trigger or support a decision

4. An evaluation plan will provide: • Feedback • As a quality control measure • Control • To assess value and worth to organization • Research • For knowledge to improve • Intervention • To affect how training is viewed, used and shared • Power • To fairly represent results to stakeholders Source: Knuth, 2006

5. Questions we must answer … • How do participants feel about our training program? • Are participants learning? • Is their learning transferring to the job? • Does the organization benefit from our training efforts?

6. Finding answers … Kirkpatrick’s 4 Levels of Evaluation • Level 1: Reaction • How do participants feel about our training program? • Level 2: Learning • Are participants learning? • Level 3: Behavior • Is their learning transferring to the job? • Level 4: Results • Does the organization benefit? Source: Kirkpatrick,1998

7. Kirkpatrick’s Model Measures: • Level 1: Reaction • Customer satisfaction • Learners motivation to learn • Level 2: Learning • Attitude change • Knowledge improvement and skill gain • Level 3: Behavior • Change in behavior • Transfer of skills • Level 4: Results • Financial Impact: On costs, production, etc. • Non-financial Impact: On morale, motivation, etc. Source: Kirkpatrick,1998

8. Appeal of Kirkpatrick’s Model: • Assesses important areas • Widely known • Simple framework • Easy to explain and understand

9. However . . . • Widely Know ≠ Widely Used • Level 1: Often (over 90%) • Level 2: Sometimes (less than 35%) • Level 3 & 4: Rarely (less than 15%) • Why is this a problem? • Level 3 and 4 often perceived as: • Difficult to measure • Time consuming • Beyond the realm of most trainers • Level 1 result does not always mean similar Learning / Transfer / ROI results Source: Pershing & Gilmore, 2004

10. Other problems … • Undermines Management Partnership • Training ≠“Silver Bullet” • Training is only one strategy within entire Performance System • Level 3 & 4 should include evaluations of entire Performance System - not just training • Lacks Performance System Focus • What about rest of Performance Environment? • What factors impede / enable usage of training? • Feedback Goes to Wrong People • Feedback to training function only is incomplete • Must include Performance Environment owners Source: Brinkerhoff & Dressler, 2002

11. Alternatives to Kirkpatrick: • Numerous alternative evaluation options exist • Recommend implementation of Brinkerhoff & Dressler’s Success Case Evaluation Model • This alternative focuses on training’s business impactas part ofentire performance system.

12. Success Case Model Answers: • What is the business impact of instructional program? • What is the organization doing that is facilitating performance improvement? • What is the organization doing that is impeding performance improvement? Source: Brinkerhoff & Dressler, 2002

13. Success Case Model Approach: • Brief survey to large sample to assess: “To what extent have you used your recent training in a way that you believe has made a significant difference to the business?” • In depth small sample review of both: • successful groups: • nature and business value of their application of learning, and • performance context factors (support) • unsuccessful groups: • performance context factors (obstacles) • other factors preventing use of learning Source: Brinkerhoff & Dressler, 2002

14. In Summary: • An evaluation tool must be integrated into the training programs at BIG. • Kirkpatrick’s Four Levels of Evaluation is well known, but has limitations: • Unlikely completion of all 4 Levels • Lacks performance system focus • Brinkerhoff & Dressler’s Success Case Evaluation Modelis the recommended next step approach. • Rapid evaluation / feedback process • Addresses key business impact issues • Contemplates entire performance environment

15. References Brinkerhoff, R. O. & Dressler, D. (in press). Using evaluation to build organizational performance and learning capability: A strategy and a method. Performance Improvement. Kaufman, R., Keller, J., & Watkins, R. (1995). What works and what doesn't: Evaluation beyond Kirkpatrick. Performance & Instruction, 35, (2). 205-209. Kirkpatrick, D. L. (1998). The four levels: An overview. Ch. 3 in Evaluating Training Programs: The Four Levels, 2nd ed. San Francisco: Berrett-Koehler. Knuth, R. (2006). Unit 1: Introduction to Evaluation. Retrieved May 18, 2006, from Indiana University R526 Syllabus http://www.indiana.edu/~istr561/knuth06sum/unit1print.html Morrison, Gary R., Kemp, Jerrold E., & Ross, Steven M. (2001). Chapter 10, The Many Faces of Evaluation. In Designing Effective Instruction (3rd edition). New York: John Wiley & Sons. Pershing, J., Gilmore, E. (2004). Evaluating Training Programs Kirkpatrick’s Four Levels. [PowerPoint Presentation] Retrieved May 18, 2006, from Indiana University R526 Syllabus http://www.indiana.edu/~istr561/knuth06sum/PPTs/r561d1_kirk_perspective.ppt Phillips, J. (1997). Handbook of Training Evaluation and Measurement Methods (Improving Human Performance Series) 3rd Edition. Butterworth-Heinemann. Chapters 1 - 3. Van Tiem, Darlene M., Moseley, James L., Dessinger, Joan Conway (2004). Chapter 7, Evaluation. In Fundamentals of Performance Technology: A Guide to Improving People, Process, and Performance, 2nd Edition. Washington, DC: International Society for Performance Improvement.