Making Evaluation Work

1. Making Evaluation Work Patti Phillips, Ph.D. patti@roiinstitute.net

2. During today’s webcast we will: • Review the 12 basic problems with evaluation. • Learn key features of the 3 steps to building a successful evaluation practice. • Learn how to overcome resistance to evaluation.

3. Tools for the job: • Pen or pencil • Paper or notebook

4. Poll Question Which of the following represents your biggest problem with evaluation? • Too many theories and models • Models are too complex • Lack of understanding of evaluation • Evaluation is considered a post-program activity • Lack of support from key stakeholders • Improper use of evaluation data • Other

5. Evaluation is too hard. There are too many theories and models. I don’t know how to do it! Evaluation is too subjective – no statistical precision Evaluation is post-program only—I need to justify the program before we launch it. What’s the ROI of evaluation?? Managers aren’t rewarded to use the data. If the results are negative, it will impact our bonus. How do we ensure we compare apples to apples? I wonder how long we will have to do this.

6. Three steps build a successful evaluation practice. Setting the Stage Developing the Practice Implementing the Practice Source: Phillips, J. J., Phillips, P. P., Hodges T. K. (2004) Make Evaluation Work. Alexandria: ASTD

7. Today we are talking about these components of each step. Setting the Stage Developing the Practice Implementing the Practice Measurement Culture Evaluation Readiness Stakeholder Needs Evaluation Purpose Evaluation Model Expertise Budgets

8. Setting the Stage • Does your organization implement the Balanced Scorecard, Six Sigma or any other quality management process? • Are there any type of tracking processes in place such as activity-based costing, cycle time reduction, customer satisfaction measurements? • Do you know the strategic business goals of your organization? • Do you know how the learning function links to these goals?

9. Setting the Stage • Is your organization large with a variety of programs? • Is your organization undergoing significant change? • Is there pressure from senior management to measure results? • Is the image of your training function less than desirable? • Is there an increased focus on linking training to strategy? For complete assessment go to Tools & Templates at www.roiinstitute.net.

10. Action Step 1

11. Setting the Stage

12. Setting the Stage

13. Poll Question Are you providing the measures needed by your stakeholders? • Yes • No • Maybe

14. Developing the Practice What purpose will an evaluation system serve you, your team, and your organization?

15. Developing the Practice • A successful evaluation system can help you: • Determine success in achieving program objectives. • Identify strengths and weaknesses in the learning and development process. • Set priorities for learning and development resources. • Test the clarity and validity of tests, cases and exercises. • Identify the participants who were most (or least) successful with the program. • Reinforce major points made during the program. • Decide who should participate in future programs. • Compare the benefits to the costs of a learning and development program. • Enhance the accountability of learning and development. • Assist in marketing future programs. • Determine if a program was an appropriate solution. • Establish a database to assist management with decision making.

16. Action Step 2 • Write your evaluation purpose statement. • Determine how you know you are successful?

17. Purpose drives the initiative to achieve outcomes. Implementation of our evaluation strategy gives us the information we need to help develop our people, improve our processes, and contribute to the bottom line – literally! We ensure that taxpayer dollars are allocated to the programs that directly support the agency mission.

18. How will you know you have achieved your purpose? • Unnecessary programs have been eliminated • Training budget has grown • Line supervisors are more supportive • Staff development funds have increased • Evaluation funds have increased • Successful programs are expanding throughout the enterprise

19. HRD Theory • Psychological Theory • Sociology • Systems Theory • Economics Theory • Evaluation Models • Cost-Benefit Analysis • Kirkpatrick’s Four-Level Framework • Phillips’ Five-Level Framework • Kaufman’s Five Levels of Evaluation • Utility Analysis • CIRO • CIPP • Marshall & Schriver’s Model for Evaluating Knowledge and Skills • Business Impact ISD Model • Success Case Evaluation • Brown & Reed’s Integral Framework • Balanced Scorecard • ADDIE Model • Assess • Design • Develop • Implement • Evaluate Developing the Practice Here put my diagram from dissertation but make the fourth box the criteria. List the criteria in order of what I found in the research. Phillips, P. P. (2003). Training Evaluation in the Public Sector. Doctoral Dissertation. The University of Southern Mississippi. International Development.

20. Developing the Practice • Consider the following. • Is the process simple? • Is the process economical? • Is the process credible? • Is the process theoretically sound? • Is the process based on generally accepted principles? • Does the process account for other factors that influence outcome measures? • Is the process flexible? • Is the process applicable with all types of data? • Does the process include a mechanism to develop program costs? • Are calculations based on accepted formulas? • Does the process have a successful track record? Here put my diagram from dissertation but make the fourth box the criteria. List the criteria in order of what I found in the research.

21. Action Step 3 • Review your current or potential evaluation model; how does it stack up to the following criteria? • Simple • Economical • Credible • Theoretically sound • Based on generally accepted principles • Accounts for other factors • Flexible • Applicable to all types of data • Accounts for all costs • Accepted calculations • Successful track record

22. What do you need to know? Statistical theory and methods Research design Analysis methods Interpretation and reporting of data Theories and types of evaluation What do you do with what you know? Identify customer expectations Select or design appropriate strategies, research design and measures Communicate and gain support for the evaluation plan Manage data collection Analyze and interpret data Report conclusions and make recommendations Implementing the Practice Source: Bernthal, P. R., K. Colteryahn, P. Davis, J. Naughton, W. J. Rothwell, and R. Wellins. (2004) ASTD 2004 Competency Study: Mapping the Future, Alexandria: ASTD.

23. Implementing the Practice Need Help? • Acquire formal education • Participate in professional organizations • Attend conferences • Attend workshops • Get Certified in ROI • Learn on the Job • Tap into a Network

24. Implementing the Practice What will it cost? • Salaries & employee benefits for L&D staff (No. of staff x avg. salary x employee benefits factor x no. of hours on evaluation project) • Meals, Travel, and Incidental Expenses • Participant Costs (for time involved in evaluation • Fees and Licenses • Office Supplies and Expense of Printing Materials • Outside Services • Technology • General Overhead Allocation • Other Miscellaneous Expenses 3% - 5% of total training budget

25. Poll Question What if I can’t afford to spend 3-5% of my existing budget on evaluation? My question to you is – Can you afford NOT to? • Yes • No • Maybe

26. Implementing the Practice Another way to put it… • Total cost of effective evaluation system 3%-5% of budget • Cost of processing one reaction questionnaire $2.00 • Application/behavior change study (internal) $2,000 - $3,000 • Application/behavior change study (external) $15,000 - $20,000 • ROI Impact Study (internal) $5,000 - $10,000 • ROI Impact Study (external) $15,000 - $50,000 • Fee to attend two-day workshop $850* • Fee to attend one-day workshop $450* • Fee for ROI Certification $2,995** • Cost of Handbook of Training Evaluation and Measurement Methods $62.50 • Cost of ROI foldout model Free** • Value of an effective, efficient evaluation system PRICELE$$ *ASTD Member Price **Available at info@roiinstitute.net

27. What are the steps to building a successful evaluation practice? • Set the Stage Recognize your organization’s culture and readiness for measurement along with your stakeholders’ data needs, so that you can • Develop the Practice By first defining your evaluation purpose and selecting the most appropriate model. Then, • Implement the Practice By developing the necessary expertise to do the job within your existing resources. Setting the Stage Developing the Practice Implementing the Practice Recognize your culture and readiness for measurement as well as your stakeholder data needs So that you an define your evaluation purpose and select the appropriate model. Then, develop the expertise needed to do the job with the resources you have available.

28. What if “they” resist? • Identify a champion • Assign roles and responsibilities • Establish evaluation targets • Develop a project plan for implementation • Revise/develop policies and procedures • Involve the entire learning staff • Teach the staff • Select programs for comprehensive evaluation • Report progress on studies • Prepare the management team

29. Action Plan • Name: You and Your Team Start Date: February 2, 2005 • Project: Making Evaluation Work – An Implementation Plan • Action StepsCompletion Date • Identify stakeholder data needs. ________________ • Who are our stakeholders? • What are their data needs? • What data do we currently measure? • What is needed? • Write evaluation purpose statement. ________________ • How will we know we are successful? • Compare evaluation processes to criteria. ________________

30. Make evaluation work…. for you!