HOW TO ADDRESS THE THREE MOST CHALLENGING ROI ISSUES

1. HOW TO ADDRESS THE THREE MOST CHALLENGING ROI ISSUES Jack J. Phillips, Ph.D. Chairman, ROI Institute

2. Objectives • After completing this session, participants should perceive the content to be: • • Relevant to their work • • Immediately applicable • Participants will be able to: • • Explain the benefits of using return on investment (ROI). • • Describe the steps in the ROI Methodology. • • Explain the three most challenging areas in an ROI study. • • Identify the principal uses of ROI data.

3. The “Show Me” Evolution From Executives Term Issue Show Me! Collect Impact Data Show Me the Money! And Convert Data to Money And Isolate the Effects of the Project Show Me the Real Money! Show Me the Real Money, And Make me Believe it! And Compare the Money to the Cost of the Project

4. Why ROI Now? • • ROI is the ultimate measure • • ROI has been the elusive measure • • ROI has a rich history of application • • Operating managers understand and relate to ROI • • ROI builds excitement among stakeholders • • ROI is a top executive requirement

5. The ROI Methodology Captures Six Types of Measures • • Reaction and Planned Action • • Learning • • Application • • Business Impact • • Return on Investment • • Intangible Measures • ……..and includes a technique to isolate the effects of the program or solution

6. Status of Measurement *Percent of programs evaluated at this level † Best Practice benchmarking (user for 5 plus years) >>Add your numbers in each box

7. Status of Measurement *Percent of programs evaluated at this level † Best Practice benchmarking (user for 5 plus years) >>Add your numbers in each box

8. Status of Measurement *Percent of programs evaluated at this level † Best Practice benchmarking (user for 5 plus years) >>Add your numbers in each box

9. Five Levels of Measurement - Examples * Correlates with Application

10. Five Levels of Measurement - Examples

11. Five Levels of Measurement - Examples

12. The ROI Methodology Evaluation Planning Data Collection LEVEL 1: REACTION AND PLANNED ACTION LEVEL 3: APPLICATION Collect Data During Program Implementation Collect Data After Program Implementation Develop Evaluation Plans and Baseline Data Develop/ Review Objective of Program LEVEL 2: LEARNING LEVEL 4: BUSINESS IMPACT

13. Capture Costs of Program Data Analysis Reporting Isolate the Effects of Program Calculate the Return On Investment Convert Data To Monetary Value Generate Impact Study LEVEL 5: ROI Identify Intangible Measures Intangible Benefits

14. ROI Calculation Net Project Benefits Project Costs ROI = Cost of project $230,000 Benefits of project (1st year) $430,000 $430,000 - $230,000 $230,000 ROI = = 0.87 x 100 = 87%

15. Evaluation Framework

16. Guiding Principles • When a higher level evaluation is conducted, data must be collected at lower levels. • When an evaluation is planned for a higher level, the previous level of evaluation does not have to be comprehensive. • When collecting and analyzing data, use only the most credible sources. • When analyzing data, choose the most conservative among alternatives. • At least one method must be used to isolate the effects of the program or project. • If no improvement data are available, it is assumed that little or no improvement has occurred.

17. Guiding Principles Estimates of improvement should be adjusted for the potential error of the estimate. Extreme data items and unsupported claims should not be used in ROI calculations. Only the first year of benefits should be used in the ROI analysis of short-term projects. Fully loaded all costs of the solution, project of program when analyzing ROI. Intangible measures are defined as measures that are purposely not converted to monetary value. The results from the ROI Methodology must be communicated to all key stakeholders.

18. An evaluation system must have five elements. An Evaluation Framework Applications and Practice Implementation A Process Model Operating Standards and Philosophy

19. Criteria for Selecting Programs for Level 4 and Level 5 Evaluation

20. Alignment Model Start Here End Here Payoff Needs 5 ROI Objectives 5 ROI 4 Impact Objectives 4 Business Needs Impact Performance Needs 3 Application Objectives 3 Application Initial Analysis Measurement and Evaluation 2 Learning Objectives 2 Learning Learning Needs 1 Reaction Objectives 1 Reaction Preference Needs Project Business Alignment and Forecasting The ROI Process Model

21. Challenge #1

22. Matching Exercise Survey Test Questionnaire Interview Focus Groups Observation Performance Records

23. Increasing Questionnaire Response Rates • • Provide advance communication about the questionnaire. • • Clearly communicate the reason for the questionnaire. • • Indicate who will see the results of the questionnaire. • • Show how the data will be integrated with other data. • • Keep the questionnaire simple and as brief as possible.

24. Increasing Questionnaire Response Rates • • Keep questionnaire responses anonymous – or at least confidential. • • Make it easy to respond; include a self-addressed, stamped envelope/e-mail. • • Use the local manager to distribute the questionnaires, show support, and encourage response. • • If appropriate, let the target audience know that they are part of a carefully selected sample. • • Use one or two follow-up reminders.

25. Increasing Questionnaire Response Rates • • Have the introduction letter signed by a top executive. • • Enclose a giveaway item with the questionnaire (pen, money, etc.). • • Provide an incentive (or chance of incentive) for quick response. • • Send a summary of results to target audience. • • Distribute questionnaire to a captive audience. • • Consider an alternative distribution channel, such as e-mail.

26. Increasing Questionnaire Response Rates • • Have a third party gather and analyze data. • • Communicate the time limit for submitting responses. • • Consider paying for the time it takes to complete the questionnaire. • • Review the questionnaire at the end of the formal session. • • Carefully select the survey sample. • • Allow completion of the survey during work hours.

27. Increasing Questionnaire Response Rates • • Add emotional appeal. • • Design questionnaire to attract attention, with a professional format. • •Let participants know what actions will be taken with the data. • • Provide options to respond (paper, email, web-site). • • Use a local coordinator to help distribute and collect questionnaires. • • Frame questions so participants can respond appropriately and make the questions relevant.

28. Improving Post Program Data Collection • • Plan early: before program is conducted • • Communicate expectations • • Secure commitment to provide • • Build in the data collection tools • • Use at least 12 techniques in • • Share results • • Use the data

29. Challenge #2

30. Isolating the Effects of the Program Benchmarking Data • Method • Control Groups • Trend Line Analysis • Forecasting Methods • Participant Estimates • Manager Estimates • Sr. Management Estimates • Expert Input • Customer Input • * Survey of Users, N = 235

31. Control Group Design Control Group M1 M2 Program M2 M1 Experimental Group

32. Use of Control Groups Example Retention Solution at the Federal Information Agency • An opportunity to participate in a master’s degree program at agency expense and agency time • One hundred high-potential employees chosen for program for a three-year master’s program in information science • Experimental group of one hundred were involved in the program; another one hundred in control group were not involved • Observed employee turnover performance of both groups during the same time • Neither group is aware of the control group arrangement • How would you select the control group?

33. 1.85% Pre Program Six-Month Average Use of Trend Line Analysis Fraud Program Conducted Projected Average — Using Pre Data as a Base 14.5% Fraud Incident Rates 7% Post Program Six-Month Average J F M A M J JA S O N D J MONTHS

34. Example of Estimation • Performance Improvement: Time to Process Visas =15 minutes

35. Using Estimates to Isolate the Effects • Describe the task and the process. • Explain why the information was needed and how it will be used. • Ask participants to identify any other factors that may have contributed to the increase. • Have participants discuss the linkage between each factor and the specific output measure.

36. Using Estimates to Isolate the Effects • Provide participants with any additional information needed to estimate the contribution of each factor. • Obtain the actual estimate of the contribution of each factor. The total must be 100%. • Obtain the confidence level from each employee for the estimate for each factor (100%=certainty; 0%=no confidence).

37. The Power of Estimates • Research • Comparison with other methods • Handling objections • Management reactions • Participant reaction • Use as a last resort

38. Matching Exercise Control Group Trend Line Analysis Forecasting Participant’s estimate Use of Customer Input Expert’s Estimate

39. Challenge #3

40. Matching Exercise

41. Typical Intangible Benefits

42. Typical Intangible Benefits

43. ROI Is Reported In Two Ways Program Benefits Program Costs Benefits-Cost Ratio (BCR) = Net Program Benefits Program Costs ROI (%) X 100 =

44. ROI Is Reported In Two Ways $500,000 $400,000 $500,000 - $400,000 $400,000 BCR= = 1.25 ROI= x 100 =25%

45. Setting the ROI Objective • What’s a great ROI? It depends on the objectives • • Set the value as with other investments – e.g.15% • • Set the value slightly above other investments – e.g. 25% • • Set at breakeven – e.g.0% • • Set at client expectations –e.g. 30%

46. ROI Red Flags Not using a balanced profile of data Failure to isolate the effects of the program Overstating the monetary benefits stream Leaving out some costs Not being conservative Not being consistent

47. ROI Methodology : The Payoff • • Align projects to business needs • • Show contributions of selected projects • • Earn respect of senior management/administrators • • Build staff morale • • Justify/defend budgets • • Improve support for projects

48. ROI Methodology : The Payoff • • Enhance design and implementation processes • • Identify inefficient projects that need to be redesigned or eliminated • • Identify successful projects that can be implemented in other areas • • Earn a “seat at the table”

49. Questions? Web: www.roiinstitute.net Email: jack@roiinstitute.net Connect with us on: