Creative Approaches
1 / 26

ROI and Evaluation - PowerPoint PPT Presentation

  • Updated On :

Creative Approaches to Evaluating E-Learning Presented by: Elizabeth Childs Tammy Dewar Dave Whittington Workshop Goals Explore the strengths and limitations of the Kirkpatrick model and discuss e-learning implementation examples at each level.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'ROI and Evaluation' - jaden

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Slide1 l.jpg

Creative Approaches


Evaluating E-Learning

Presented by:

Elizabeth Childs

Tammy Dewar

Dave Whittington

Workshop goals l.jpg
Workshop Goals

  • Explore the strengths and limitations of the Kirkpatrick model and discuss e-learning implementation examples at each level.

  • Review a number of Return on Investment (ROI) models, their strengths and limitations, and how to choose one to fit your organizational context.

  • Develop a creative strategy for evaluating e-learning that can be customized to organizational goals and benchmark data from other organizations.

Bingo game l.jpg
Bingo Game

“We shouldn’t try to do something better until we first determine if we should do it at all” – Dwight D. Eisenhower –

Levels of evaluation l.jpg
Levels of Evaluation:

  • Level 1: Reaction and Planned Action

  • Level 2: Cognitive Learning and Retention

  • Level 3: On-The-Job Application

  • Level 4: Performance Results

  • Level 5: Return-On-Investment (ROI)

  • * Jack Phillips / Donald L. Kirkpatrick

Evaluation frequencies l.jpg

Evaluation Frequencies:

Jack phillips the need for roi l.jpg
Jack Phillips – The Need for ROI

  • Phillips advises trainers to ask themselves these questions:

    • What is the life cycle of the training program?

    • How many people will use it?

    • How much money is involved?

  • The Phillips rule of thumb: Big projects, use ROI. Small ones, forget it. But when you do it, do it right.

  • Typical roi characteristics l.jpg

    Travel time/costs

    Class time-participants

    Class time-instructors

    Opportunity costs-participants

    Records administration

    Development time

    Software/hardware costs

    Maintenance costs

    Measurable performance improvements

    Support costs-printing, mailing, facilities

    Typical ROI characteristics

    Calculating roi l.jpg

    Program Benefits

    • Benefit-Cost Ratio =

    Program Costs

    Net Program Benefits

    • ROI =


    Program Costs

    Calculating ROI

    Different ways of representing roi l.jpg
    Different Ways of Representing ROI

    • As a percentage.

      • If the customer gains benefits of $1,000,000 in 12 months on a total investment of $250,000 in the same time period, their ROI expressed as a percentage can be calculated as follows:

        • If, Return = Payback – Investment ROI = [(Payback – Investment)/Investment)]*100, or in this case: [($1,000,000-$250,000)]/$250,000*100 = 300%

    • As a ratio

      • Return/Investment = $1,000,000/$250,000 = 4:1

    • As a time to break-even.

      • ($250,000/$1,000,000)*12 months = (0.25)*12 = 3 months or 90 days

    Different decisions required different focus for roi l.jpg
    Different decisions required = different focus for ROI

    Source: Jay Cross “A Fresh Look at ROI”

    Roi best practices l.jpg
    ROI – Best Practices

    • Know what you’re measuring! (isolate data from other factors)

    • Sample, don’t saturate! (pick a few impressive factors)

    • Show them the money! (both tangible and intangibles-travel time, more job satisfaction)

    • Compare apples to apples! (pre- and post- estimates for same factors)

      • Source: Jack Phillips

    Roi model 1 docent s approach l.jpg
    ROI Model # 1Docent’s approach

    • In order to collect the data required to substantiate a 400% or more ROI claim, Docent prefers to work directly with the “CXOs” and line of business managers, and not the traditional “training manager.”

    • The traditional training department metrics will typically yield ROIs of roughly 100% for a web-based training project.

    Docent roi approach white paper 2002 l.jpg
    Docent ROI Approach White Paper (2002)

    • When developing the ROI estimate, the analysis should consider the following four categories of potential benefits:

      • Hard cost savings vs. alternative solutions

      • Hard revenue impacts vs. alternative solutions

      • Soft competitive benefits vs. alternatives

      • Soft benefits to individuals vs. alternatives

    Roi model 2 collis and moonen 2001 l.jpg
    ROI Model # 2Collis and Moonen, 2001

    • Used to predict ROI and provide information about how to implement e-learning for various stakeholders

    • They suggest concentrating on a more intuitive (but still systematic) calculation that emphasizes the relative comparison of ROI for various stakeholders.

    • They compare costs (investments) and benefits (returns) on a small number of key issues that matter most to those involved.

    Collis and moonen roi model l.jpg
    Collis and Moonen ROI Model

    • Keep focus local

    • Identify major actors

    • Concentrate on project outcomes relating to economic and monetary terms, qualitative terms (often attitudes) and efficiency terms (time or effort)

    • For each actor, identify perceived impact in each of the project outcomes using a scale from –5 (very negative) to +5 (very positive)

    Roi model 3 doughty and shaw 2000 l.jpg
    ROI Model # 3Doughty and Shaw, 2000

    • A model to help inform investment decisions

    • Based on the main concerns - Quality, Time and Cost - the method does not reduce these to a single measure

    • Respects multiple stakeholder perspectives

    • Considers levels of confidence and risks

    • Used to predict/estimate ROI

    Doughty and shaw 2000 simplified process l.jpg
    Doughty and Shaw, 2000 Simplified Process

    • Identify the Problem/Opportunity

    • Identify Stakeholders

    • Record their different perspectives

    • Identify alternatives (maybe go/no go)

    • Record perceived costs and benefits and where the evidence will come from

    • Estimate levels of confidence and risks associated with errors

    • Review alternatives

    Roi model 4 value creation forman 2002 l.jpg
    ROI Model # 4 Value Creation – Forman (2002)

    • ROI not just about cost reduction

    • Relies on the definition of the term “value” within the organization

    • Categories of value measures that can be defined and used as priorities in the evaluation are based on what is real and vital for the organizations success. They commonly include:

      • Business strategy measures

      • Time measures

      • Human capital measures

      • Customer measures

    Roi model 5 the value proposition rosenberg l.jpg
    ROI Model # 5 The Value Proposition -Rosenberg

    • Focus of evaluation is on ability of elearning to deliver value to the business

    • Elearning value = the sum of (cost efficiency + quality + service + speed) where:

      • Cost efficiency is measured by delivery cycle time and productivity

      • Quality is measured by assessing the increase in knowledge/skill/ability intended by the program.

      • Service is measured by assessing how accessible is the elearning program

      • Speed is measured by assessing how responsive is the system is to the changing demands of the work.

    Return on expectations l.jpg
    Return on Expectations

    • Benedet suggests that organizations considering implementing an ROI strategy start the process by delivering an ROE (return-on-expectation) survey.

      “It should be highly visible, and should pave the way for future sponsorship and participation in real ROI projects.” She advises companies to look at an opportunity and determine whether measures are in place. “If not,” she advises, “do a return on expectations.”

    • Source: Brenda Benedet, Director of e-learning strategies for SkillSoft

    Choosing your roi factors l.jpg
    Choosing Your ROI Factors

    • Have you defined what “counts” as return on investment in your organization?

    • Can you show a baseline for comparisons?

    • How well do you track current costs of training development and implementation?

    • How closely can performance data be identified, isolated, and linked to the training?

    References l.jpg

    Doughty and Shaw

    ASTD website –

    Docent website –

    Flexible Learning in a Digital Age, Betty Collis and Jef Moonen, Kogan Page Ltd, London, 2001.

    Handbook of Training and Evaluation and Measurement Methods (3rd ed.), Jack J. Phillips, Gulf Publishing, Houston Texas, 1997.

    In Action: Implementing Evaluation Systems and Processes, Jack J. Phillips, (Editor), ASTD, Alexandria, Virginia, 1998.

    References continued l.jpg
    References Continued

    Evaluating Training Programs (2nd Edition), Donald L. Kirkpatrick, Berrett-Koehler Publishers, San Francisco, California, 1998.

    Evaluating Professional Development. Thomas R. Guskey. Corwin Press: Thousand Oaks, CA (2000).

    e-Learning. M. J. Rosenberg. McGraw-Hill, New York. (2001).

    ASTD E-Learning Handbook. McGraw-Hill, New York. (2002).

    Resources l.jpg

    • Course Evaluator

      • “More than a year ago, MVU began developing rigorous standards to guide the design and evaluation of online course quality. Based on decades of research and the work of the best minds in the field of Instructional Design, we have recently completed a comprehensive set of standards that can now be used to design and evaluate online courses.

    • Working through evaluating elearning

    Contact info l.jpg
    Contact Info

    • Elizabeth Childs – [email protected]

    • Tammy Dewar – [email protected]

    • Dave Whittington – [email protected]