slide1 l.
Skip this Video
Loading SlideShow in 5 Seconds..
Presented by: Elizabeth Childs Tammy Dewar Dave Whittington PowerPoint Presentation
Download Presentation
Presented by: Elizabeth Childs Tammy Dewar Dave Whittington

Loading in 2 Seconds...

play fullscreen
1 / 26

Presented by: Elizabeth Childs Tammy Dewar Dave Whittington - PowerPoint PPT Presentation

  • Uploaded on

Creative Approaches to Evaluating E-Learning Presented by: Elizabeth Childs Tammy Dewar Dave Whittington Workshop Goals Explore the strengths and limitations of the Kirkpatrick model and discuss e-learning implementation examples at each level.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Presented by: Elizabeth Childs Tammy Dewar Dave Whittington

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Creative Approaches


Evaluating E-Learning

Presented by:

Elizabeth Childs

Tammy Dewar

Dave Whittington

workshop goals
Workshop Goals
  • Explore the strengths and limitations of the Kirkpatrick model and discuss e-learning implementation examples at each level.
  • Review a number of Return on Investment (ROI) models, their strengths and limitations, and how to choose one to fit your organizational context.
  • Develop a creative strategy for evaluating e-learning that can be customized to organizational goals and benchmark data from other organizations.
bingo game
Bingo Game

“We shouldn’t try to do something better until we first determine if we should do it at all” – Dwight D. Eisenhower –

levels of evaluation
Levels of Evaluation:
  • Level 1: Reaction and Planned Action
  • Level 2: Cognitive Learning and Retention
  • Level 3: On-The-Job Application
  • Level 4: Performance Results
  • Level 5: Return-On-Investment (ROI)
  • * Jack Phillips / Donald L. Kirkpatrick
jack phillips the need for roi
Jack Phillips – The Need for ROI
  • Phillips advises trainers to ask themselves these questions:
      • What is the life cycle of the training program?
      • How many people will use it?
      • How much money is involved?
  • The Phillips rule of thumb: Big projects, use ROI. Small ones, forget it. But when you do it, do it right.
typical roi characteristics
Travel time/costs

Class time-participants

Class time-instructors

Opportunity costs-participants

Records administration

Development time

Software/hardware costs

Maintenance costs

Measurable performance improvements

Support costs-printing, mailing, facilities

Typical ROI characteristics
calculating roi

Program Benefits

  • Benefit-Cost Ratio =

Program Costs

Net Program Benefits

  • ROI =


Program Costs

Calculating ROI
different ways of representing roi
Different Ways of Representing ROI
  • As a percentage.
    • If the customer gains benefits of $1,000,000 in 12 months on a total investment of $250,000 in the same time period, their ROI expressed as a percentage can be calculated as follows:
      • If, Return = Payback – Investment ROI = [(Payback – Investment)/Investment)]*100, or in this case: [($1,000,000-$250,000)]/$250,000*100 = 300%
  • As a ratio
    • Return/Investment = $1,000,000/$250,000 = 4:1
  • As a time to break-even.
    • ($250,000/$1,000,000)*12 months = (0.25)*12 = 3 months or 90 days
different decisions required different focus for roi
Different decisions required = different focus for ROI

Source: Jay Cross “A Fresh Look at ROI”

roi best practices
ROI – Best Practices
  • Know what you’re measuring! (isolate data from other factors)
  • Sample, don’t saturate! (pick a few impressive factors)
  • Show them the money! (both tangible and intangibles-travel time, more job satisfaction)
  • Compare apples to apples! (pre- and post- estimates for same factors)
        • Source: Jack Phillips
roi model 1 docent s approach
ROI Model # 1Docent’s approach
  • In order to collect the data required to substantiate a 400% or more ROI claim, Docent prefers to work directly with the “CXOs” and line of business managers, and not the traditional “training manager.”
  • The traditional training department metrics will typically yield ROIs of roughly 100% for a web-based training project.
docent roi approach white paper 2002
Docent ROI Approach White Paper (2002)
  • When developing the ROI estimate, the analysis should consider the following four categories of potential benefits:
      • Hard cost savings vs. alternative solutions
      • Hard revenue impacts vs. alternative solutions
      • Soft competitive benefits vs. alternatives
      • Soft benefits to individuals vs. alternatives
roi model 2 collis and moonen 2001
ROI Model # 2Collis and Moonen, 2001
  • Used to predict ROI and provide information about how to implement e-learning for various stakeholders
  • They suggest concentrating on a more intuitive (but still systematic) calculation that emphasizes the relative comparison of ROI for various stakeholders.
  • They compare costs (investments) and benefits (returns) on a small number of key issues that matter most to those involved.
collis and moonen roi model
Collis and Moonen ROI Model
  • Keep focus local
  • Identify major actors
  • Concentrate on project outcomes relating to economic and monetary terms, qualitative terms (often attitudes) and efficiency terms (time or effort)
  • For each actor, identify perceived impact in each of the project outcomes using a scale from –5 (very negative) to +5 (very positive)
roi model 3 doughty and shaw 2000
ROI Model # 3Doughty and Shaw, 2000
  • A model to help inform investment decisions
  • Based on the main concerns - Quality, Time and Cost - the method does not reduce these to a single measure
  • Respects multiple stakeholder perspectives
  • Considers levels of confidence and risks
  • Used to predict/estimate ROI
doughty and shaw 2000 simplified process
Doughty and Shaw, 2000 Simplified Process
  • Identify the Problem/Opportunity
  • Identify Stakeholders
  • Record their different perspectives
  • Identify alternatives (maybe go/no go)
  • Record perceived costs and benefits and where the evidence will come from
  • Estimate levels of confidence and risks associated with errors
  • Review alternatives
roi model 4 value creation forman 2002
ROI Model # 4 Value Creation – Forman (2002)
  • ROI not just about cost reduction
  • Relies on the definition of the term “value” within the organization
  • Categories of value measures that can be defined and used as priorities in the evaluation are based on what is real and vital for the organizations success. They commonly include:
      • Business strategy measures
      • Time measures
      • Human capital measures
      • Customer measures
roi model 5 the value proposition rosenberg
ROI Model # 5 The Value Proposition -Rosenberg
  • Focus of evaluation is on ability of elearning to deliver value to the business
  • Elearning value = the sum of (cost efficiency + quality + service + speed) where:
    • Cost efficiency is measured by delivery cycle time and productivity
    • Quality is measured by assessing the increase in knowledge/skill/ability intended by the program.
    • Service is measured by assessing how accessible is the elearning program
    • Speed is measured by assessing how responsive is the system is to the changing demands of the work.
return on expectations
Return on Expectations
  • Benedet suggests that organizations considering implementing an ROI strategy start the process by delivering an ROE (return-on-expectation) survey.

“It should be highly visible, and should pave the way for future sponsorship and participation in real ROI projects.” She advises companies to look at an opportunity and determine whether measures are in place. “If not,” she advises, “do a return on expectations.”

  • Source: Brenda Benedet, Director of e-learning strategies for SkillSoft
choosing your roi factors
Choosing Your ROI Factors
  • Have you defined what “counts” as return on investment in your organization?
  • Can you show a baseline for comparisons?
  • How well do you track current costs of training development and implementation?
  • How closely can performance data be identified, isolated, and linked to the training?

Doughty and Shaw

ASTD website –

Docent website –

Flexible Learning in a Digital Age, Betty Collis and Jef Moonen, Kogan Page Ltd, London, 2001.

Handbook of Training and Evaluation and Measurement Methods (3rd ed.), Jack J. Phillips, Gulf Publishing, Houston Texas, 1997.

In Action: Implementing Evaluation Systems and Processes, Jack J. Phillips, (Editor), ASTD, Alexandria, Virginia, 1998.

references continued
References Continued

Evaluating Training Programs (2nd Edition), Donald L. Kirkpatrick, Berrett-Koehler Publishers, San Francisco, California, 1998.

Evaluating Professional Development. Thomas R. Guskey. Corwin Press: Thousand Oaks, CA (2000).

e-Learning. M. J. Rosenberg. McGraw-Hill, New York. (2001).

ASTD E-Learning Handbook. McGraw-Hill, New York. (2002).

  • Course Evaluator
    • “More than a year ago, MVU began developing rigorous standards to guide the design and evaluation of online course quality. Based on decades of research and the work of the best minds in the field of Instructional Design, we have recently completed a comprehensive set of standards that can now be used to design and evaluate online courses.
  • Working through evaluating elearning
contact info
Contact Info
  • Elizabeth Childs –
  • Tammy Dewar –
  • Dave Whittington –