making evaluation work l.
Download
Skip this Video
Download Presentation
Making Evaluation Work

Loading in 2 Seconds...

play fullscreen
1 / 30

Making Evaluation Work - PowerPoint PPT Presentation


  • 78 Views
  • Uploaded on

Making Evaluation Work. Patti Phillips, Ph.D. patti@roiinstitute.net. During today’s webcast we will:. Review the 12 basic problems with evaluation. Learn key features of the 3 steps to building a successful evaluation practice. Learn how to overcome resistance to evaluation.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Making Evaluation Work' - yuri


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
making evaluation work

Making Evaluation Work

Patti Phillips, Ph.D.

patti@roiinstitute.net

during today s webcast we will
During today’s webcast we will:
  • Review the 12 basic problems with evaluation.
  • Learn key features of the 3 steps to building a successful evaluation practice.
  • Learn how to overcome resistance to evaluation.
slide3

Tools for the job:

  • Pen or pencil
  • Paper or notebook
poll question
Poll Question

Which of the following represents your biggest problem with evaluation?

  • Too many theories and models
  • Models are too complex
  • Lack of understanding of evaluation
  • Evaluation is considered a post-program activity
  • Lack of support from key stakeholders
  • Improper use of evaluation data
  • Other
slide5

Evaluation is too hard.

There are too many theories and models.

I don’t know how to do it!

Evaluation is too subjective – no statistical precision

Evaluation is post-program only—I need to justify the program before we launch it.

What’s the ROI of evaluation??

Managers aren’t rewarded to use the data.

If the results are negative, it will impact our bonus.

How do we ensure we compare apples to apples?

I wonder how long we will have to do this.

three steps build a successful evaluation practice
Three steps build a successful evaluation practice.

Setting the Stage

Developing the Practice

Implementing the Practice

Source: Phillips, J. J., Phillips, P. P., Hodges T. K. (2004) Make Evaluation Work. Alexandria: ASTD

today we are talking about these components of each step
Today we are talking about these components of each step.

Setting the Stage

Developing the Practice

Implementing the Practice

Measurement Culture

Evaluation Readiness

Stakeholder Needs

Evaluation Purpose

Evaluation Model

Expertise

Budgets

slide8

Setting the Stage

  • Does your organization implement the Balanced Scorecard, Six Sigma or any other quality management process?
  • Are there any type of tracking processes in place such as activity-based costing, cycle time reduction, customer satisfaction measurements?
  • Do you know the strategic business goals of your organization?
  • Do you know how the learning function links to these goals?
slide9

Setting the Stage

  • Is your organization large with a variety of programs?
  • Is your organization undergoing significant change?
  • Is there pressure from senior management to measure results?
  • Is the image of your training function less than desirable?
  • Is there an increased focus on linking training to strategy?

For complete assessment go to Tools & Templates at www.roiinstitute.net.

poll question13
Poll Question

Are you providing the measures needed by your stakeholders?

  • Yes
  • No
  • Maybe
slide14

Developing the Practice

What purpose will an evaluation system serve you, your team, and your organization?

slide15

Developing the Practice

  • A successful evaluation system can help you:
  • Determine success in achieving program objectives.
  • Identify strengths and weaknesses in the learning and development process.
  • Set priorities for learning and development resources.
  • Test the clarity and validity of tests, cases and exercises.
  • Identify the participants who were most (or least) successful with the program.
  • Reinforce major points made during the program.
  • Decide who should participate in future programs.
  • Compare the benefits to the costs of a learning and development program.
  • Enhance the accountability of learning and development.
  • Assist in marketing future programs.
  • Determine if a program was an appropriate solution.
  • Establish a database to assist management with decision making.
slide16

Action Step 2

  • Write your evaluation purpose statement.
  • Determine how you know you are successful?
purpose drives the initiative to achieve outcomes
Purpose drives the initiative to achieve outcomes.

Implementation of our evaluation strategy gives us the information we need to help develop our people, improve our processes, and contribute to the bottom line – literally!

We ensure that taxpayer dollars are allocated to the programs that directly support the agency mission.

how will you know you have achieved your purpose
How will you know you have achieved your purpose?
  • Unnecessary programs have been eliminated
  • Training budget has grown
  • Line supervisors are more supportive
  • Staff development funds have increased
  • Evaluation funds have increased
  • Successful programs are expanding throughout the enterprise
slide19

HRD Theory

  • Psychological Theory
  • Sociology
  • Systems Theory
  • Economics Theory
  • Evaluation Models
  • Cost-Benefit Analysis
  • Kirkpatrick’s Four-Level Framework
  • Phillips’ Five-Level Framework
  • Kaufman’s Five Levels of Evaluation
  • Utility Analysis
  • CIRO
  • CIPP
  • Marshall & Schriver’s Model for Evaluating Knowledge and Skills
  • Business Impact ISD Model
  • Success Case Evaluation
  • Brown & Reed’s Integral Framework
  • Balanced Scorecard
  • ADDIE Model
  • Assess
  • Design
  • Develop
  • Implement
  • Evaluate

Developing the Practice

Here put my diagram from dissertation but make the fourth box the criteria. List the criteria in order of what I found in the research.

Phillips, P. P. (2003). Training Evaluation in the Public Sector. Doctoral Dissertation. The University of Southern Mississippi. International Development.

slide20

Developing the Practice

  • Consider the following.
  • Is the process simple?
  • Is the process economical?
  • Is the process credible?
  • Is the process theoretically sound?
  • Is the process based on generally accepted principles?
  • Does the process account for other factors that influence outcome measures?
  • Is the process flexible?
  • Is the process applicable with all types of data?
  • Does the process include a mechanism to develop program costs?
  • Are calculations based on accepted formulas?
  • Does the process have a successful track record?

Here put my diagram from dissertation but make the fourth box the criteria. List the criteria in order of what I found in the research.

slide21

Action Step 3

  • Review your current or potential evaluation model; how does it stack up to the following criteria?
    • Simple
    • Economical
    • Credible
    • Theoretically sound
    • Based on generally accepted principles
    • Accounts for other factors
    • Flexible
    • Applicable to all types of data
    • Accounts for all costs
    • Accepted calculations
    • Successful track record
slide22
What do you need to know?

Statistical theory and methods

Research design

Analysis methods

Interpretation and reporting of data

Theories and types of evaluation

What do you do with what you know?

Identify customer expectations

Select or design appropriate strategies, research design and measures

Communicate and gain support for the evaluation plan

Manage data collection

Analyze and interpret data

Report conclusions and make recommendations

Implementing the Practice

Source: Bernthal, P. R., K. Colteryahn, P. Davis, J. Naughton, W. J. Rothwell, and R. Wellins. (2004) ASTD 2004 Competency Study: Mapping the Future, Alexandria: ASTD.

slide23

Implementing the Practice

Need Help?

  • Acquire formal education
  • Participate in professional organizations
  • Attend conferences
  • Attend workshops
  • Get Certified in ROI
  • Learn on the Job
  • Tap into a Network
slide24

Implementing the Practice

What will it cost?

  • Salaries & employee benefits for L&D staff (No. of staff x avg. salary x employee benefits factor x no. of hours on evaluation project)
  • Meals, Travel, and Incidental Expenses
  • Participant Costs (for time involved in evaluation
  • Fees and Licenses
  • Office Supplies and Expense of Printing Materials
  • Outside Services
  • Technology
  • General Overhead Allocation
  • Other Miscellaneous Expenses

3% - 5% of total training budget

poll question25
Poll Question

What if I can’t afford to spend 3-5% of my existing budget on evaluation? My question to you is –

Can you afford NOT to?

  • Yes
  • No
  • Maybe
slide26

Implementing the Practice

Another way to put it…

  • Total cost of effective evaluation system 3%-5% of budget
  • Cost of processing one reaction questionnaire $2.00
  • Application/behavior change study (internal) $2,000 - $3,000
  • Application/behavior change study (external) $15,000 - $20,000
  • ROI Impact Study (internal) $5,000 - $10,000
  • ROI Impact Study (external) $15,000 - $50,000
  • Fee to attend two-day workshop $850*
  • Fee to attend one-day workshop $450*
  • Fee for ROI Certification $2,995**
  • Cost of Handbook of Training Evaluation

and Measurement Methods $62.50

  • Cost of ROI foldout model Free**
  • Value of an effective, efficient evaluation system PRICELE$$

*ASTD Member Price **Available at info@roiinstitute.net

what are the steps to building a successful evaluation practice
What are the steps to building a successful evaluation practice?
  • Set the Stage

Recognize your organization’s culture and readiness for measurement along with your stakeholders’ data needs, so that you can

  • Develop the Practice

By first defining your evaluation purpose and selecting the most appropriate model. Then,

  • Implement the Practice

By developing the necessary expertise to do the job within your existing resources.

Setting the Stage

Developing the Practice

Implementing the Practice

Recognize your culture and readiness for measurement as well as your stakeholder data needs

So that you an define your evaluation purpose and select the appropriate model.

Then, develop the expertise needed to do the job with the resources you have available.

what if they resist
What if “they” resist?
  • Identify a champion
  • Assign roles and responsibilities
  • Establish evaluation targets
  • Develop a project plan for implementation
  • Revise/develop policies and procedures
  • Involve the entire learning staff
  • Teach the staff
  • Select programs for comprehensive evaluation
  • Report progress on studies
  • Prepare the management team
slide29

Action Plan

  • Name: You and Your Team Start Date: February 2, 2005
  • Project: Making Evaluation Work – An Implementation Plan
  • Action StepsCompletion Date
  • Identify stakeholder data needs. ________________
    • Who are our stakeholders?
    • What are their data needs?
    • What data do we currently measure?
    • What is needed?
  • Write evaluation purpose statement. ________________
    • How will we know we are successful?
  • Compare evaluation processes to criteria. ________________