evaluating california s web based training for medical assistants
Download
Skip this Video
Download Presentation
Evaluating California’s Web-based Training for Medical Assistants

Loading in 2 Seconds...

play fullscreen
1 / 30

ORGANIC OUTREACH - PowerPoint PPT Presentation


  • 364 Views
  • Uploaded on

Evaluating California’s Web-based Training for Medical Assistants Leslie Heyden, MA California Department of Public Health Immunization Branch Information and Education Section February 19, 2009 Overview

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'ORGANIC OUTREACH' - libitha


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
evaluating california s web based training for medical assistants

Evaluating California’s Web-based Training for Medical Assistants

Leslie Heyden, MA

California Department of Public Health

Immunization Branch

Information and Education Section

February 19, 2009

overview
Overview
  • The purpose of the CDPH IZ Branch evaluation is to assess the use and effectiveness of the first two immunization skills elearning lessons:
      • Preparing Vaccines
      • Administering Vaccines
  • The elearning is custom developed, interactive web-based training.
  • The primary target audience is medical assistants in California VFC provider practices.
  • The lessons are available via eziz.org.

2

eziz org
eziz.org
  • The new California VFC program website
  • A one-stop shop for California VFC providers and their staffs
  • A repository of a wide variety of immunization skills training and VFC program resources, including:
    • VFC forms (e.g., vaccine order and recertification forms, vaccine temperature and usage logs, patient eligibility screening forms)
    • Job aids, guidelines, posters, and up-to-the-minute news about vaccine availability

3

webinar learning objectives
Webinar learning objectives

By the end of this webinar you will be able to:

  • List the objectives of the evaluation plan
  • Explain why CDPH IZ branch selected the EZ-IZ elearning lessons to evaluate
  • Describe the methodology CDPH developed to evaluate the training
    • Indicators
    • Targets
    • Methods, including data sources
  • Describe lessons learned

5

why evaluate ez iz elearning
Why evaluate EZ-IZ’ elearning?
  • Is IZ Branch’s first elearning project
  • Has potential for high impact re: ability to reach unlimited number of learners with limited resources
  • Primary tool needed to conduct evaluation (LMS) was already being created
  • IZ Branch wanted to compare the “reach” of the elearning to existing classroom training

6

goal of activity to be evaluated
Goal of activity to be evaluated
  • To educate MAs in California VFC provider practices about best practices for vaccine preparation and administration

7

evaluation objectives
Evaluation objectives
  • By September 5, 2008 the first two elearning lessons of the EZ-IZ curriculum (Preparing Vaccines and Administering Vaccines) will launch, and the learning management system will begin tracking usage and learner performance.
  • By March 31, 2009, at least one person from 5% (approximately 180) of California’s active VFC provider sites (approximately 3,600) will have completed one of the elearning lessons (either Preparing Vaccines or Administering Vaccines) and have passed its posttest.

8

evaluation objectives cont d
Evaluation objectives (cont’d)
  • By September 5, 2009, at least one person from 20% (approximately 720) of active California VFC provider sites will have completed the Preparing Vaccines and Administering Vaccines elearning lessons and passed the posttest for each.

9

evaluation objectives cont d10
Evaluation objectives (cont’d)
  • By November 30, 2008 all user feedback about the EZ-IZ website will have been gathered and analyzed, and recommendations made.
    • Recommendations concerning website usability will be forwarded to and resolved by the web developer and/or the web editor.
    • Recommendations concerning the EZ-IZ Curriculum and the lessons in it will be forwarded to and resolved by the instructional designer.
    • Recommendations about programming and usability issues within the elearning lessons will be forwarded to and resolved by the elearning vendor.
    • Recommendations concerning the learning management system (LMS) will be forwarded to and resolved by the LMS vendor.

10

stakeholders
Stakeholders
  • Management
    • John Talarico, Interim CDPH IZ Branch Chief
    • Tiffany Sutter, Section Chief: Information & Education
    • Natalie Nakahara, Project Manager and Creative Director
  • California VFC Program
    • Claudia Aguiluz, VFC Program Coordinator
    • VFC field staff and field services reps
    • VFC provider staff (particularly MAs who work in private provider practices)

11

stakeholders cont d
Stakeholders (cont’d)
  • Immunization (federal, state, and local)
    • CDC (NCIRD)
    • Steve Nickell, Research Coordinator, IZ Branch
    • Sandra Jo Hammer, Nurse Consultant, IZ Branch
    • CDPH IZ Branch Evaluation team (Leslie Heyden, Natalie Nakahara, and Steve Nickell)
    • IZ Coordinators in LHDs
  • Others
    • California Medical Assistants Association (CMAA) and California medical assistant training programs

12

slide13

Table 3: Program Component/Activity Description

OutcomesShort term Long term

Inputs

ActivitiesInitial Subsequent

Outputs

Funding

Staff

Time

Review available materials & choose content

Design training

Employ vendor to program elearning and create learning management system

Make training available via website

Promote training

Review of content by content experts

Beta test on target audience

Collect & assemble feedback

Decide on changes

Revise training

Create a “one-stop shop” learning and resource portal for CA VFC providers, immunizers and other staff with clinical or administrative immunization-related responsibilities

(Live) online elearning lessons, with learning management system tracking

VFC providers will adapt to elearning as a training delivery method

Increased skill and knowledge (pre- posttest comparison) about vaccination among medical assistants & other office staff

Steady increase in the number of MAs who take the elearning lessons

Fewer real-time “person hours” required to train MAs

Fewer vaccine administration errors

Higher coverage rates

Fewer missed opportunities

Fewer VPDs

Increased provider/ user satisfaction

primary data source ez iz lms
Primary data source—EZ-IZ LMS
  • An LMS (Learning Management System) is software that automates the administration of training.
  • The LMS registers users, tracks courses in a curriculum, and records data from learners
  • The LMS provides reports to management.

14

evaluation question 1
Evaluation question #1
  • Q: Did the Preparing Vaccines and Administering Vaccines elearning lessons launch by September 5, 2008 (target)?
  • Indicator and method
    • The Preparing Vaccines and Administering Vaccines elearning lessons launch: request IZ branch staff to access lessons and report any problems accessing the lessons
  • Data source: feedback (verbal and email) from IZ Branch staff (and confirmation from LMS vendor)

15

evaluation question 2
Evaluation question #2
  • Q: Did the learning management system (LMS) begin tracking usage and learner performance data by September 5, 2008 (target)?
  • Indicators and methods
    • LMS is operational: verify with vendor that LMS is operational
    • LMS is gathering usage data: obtain registration and login data from LMS report
    • LMS is gathering learner performance data: obtain pretest, lesson start, lesson completion, and posttest data from LMS report

16

data source lms
Data source—LMS

Lesson

Post-test

Pre-test

19

evaluation question 3
Evaluation question #3
  • Q: How many provider sites have at least one person who has completed one lesson and passed the posttest?
  • Target and indicator
    • 5% of provider sites have at least one person who has completed one lesson and passed its posttest by March 31, 2009
  • Method
    • Generate LMS report of provider sites that shows lessons completed and posttest scores by provider site and user
  • Data source: LMS

20

evaluation question 4
Evaluation question #4
  • Q: How many provider sites have at least one person who has completed one lesson and passed the posttest?
  • Target and indicator
    • 20% of provider sites have at least one person who has completed one lesson and passed its posttest by September 5, 2009 (one year after launch)
  • Method
    • Generate LMS report of provider sites that shows lessons completed and posttest scores by provider site and user
  • Data source: LMS

22

evaluation question 5
Evaluation question #5
  • Q: What feedback has been received from EZ-IZ users that could indicate potential obstacles to accessing and completing the elearning lessons?
  • Targets and indicators
    • 100% of emails and 100% of all other feedback that could indicate technical problems with the website
    • 100% of emails and 100% of all other feedback that could indicate technical or any other problems with the lessons
  • Method
    • Emails about technical problems and overall feedback about the EZIZ website will be automatically sent to a specific (ezig.org) MS Outlook folder; phone calls will be logged and categorized by November 30, 2008
  • Data source: emails submitted by users via links on eziz.org

23

development of evaluation plan challenges
Development of evaluation plan—Challenges
  • Very few challenges. IZ Branch Research Coordinator collaborated on and provided very valuable input into plan.
  • Due to limited resources, not able to build as robust an LMS as originally planned.
  • Proposed revisions to methodology during plan development required consulting with LMS vendor who then had to modify LMS database or do additional programming.

25

lessons learned
Lessons learned
  • Did not add question about MA’s experience; could not separate primary target audience from secondary target audience.
  • Learned that more data was needed than provided by LMS. Added question to QAR for field reps to administer.
  • The project team learned a lot during this entire pilot effort (including the evaluation plan) and is applying lessons learned during the current development of the next two lessons.

26

this document
This document:

This document can be found on the CDC website at:

http://www.cdc.gov/vaccines/programs/progeval/downloads/cali-eval-02-09.ppt

ad