Evaluating California’s Web-based Training for Medical Assistants - PowerPoint PPT Presentation

Evaluating california s web based training for medical assistants l.jpg
Download
1 / 30

Evaluating California’s Web-based Training for Medical Assistants Leslie Heyden, MA California Department of Public Health Immunization Branch Information and Education Section February 19, 2009 Overview

Related searches for Evaluating California’s Web-based Training for Medical Assistants

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Evaluating California’s Web-based Training for Medical Assistants

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Evaluating california s web based training for medical assistants l.jpg

Evaluating California’s Web-based Training for Medical Assistants

Leslie Heyden, MA

California Department of Public Health

Immunization Branch

Information and Education Section

February 19, 2009


Overview l.jpg

Overview

  • The purpose of the CDPH IZ Branch evaluation is to assess the use and effectiveness of the first two immunization skills elearning lessons:

    • Preparing Vaccines

    • Administering Vaccines

  • The elearning is custom developed, interactive web-based training.

  • The primary target audience is medical assistants in California VFC provider practices.

  • The lessons are available via eziz.org.

  • 2


    Eziz org l.jpg

    eziz.org

    • The new California VFC program website

    • A one-stop shop for California VFC providers and their staffs

    • A repository of a wide variety of immunization skills training and VFC program resources, including:

      • VFC forms (e.g., vaccine order and recertification forms, vaccine temperature and usage logs, patient eligibility screening forms)

      • Job aids, guidelines, posters, and up-to-the-minute news about vaccine availability

    3


    Eziz org home page l.jpg

    eziz.org—home page

    4


    Webinar learning objectives l.jpg

    Webinar learning objectives

    By the end of this webinar you will be able to:

    • List the objectives of the evaluation plan

    • Explain why CDPH IZ branch selected the EZ-IZ elearning lessons to evaluate

    • Describe the methodology CDPH developed to evaluate the training

      • Indicators

      • Targets

      • Methods, including data sources

    • Describe lessons learned

    5


    Why evaluate ez iz elearning l.jpg

    Why evaluate EZ-IZ’ elearning?

    • Is IZ Branch’s first elearning project

    • Has potential for high impact re: ability to reach unlimited number of learners with limited resources

    • Primary tool needed to conduct evaluation (LMS) was already being created

    • IZ Branch wanted to compare the “reach” of the elearning to existing classroom training

    6


    Goal of activity to be evaluated l.jpg

    Goal of activity to be evaluated

    • To educate MAs in California VFC provider practices about best practices for vaccine preparation and administration

    7


    Evaluation objectives l.jpg

    Evaluation objectives

    • By September 5, 2008 the first two elearning lessons of the EZ-IZ curriculum (Preparing Vaccines and Administering Vaccines) will launch, and the learning management system will begin tracking usage and learner performance.

    • By March 31, 2009, at least one person from 5% (approximately 180) of California’s active VFC provider sites (approximately 3,600) will have completed one of the elearning lessons (either Preparing Vaccines or Administering Vaccines) and have passed its posttest.

    8


    Evaluation objectives cont d l.jpg

    Evaluation objectives (cont’d)

    • By September 5, 2009, at least one person from 20% (approximately 720) of active California VFC provider sites will have completed the Preparing Vaccines and Administering Vaccines elearning lessons and passed the posttest for each.

    9


    Evaluation objectives cont d10 l.jpg

    Evaluation objectives (cont’d)

    • By November 30, 2008 all user feedback about the EZ-IZ website will have been gathered and analyzed, and recommendations made.

      • Recommendations concerning website usability will be forwarded to and resolved by the web developer and/or the web editor.

      • Recommendations concerning the EZ-IZ Curriculum and the lessons in it will be forwarded to and resolved by the instructional designer.

      • Recommendations about programming and usability issues within the elearning lessons will be forwarded to and resolved by the elearning vendor.

      • Recommendations concerning the learning management system (LMS) will be forwarded to and resolved by the LMS vendor.

    10


    Stakeholders l.jpg

    Stakeholders

    • Management

      • John Talarico, Interim CDPH IZ Branch Chief

      • Tiffany Sutter, Section Chief: Information & Education

      • Natalie Nakahara, Project Manager and Creative Director

    • California VFC Program

      • Claudia Aguiluz, VFC Program Coordinator

      • VFC field staff and field services reps

      • VFC provider staff (particularly MAs who work in private provider practices)

    11


    Stakeholders cont d l.jpg

    Stakeholders (cont’d)

    • Immunization (federal, state, and local)

      • CDC (NCIRD)

      • Steve Nickell, Research Coordinator, IZ Branch

      • Sandra Jo Hammer, Nurse Consultant, IZ Branch

      • CDPH IZ Branch Evaluation team (Leslie Heyden, Natalie Nakahara, and Steve Nickell)

      • IZ Coordinators in LHDs

    • Others

      • California Medical Assistants Association (CMAA) and California medical assistant training programs

    12


    Slide13 l.jpg

    Table 3: Program Component/Activity Description

    OutcomesShort term Long term

    Inputs

    ActivitiesInitial Subsequent

    Outputs

    Funding

    Staff

    Time

    Review available materials & choose content

    Design training

    Employ vendor to program elearning and create learning management system

    Make training available via website

    Promote training

    Review of content by content experts

    Beta test on target audience

    Collect & assemble feedback

    Decide on changes

    Revise training

    Create a “one-stop shop” learning and resource portal for CA VFC providers, immunizers and other staff with clinical or administrative immunization-related responsibilities

    (Live) online elearning lessons, with learning management system tracking

    VFC providers will adapt to elearning as a training delivery method

    Increased skill and knowledge (pre- posttest comparison) about vaccination among medical assistants & other office staff

    Steady increase in the number of MAs who take the elearning lessons

    Fewer real-time “person hours” required to train MAs

    Fewer vaccine administration errors

    Higher coverage rates

    Fewer missed opportunities

    Fewer VPDs

    Increased provider/ user satisfaction


    Primary data source ez iz lms l.jpg

    Primary data source—EZ-IZ LMS

    • An LMS (Learning Management System) is software that automates the administration of training.

    • The LMS registers users, tracks courses in a curriculum, and records data from learners

    • The LMS provides reports to management.

    14


    Evaluation question 1 l.jpg

    Evaluation question #1

    • Q: Did the Preparing Vaccines and Administering Vaccines elearning lessons launch by September 5, 2008 (target)?

    • Indicator and method

      • The Preparing Vaccines and Administering Vaccines elearning lessons launch: request IZ branch staff to access lessons and report any problems accessing the lessons

    • Data source: feedback (verbal and email) from IZ Branch staff (and confirmation from LMS vendor)

    15


    Evaluation question 2 l.jpg

    Evaluation question #2

    • Q: Did the learning management system (LMS) begin tracking usage and learner performance data by September 5, 2008 (target)?

    • Indicators and methods

      • LMS is operational: verify with vendor that LMS is operational

      • LMS is gathering usage data: obtain registration and login data from LMS report

      • LMS is gathering learner performance data: obtain pretest, lesson start, lesson completion, and posttest data from LMS report

    16


    Data source registration data l.jpg

    Data source—registrationdata

    17


    Learning history page l.jpg

    Learning History Page

    18


    Data source lms l.jpg

    Data source—LMS

    Lesson

    Post-test

    Pre-test

    19


    Evaluation question 3 l.jpg

    Evaluation question #3

    • Q: How many provider sites have at least one person who has completed one lesson and passed the posttest?

    • Target and indicator

      • 5% of provider sites have at least one person who has completed one lesson and passed its posttest by March 31, 2009

    • Method

      • Generate LMS report of provider sites that shows lessons completed and posttest scores by provider site and user

    • Data source: LMS

    20


    Data source pre and posttests l.jpg

    Data source—pre- and posttests

    21


    Evaluation question 4 l.jpg

    Evaluation question #4

    • Q: How many provider sites have at least one person who has completed one lesson and passed the posttest?

    • Target and indicator

      • 20% of provider sites have at least one person who has completed one lesson and passed its posttest by September 5, 2009 (one year after launch)

    • Method

      • Generate LMS report of provider sites that shows lessons completed and posttest scores by provider site and user

    • Data source: LMS

    22


    Evaluation question 5 l.jpg

    Evaluation question #5

    • Q: What feedback has been received from EZ-IZ users that could indicate potential obstacles to accessing and completing the elearning lessons?

    • Targets and indicators

      • 100% of emails and 100% of all other feedback that could indicate technical problems with the website

      • 100% of emails and 100% of all other feedback that could indicate technical or any other problems with the lessons

    • Method

      • Emails about technical problems and overall feedback about the EZIZ website will be automatically sent to a specific (ezig.org) MS Outlook folder; phone calls will be logged and categorized by November 30, 2008

    • Data source: emails submitted by users via links on eziz.org

    23


    Data source feedback from users l.jpg

    Data source—feedback from users

    24


    Development of evaluation plan challenges l.jpg

    Development of evaluation plan—Challenges

    • Very few challenges. IZ Branch Research Coordinator collaborated on and provided very valuable input into plan.

    • Due to limited resources, not able to build as robust an LMS as originally planned.

    • Proposed revisions to methodology during plan development required consulting with LMS vendor who then had to modify LMS database or do additional programming.

    25


    Lessons learned l.jpg

    Lessons learned

    • Did not add question about MA’s experience; could not separate primary target audience from secondary target audience.

    • Learned that more data was needed than provided by LMS. Added question to QAR for field reps to administer.

    • The project team learned a lot during this entire pilot effort (including the evaluation plan) and is applying lessons learned during the current development of the next two lessons.

    26


    Added question to qar l.jpg

    Added question to QAR

    27


    Questions l.jpg

    Questions?

    28


    Slide29 l.jpg

    Leslie HeydenHealth Educator & Instructional DesignerLeslie.Heyden@cdph.ca.gov(510) 620-3754www.eziz.org

    29


    This document l.jpg

    This document:

    This document can be found on the CDC website at:

    http://www.cdc.gov/vaccines/programs/progeval/downloads/cali-eval-02-09.ppt


  • Login