1 / 41

Framework for Excellence

Framework for Excellence. Measuring Results Which helps in:Refining Site AnalysisMarketingCurriculum DesignNeeds AssessmentCourse Delivery and DevelopmentFurther Measurement and Evaluation!. PresentersCheryl Hamill, RN, MS, ACRN

benjamin
Download Presentation

Framework for Excellence

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


    2. Framework for Excellence Measuring Results Which helps in: Refining Site Analysis Marketing Curriculum Design Needs Assessment Course Delivery and Development Further Measurement and Evaluation!

    3. Presenters Cheryl Hamill, RN, MS, ACRN & Nancy Showers, DSW Delta Region AETC HIVQual Results 2002-2003 Sample RW Title III Community Health Center in Mississippi Mari Millery, PhD NY/NJ AETC Lessons from Assessing Knowledge & Practice Outcomes of Level III Trainings Jennifer Gray, RN, PhD & Richard Vezina, MPH TX/OK AETC, Women & HIV Symposium (JG) Pacific AETC, Asilomar Faculty Development Conference (RV) Debbie Isenberg, MPH, CHES & Margaret Clawson, MPH Southeast AETC Intensive On-Site Training Evaluation: A Mixed Methods Approach Brad Boekeloo, PhD, ScM NMAETC, Delta AETC Analysis of HIV Patient-Provider Communication

    4. Measurement and Evaluation Why evaluate? To determine if the training was successful in meeting aims (for participants and faculty) To decide how to change training content To improve the quality of training Why measure provider behavior change? To determine if training has the desired effect on participants and ultimately, on quality of care

    5. Kirkpatrick’s Model (from Kirkpatrick, Donald L. Evaluating Training Programs (2nd edition) 1998)

    6. The HIVQUAL Project Nancy Showers, DSW Delta Region AETC

    7. The HIVQUAL Project Capacity–building and organizational support for QI Individualized on-site consultation services Strengthen HIV-specific QI structure Foster leadership support for quality Guide performance measurement Facilitate implementation of QI projects Train HIV staff in QI methods Performance measurement data with comparative reports Partnership with HRSA to support quality management in Ryan White CARE Act community-based programs

    8. HIVQUAL Participants - 2003

    9. Annual PAP Test

    10. Annual Syphilis Screen

    11. Hepatitis C Status Known

    12. Adherence Discussed

    13. Viral Load Every 4 Months

    14. MAC Prophylaxis (CD4<50)

    15. Annual Dental Exam

    16. Annual Mental Health Assessment

    17. Delta AIDS Education and Training Center (DRAETC) Mississippi LPS - Training Summary Report Reporting period: July 1, 2002 - June 30, 2003 for Targeted RW Title-Funded Community Health Centers Cheryl Hamill, MS, RN, ACRN Instructor of Medicine Resource Center Director http://hivcenter.library.umc.edu

    19. Lessons from Assessing Knowledge and Practice Outcomes of Level III Trainings Mari Millery, PhD

    20. Decided to focus more outcome evaluation efforts on Level III because it is the most intensive and a high priority modality; and participants can be asked to devote time to extra paperwork Pre-test, post-test, and 3-month follow-up surveys Measures: Self-rating of comfort in performing clinical tasks Case-based knowledge questions

    23. Lessons Learned Can be done but getting follow-up surveys back is a challenge Preliminary results are encouraging – self-reported practice comfort and case-based knowledge questions appear to work as measures Survey needs to be minimum length Dropped knowledge questions in post-test because they were too soon after baseline – post-test focuses on feedback on training Nature of Level III varies: intensity/length, profession trained, topics covered, etc. Developed special versions for nurses and HepC 40 surveys collected with revised instruments this year – still working on getting all follow-up surveys back

    24. Measuring Training Outcomes Through Qualitative Interviewing TX/OK AETC Women & HIV Symposium (JG) and Asilomar Faculty Development Conference (RV) Jennifer Gray, RN, PhD (JG) Richard Vezina, MPH (RV) TX/OK AETC Pacific AETC

    25. TX/OK AETC Women & HIV Symposium (JG) First time region-wide symposium Multidisciplinary planning committee Lack of knowledge about gender-specific care Increased # of HIV infections among women in the region. Symposium goal: Improved care of HIV+ women Annual region-wide training conference 125 Participants, all PAETC faculty and program staff Conference goals: Improved skills and knowledge among faculty/trainers Improved training outcomes throughout region as a result of staff development

    26. Evaluation Plans JG Email one month post to all registrants Simple open-ended questions, for all disciplines Identify how content was used with patients and shared with peers. RV Post-Post: Form A: Self-assessment at end of Conference Identify skills and content learned, areas in which to integrate new skills and content Form B: 6 month Follow-Up Individualized telephone interviews, reviewing Form A Focus on how skills/content were applied; barriers

    27. Why these evaluation methods? Able to assess at multiple levels (Kirkpatrick model): Level 2 (Learning: improved knowledge) (RV) Level 3 (Behavior: change in practices) (JG, RV) Seeking specific content regarding conference (RV) Limited resources and time (JG) No existing tool found that met needs (JG)

    28. Findings Major Themes: (RV) Identified high need for continued skills training Transferred new skills/information to coworkers and employees Barrier to continued integration: Time constraints Major Themes: (JG) Impact on patients 13 had taught patients information learned at the symposium 3 had used info for referrals 3 system changes- i. e. assessment forms, clinical strategies Shared information with others: 8 informally, 1 structured, 4 created materials Most common topics: medication/adherence, HIV in general

    29. Strengths & Challenges of Methods What went well: Announced at end of symposium/conf. (JG, RV) Brief instrument encouraged higher response (JG) Longer instrument yielded rich responses (RV) What’s Next: Provide Incentives (JG, RV) Change instrument Shorter, easier instrument for higher response rate (RV) longer instrument for greater depth (JG) More effective confirmation of contact information (JG, RV)

    30. Intensive On-site Training Evaluation: A Mixed Methods Approach Debbie Isenberg, MPH, CHES Margaret Clawson, MPH Southeast AETC Intensive On-Site Training (IOST) Involves training, consultation, technical assistance and information dissemination (Levels I-V) Targeted towards new Ryan White Title III and other rural health sites Central office-based clinical instructor spends a half day to a full day at the site Intensive On-Site Training (IOST) Involves training, consultation, technical assistance and information dissemination (Levels I-V) Targeted towards new Ryan White Title III and other rural health sites Central office-based clinical instructor spends a half day to a full day at the site

    31. Study Overview Main research questions Process and Impact (Reaction and Learning) What was the quality of the training? How well were learning objectives met? What are the trainees’ intentions to change their clinical practice? Outcome (Learning and Behavior) How has the provider’s experience in the clinical training program impacted his/her ability (if at all) to provide HIV quality care to PLWH?

    32. Study Protocol Phase One Post training CQI form completed by participants Phase Two Recruitment packets mailed 3 months after last IOST Research staff contact potential participants 1 week later for interview Phase Three Reminder letter for 2nd interview sent 9 months after initial interview (total 12 months post IOST) Research staff contact participants 1 week later for interview Recruitment packets mailed 3 months after last IOST Recruitment letter Written informed consent information sheet Demographics survey Interview questions Research staff contact potential respondents 1 week later confirm packet receipt answer questions obtain oral consent schedule an interview time Recruitment packets mailed 3 months after last IOST Recruitment letter Written informed consent information sheet Demographics survey Interview questions Research staff contact potential respondents 1 week later confirm packet receipt answer questions obtain oral consent schedule an interview time Recruitment packets mailed 3 months after last IOST Recruitment letter Written informed consent information sheet Demographics survey Interview questions Research staff contact potential respondents 1 week later confirm packet receipt answer questions obtain oral consent schedule an interview time Recruitment packets mailed 3 months after last IOST Recruitment letter Written informed consent information sheet Demographics survey Interview questions Research staff contact potential respondents 1 week later confirm packet receipt answer questions obtain oral consent schedule an interview time

    33. Content: Phase Two and Three Written Demographic Assessment (PIF+) Semi-Structured Phone Interview (Tape recorded) Quantitative: participant asked to rate the effect of training in each specific training area Qualitative: participant asked to give concrete examples of how training has affected their skills in the clinical area If no effect reported, participants are asked for more explanation Topic Areas:Provider knowledge and ability Identification of risk behavior, signs and symptoms of early and advanced HIV disease Viral load, CD4 counts, and treatment implications Comorbidity of substance abuse and mental illness PEP and PHS Guidelines Topic Areas:Provider knowledge and ability Identification of risk behavior, signs and symptoms of early and advanced HIV disease Viral load, CD4 counts, and treatment implications Comorbidity of substance abuse and mental illness PEP and PHS Guidelines

    34. Strengths and Challenges

    35. Lessons Learned Think about what motivates the training audience to participate in the study when deciding on study design Develop the protocol to lower respondent form and time burden Don’t be afraid to change the protocol midway in the study if not working Consider the resources that you have to collect and analyze the data in choosing a study design

    36. Analysis of HIV Patient-Provider Communication Bradley O. Boekeloo, Ph.D., Sc.M. University of Maryland

    37. Methods Providers Randomized (n=8) Brief cultural competency training vs. none Audiotapes of HIV Visits (n=24) 3 patient visits tape recorded per physician. Tapes transcribed. Patient Exit Questionnaire (n=24) Interviewer read patient questions and patient answered on an answer form.

    40. Hypothesis Based on Exploratory Data and Next Steps Brief Intervention not enough for change Patients may be more comfortable discussing medical therapy than personal risk behaviors Try to determine whether different types of communication on audiotapes account for differences in patient comfort communicating with physician.

    41. Presenter Contact Information NY/NJ AETC: Mari Millery, PhD 212-305-0409 mm994@columbia.edu Delta Region AETC: - Cheryl Hamill, RN, MS, ACRN 601-984-5552 chamill@medicine.umsmed.edu - Nancy Showers, DSW 732-603-9681 njshowers@aol.com Southeast AETC: - Margaret Clawson, MPH 404-712-8448 mclawso@emory.edu - Debbie Isenberg, MPH, CHES 404-727-2931 disenbe@emory.edu

    42. Conference Call Evaluation Call 8: July 27, 2004 http://www.ihi.org/feedback/survey.asp?surveycode=AETCCall072704 Survey Code: AETCCall072704 For assistance contact: Lorna Macdonald at lmacdonald@ihi.org

More Related