Tool Development and Testing For Preceptor Development

1. NYU Langone Nursing Research Conference June 14, 2017 Tool Development and Testing For Preceptor Development Principal Investigator: Elizabeth Cotter PhD, RN- BC Nurse Educator -St. Francis Hospital Associate Professor Molloy College Co-Investigator: Patrica Eckardt PhD, RN Associate Professor Molloy College

2. St. Francis Hospital, Roslyn New York • 306 acute care beds • 24,353 Total surgery a year • 1,375 Open Heart Surgery • 4,116 Other cardiothoracic surgery • 11,176 Cardiac Catheterization procedures • 44, 035 Cardiac Arrhythmia Procedures • US News & World Report - Americas Health Best Hospitals for cardiac care ( 10th consecutive year) • Leapfrog Group and Consumer Reports – Received “A” nation wide safety score 10 out of 11 times since the safety score debut in 2012

3. St. Francis Hospital • Magnet Designation 2006, 2010, 2014, Re-designated in 2018 • Beacon Award: Gold Level- 2 Critical Care Units And IMCU( Step Down) Silver Level- 1 Critical Care Unit • Prism Award - 4 Medical/Surgical Telemetry Units • BSN or higher - 86 % • Professional Certification -33%

4. Professional Nursing Practice Model • Do you want any part of the nyberg scale here?

5. Transforming the Preceptor Program NEW OLD • Selection by NM & CNS • 6 hour classroom with a 2 hour renewal class every other year • CNS / NM involvement More robust selection process Blended program with competency education Clinical observation Annual Review CNS/NM involvement

6. Preceptor Selection REQUIRED Interest in working with new employees/students Self Assessment of caring attributes Demonstrates positive performance Competent in physical assessment PREFERRED • BSN minimum • Three years experience on the unit as a clinical nurse • One year employment as a certified surgical technician (OR only) • National Nursing Certification

9. Preceptor New RN Unit Managers CNS Teamwork for Success

10. Background of the Problem • There are few tools for preceptor identification, selection and evaluation. • The literature review revealed a preceptor selection process based on the attributes of a preceptor • Currently there is no psychometrically validated tool available for selection of preceptors to train the novice nurse in today’s healthcare environment

11. Purpose There were two parts to this study. The purpose of part I was to examine the content and face validity of the Cotter Preceptor Selection Instrument (CPSI) and the Proficiency Profile Self-Appraisal instrument( PPSA)by Hillaweg before further testing of construct validity and the piloting of the CPSI.

12. Research Questions • What is the content validity and face validity of the Cotter Preceptor Selection Instrument ? • What is the content validity and face validity of the Proficiency Profile Self-Appraisal (PPSA) tool ? • What is the construct validity of the Cotter Preceptor Selection Instrument (CPSI)?

13. Proficiency Profile Self-Appraisal (PPSA) tool Hilliwig Clinical Competence Communication Developed in 1993 90 items Professional Activities Teaching Ability Leadership

14. Peer Review of Preceptor– Unit Based Council Values Nursing Process Transformational Leadership Empowerment Peer Review Flexibility Collaborative/ Communication Skills Commitment Professional Development Conflict Resolution 1 = Needs Improvement 2 = Meets Expectations 3 = Exceeds Expectations

15. Study Method and Design

16. Part I Procedure The PPSA and the CPSI were evaluated by 4 Nursing Staff Development experts . The 2 tools were evaluated for face validity and content validity. A Fleiss Kappa was estimated for inter-rater agreement on the CPSI to account for chance inter-rater agreement

17. Procedure Part I

18. Part II : Procedure • For this part of the study we looked at concurrent validity. The concurrent criterion related validity is confirmed when scores on a tool are positively correlated with significant magnitude to a related criterion at the same point in time. 3 tools evaluated: • The CPSI score of the preceptor candidate • The preceptor candidates rating on their annual performance record completed by their unit manager. • The preceptor’s rating by their orientee

19. Results Part IFace and Content Validity • The 4 experts examined and rated each item of both the PPSA and CPSI as non-relevant (1 or 2 rating) or relevant (3 or 4 rating). The proportion of items on the PPSA tool on the scale that received a relevant rating of 3 or 4 by each expert was .99 and the estimated average of CVI was 3.636. • The proportion of items on the CPSI on the scale that received a relevant rating of 3 or 4 by each expert was 1.0 (S-CVI UA) and the estimated average of CVI (S-CVI-AVG) was 3.79. • A kappa inter-rater agreement estimates was also conducted. The kappa estimates inter-rater agreement while accounting for random chance of that agreement. The CPSI had a Fleiss kappa is .644 which indicates substantial agreement

20. Results Part IICriterion Validity • Descriptive statistics on each item were reported. The total score ranges from 39-42. On average, nurses with more experience scored higher on the CPSI than those with less experience. • A correlation matrix of all items estimated moderate to high inter-item correlations among most items. Items that had low correlations with their respective subscale score (r < .30) were re-examined for relevance. There was one subscale that had low inter-item correlations ( Professional Development subscale, n=2).

21. Results Part II continued • For inter-item reliability estimate, the Cronbach alpha of the CPSI was .853 (acceptable for a pilot instrument is >.70, DeVillis, 2012). • Additionally, as the number of items were relatively low at n= 14, the Cronbach alpha is not inflated due to larger number of items on the tool (Streiner, 2003 ).

22. Results Part IICriterion Validity • The CPSI was evaluated for criterion validity with two other evaluation tools: the preceptor candidate’s annual performance appraisal and the orientee’s evaluation of the preceptor. • The CPSI and Nurse manager’s annual evaluation of the preceptor were strongly correlated, r(13) = .55, p = .026. • There was a small to moderate positive correlation between the CPSI and orientee’s evaluation of the preceptor , r(11) = .23, p = .49.

23. Results Part II continued • There was one subscale that had low inter-item correlations ( Professional Development subscale, n=2). • Provides” learning moments” to develop peers. • Participates in learning activities, committees and or staff meetings. • The items may be measuring 2 different subconstructs, further testing of tool in larger sample will be conducted.

24. Discussion • Predictive convergent validity between the CPSI and orientee’s evaluation of the preceptor low. Time between the evaluations as preceptor and preceptor selection varied

25. Conclusion • The CPSI was found to have good construct validity both in translational validity( face and content) and criterion validity , making it a valid scale for preceptor selection. • The CPSI could be utilized by nurse managers and nurse educators to select the appropriate candidates to train the novice nurse. The tool offers a standardized rating system for preceptor selection.

26. Limitations / Recommendations •  Small sample size and limited variance on some items limits psychometric evaluation • Since this was a pilot study, further studies would be of use in validating the tool further.

27. References

28. Elizabeth Cotter PhD, RN BC 516-562-6134 Elizabeth.Cotter@chsli.org