DISASTER DRILL EVALUATION

1. DISASTER DRILL EVALUATION Gary B. Green, MD, MPH, FACEP Office of Critical Event Preparedness and Response (CEPAR) Johns Hopkins University, Baltimore, Maryland

2. The Johns Hopkins University Evidence-based Practice Center (EPC)supported by the Agency for Healthcare Research & Quality The Johns Hopkins University Office of Critical Event Preparedness and Response (CEPAR) Johns Hopkins University School of Medicine, Department of Emergency Medicine

3. Objectives: • Understand the critical role of disaster simulations (drills) and their evaluation in disaster preparedness efforts • Discuss the data collection instruments, methods, and analytical techniques used to perform an objective, quantitative disaster drill evaluation • Describe the challenges faced in the actual performance of disaster drill evaluations and successful strategies to overcome these challenges

4. Why do disaster drills? • Training • Skills-based • Evaluation • Raise Awareness • Administration, Staff, Public • Team Building • Fulfill regulatory requirements • JCAHO, others

5. Learning Opportunities: • A drill affords opportunity to learn: • Emergency Operations (Disaster) Plan • Gain/Improve Skills-Based Competence • New, modified job requirements • Operations in unfamiliar and difficult settings • Use of infrequently used equipment • Deal with Large Numbers of Patients • Teamwork • Problem Solving, Improvisation

6. Drilling as Formative Evaluation: "The purpose of evaluation may be • formative (to provide ongoing feedback so that the learners or curriculum can improve) or • summative (to provide a final "grade" assessment of the performance of the learner or curriculum)" Kern DE, Thomas PA, Howard DM, Bass EB. Curriculum Development for Medical Education: A Six Step Approach. The Johns Hopkins University Press, Baltimore 1998, pp 6-7.

7. Post-course knowledge assessment Pre-course knowledge assessment Didactic education training (modular courses) Review/modify Disaster plan Skills/Practical Training (Drills) Drill evaluation: (Institutional & Individual skills assessment) (Re) Define stakeholders & goals Modification & re-engineering of training interventions Report & analysis of strengths & weaknesses Continuous Quality Improvement (CQI) Process Applied to Disaster Preparedness Capacity Building

8. Basic Principles of Drill Evaluation: • Drill/Evaluation Planning • Components of disaster response to be evaluated • Drill evaluation tools & techniques

9. 1. Define Drill Goals and Boundaries • What are Specific Goals for this Drill? • Clinical response training • Incident Command System (ICS) effectiveness • Chemical, biological, radiation incident response • Efficiency of response • Define Boundaries of Drill Activities • Jurisdictions involved: • Single or multi-agency • Interface with outside agencies? • Hospital, pre-hospital, both? • Medical only, law enforcement, others? • Decontamination activities included?

10. 2. Consider Resources Available • Adequate time before drill? • Buy-in by key stakeholders? • Live volunteers available? • Distinct evaluation team? • Time and place available for debriefing? • Continuing normal services during drill? DO NOT BE AMBITIOUS BEYOND RESOURCE AVAILABILITY!!

11. 3. Select Specific System Response Components to be Evaluated: • Clinical Operations • Efficiency (Drill Flow) • Logistics (materials, facilities, transportation) • Other Operational Components • S&R, Security, PR, etc • Incident Command System • Most critical component • Defines borders of other components • System Integration • Interagency coordination, communications • Human Resources • Others as defined by local plan

12. “Smart” observers “Smart” casualties Post-drill interviews Group debriefing Self-assessment Photo, audiotape and/or videotape assessment Checklists Scribes Victim based recording Cross-evaluation Provider and/or victim flow tracking Computer modeling 4. Select Appropriate Evaluation Methods and Instruments Based Upon Goals and Resources:

13. Combined Approach to Evaluation • Clinical Care • Trained observers • Provider recording (triage tags, etc.) • “Smart” patients • Standardized Victim Scenarios • Drill flow • Victim Movement • Victim/Provider interactions • Entrance/exit observers • Scene Diagrams • Photos/Videotape analysis • Incident Command, System Integration • Direct observation difficult • Pattern Analysis of response • Self assessment & “cross-evaluation” • Plan Compliance (Qualitative Methods) • Checklists • Evaluators’ narrative comments • Videotape review • Debriefing comments • Survey or structured interview of participants

14. Clinical Care Evaluation • Standardized Victim Scenarios • Clearly written victim descriptions • Expert panel consensus process: • Triage Level: Primary and Secondary Triage • Critical Actions: Scene, Treatment Area • Victims moulaged per pre-written scenarios • Scenarios may change during event • Victim Based Recording • “Smart” Victims • Observer Based Recording • 6 victims/observer maximum

15. 2 TRIAGE LEVEL ڤRed ڤ Yellow ڤ Green ڤ Black CRITICAL ACTIONS ڤOpen Airway ڤOxygen ڤVentilate ڤIntubation ڤNeedle Thoracostomy ڤChest Tube ڤDirect Pressure ڤIntravenous Line ڤTransfusion ڤ Wound Care ڤMedication ڤNothing ڤSpinal Immobilization ڤExtremity Immobilization DISPOSITION ڤ Operating Room ڤ Intensive Care ڤ Hospital ڤMorgue Vital Signs Mental Status: Pulse: Respirations: Capillary Refill: Skin Appearance: Patient Description: Age: Gender: Presentation: Physican Findings: Victim Scenario Card (Triage Area)

16. 16 11 18 12 10 5 6 15 3 8 7 13 Victims Bed Chair 2 Zone III 1 17 9 14 Zone I Zone II 19 20 4

17. Evaluation of Drill Flow • Assess Efficiency of Operations • Time-based Data Collection • Relative rather than absolute times important • Flow Measured at Key Locations (Bottlenecks) • Can capture all movements with exact times for Victims, Providers and Resources • Scene entry/exit, decon, triage, treatment areas • Clear Identification System Critical • Allows Pattern Analysis

18. Mean Response Times by Triage Level and Zone

19. Evaluation of ICS • Most Difficult Component to Objectively Evaluate • Multiple simultaneous communications and decisions • Impractical to directly observe, record time and content of every communication • Observers and/or scribes can capture pre-determined “key” communications and actions • Pattern Analysis of Drill Flow • Assess active management of response components • Assess system response to situational changes • Structured Self Assessment & “Cross-Evaluation” • Adequacy, clarity of communications • Understanding of functional roles

20. Unit Leader Survey 1- Strongly Agree, 2– Neutral, 3– Strongly Disagree

21. Drill Participant Survey 1- Strongly Agree, 2– Neutral, 3– Strongly Disagree

22. Closing Perspectives • Plan drills targeted to specific goals • Include evaluation in plans for all drills • Use evaluation methods congruent with goals of drill & available resources • Seek adequate support for stronger evaluation designs • Evaluation results should drive all future planning and training activities

23. Questions?