UNIT 16 Exercise Evaluation
Objectives • Describe the Evaluation Process • Identify objective based • evaluation concepts • Describe an evaluation team • Describe an evaluator checklist, • complete with objectives and points of review. • Identify evaluator skills, attributes and pitfalls • Describe the After Action Report process • Describe the Improvement Planning process
DEFINITION OF EXERCISE EVALUATION The act of reviewing or observing and recording exercise activity or conduct, applying the behavior or activity against exercise objectives, and noting strengths, areas for improvement, deficiencies, and other observations.
DEFINITION OF EXERCISE EVALUTOR EVALUATOR: An evaluator is an exercise participant who observes and documents those actions and decisions of the players that are directly related to the exercise objectives
THE CASE FOR SYSTEMATIC EXERCISE EVALUATION • Exercise Evaluation must be addressed consistently and systematically in emergency management exercises. • It is the Cost - vs - Benefit factor
SYSTEMATIC, OBJECTIVE-BASED EVALUATION DESIGN PROCESS • Formulation of an Evaluation Plan • Based on definition and tracking of exercise objectives • Produces written evaluations for each stated objective • Provides a basis for assessment and upgrading of the emergency management system • Includes evaluation forms that are based on the objectives & corresponds w/ the exercise design criteria • Depends on the use of qualified evaluators
EXERCISE PROCESS • Establish • the foundation • Design and Development • Conduct • Evaluation • Improvement Planning • EVALUATION • PROCESS • Select • Evaluation • Team • Director • Develop • Evaluation • Plan • Select, • Organize & • Train • Evaluators • Observe & Document Player Actions • Participate in Post-Exercise Critiques & Meetings • Develop Evaluation Reports • *Track Long-Term Follow-up • * - May Not Be Evaluator Responsibility EVALUATION TASKS IN THE OVERALL EXERCISE DEVELOPMENT PROCESS
PRE-EXERCISE • PHASE • EXERCISE • PHASE • POST-EXERCISE • PHASE • Select the Evaluation • Team Leader • Develop Evaluation • Plan • Select & Organize the • Evaluation Team • Conduct Team • Meetings • Training of Evaluators • Observe Assigned • Objectives • Document Actions • Coordinate • Evaluator Team • Achievement of • Exercise Objectives • are Assessed • Post-Exercise • Meetings Attended • Written Report is • Prepared THE EXERCISE EVALUATION PROCESS
PRE-EXERCISE TASKS OF THE EVALUATION TEAM DIRECTOR • Select the evaluation methodology, and if necessary, develop the appropriate evaluation forms • Develop the Evaluation Plan • Select and train evaluation team members
EXERCISE PHASE TASKS OF THE EVALUATION TEAM DIRECTOR DURING THE EXERCISE ENSURES EVALUATORS ARE: • In the right locations • Equipped with the proper & correct documentation & equipment • Observing & documenting player actions • Provided with backup, if needed
POST-EXERCISE TASKS OF THE EVALUATION TEAM DIRECTOR • Ensures evaluators assess achievement of exercise objectives • Coordinates involvement of evaluators in post-exercise meetings • Coordinates and reviews drafting of written reports • May play a role in long-term follow-up
EVALUATION METHODOLOGY DEFINITION • EVERY EXERCISE EVALUATION METHODOLOGY IS DEFINED BY: • Objectives to be demonstrated • Evaluator team size, critique process, & scheduling • Forms, checklists, and points of review for use by the evaluators • Information & logistics support including evaluator packets • After action follow-up
EXERCISE OBJECTIVES Serve as the basis for evaluation of the exercise Standards & Criteria (EOP, SOP) Systematic Exercise Evaluation Exercise Objectives
DISCUSSION QUESTION Where can standard lists of emergency management exercise objectives be obtained?
EXERCISE OBJECTIVES • Should be demonstrable with a reasonable commitment of resources • Should be defined so that related actions occur in one location
EXERCISE OBJECTIVES Should divide the exercise into discrete functions Direction & Control Fire/ Rescue Evacuation/ Shelter & Mass Care Communication Warning Alert/ Notification
EXAMPLES • EMS: Medical Command will Triage, Treat transport victims within 45 min • Hazmat: Will identify the product & initiate offensive containment within 90 min • Pub Info: PIO will coordinate • interagency information • releases per protocols • Res Mgnt: CP/EOC will • coordinate resources
TYPES OF OBSERVATIONAL DATA • DESCRIPTIVE: • Reporting everything which • is related to the assigned function • Usually reliable because it • requires little inference
TYPES OF OBSERVATIONAL DATA • INFERENTIAL: • Requires the evaluator to make inferences before recording data • Harder to collect reliable information
TYPES OF OBSERVATIONAL DATA • EVALUATIVE: • Requires evaluative judgments as well as inferences • Most complex and difficult to collect
EVALUTOR PACKETS – FORMS • Remind the evaluator what to look for • Prompt the evaluator to gather specific information in an organized manner
POINTS OF REVIEW Are used by the evaluators to determine whether or not objectives are successfully demonstrated
DISCUSSION QUESTION What actions or conditions would an evaluator look for to determine if this objective has been met? “ Demonstrate communications capabilities with all appropriate emergency response locations, organizations, and personnel.”
EVALUATION FORM DESIGN Keep the questions short and simple • What time did responders arrive?
EVALUATION FORM DESIGN KEEP THE FORM MANAGEABLE • Try to limit to a maximum of 15-20 Points of Review • May be more extensive for complex exercises • Indicates points of review for the exercise objectives
EVALUATION FORM DESIGN Keep layout and typeface (font) simple & bold Did the EMS units within 10 minutes Did the EMS units arrive at the Scene within 10 minutes of being dispatched?
EVALUATION FORM DESIGN Do not ask for known information Were the participants aware of the location of the EOC?
EVALUATION FORM DESIGN Do not ask questions the evaluator cannot answer If an evaluator is in the field, do not ask what time the call came into the dispatch center.
EVALUATION FORM DESIGN Objective specific narrative summaries give substance to basic information on the checklist portion of the form.
EVALUATION FORM DESIGN References help the evaluator in later assessment of the exercise objectives.
STATE OF OHIO TERRORISM EXERCISE AND EVALUATION MANUAL • Developed by the State of Ohio with Direct Support from the Office for Domestic Preparedness (ODP) • Nearly 18 months from concept to completion • Based on HSEEP Volume II • Cross-walked to HSEEP Volume II Tasks and EEGs • Utilized in-state for over 75 exercises in 2004 • Utilized for ODP Direct Support Exercises in Cincinnati (MLB) and Cleveland (UASI)
EVALUATOR PITFALLS Sometimes evaluators are not objective
MINIMIZING EVALUATOR FATIGUE How can evaluator fatigue be minimized?
EVALUATOR EFFECT The presence of evaluators may influence the behavior of the players because they know what the evaluators are looking for.
REDUCING EVALUATOR EFFECT When players know the evaluation report does not reflect unfavorably on individuals.
REDUCING EVALUATOR EFFECT Evaluator Be at the appropriate place when the players arrive.
EVALUATOR BIAS This type of bias refers to errors traceable to characteristics of the evaluator or the observed situation. What are some ways to avoid evaluator bias?
RATING ERRORS Data can be influenced by certain readily identifiable errors.
ERROR OF LENIENCY What a great job by all!!! Some evaluators will rate all actions positively.
ERROR OF CENTRAL TENDENCY An individual that will describe all activities as average to avoid making any difficult decisions is committing the Error of Central Tendency.
HALO EFFECT The “Halo Effect” is the tendency for an evaluator to form an early impression of an individual or an operation and permit this impression to influence his or her observations.
HYPERCRITICAL EFFECT I know something’s wrong here. When an evaluator believes that it is his or her job to find something wrong regardless of the players’ performance.
CONTAMINATION CONTAMINATION HAS TO DO WITH THE INFLUENCE OF THE EVALUATOR’S KNOWLEDGE OR EXPECTATIONS ABOUT CERTAIN ASPECTS OF THE EXERCISE Same animals as last year
DISCUSSION QUESTIONS • If evaluators need additional information, what sources other than direct questioning might be available? • What are some reasons why it would be beneficial to allow evaluators to intervene in the exercise? • Why would you not want evaluators to intervene?
EFFECTIVE QUESTIONING Intervene only when necessary What alternative questions might be asked to obtain the information sought?
EFFECTIVE QUESTIONING Avoid leading questions Was the notification process completed within the appropriate timeframe?
EFFECTIVE QUESTIONING Avoid prompting questions Have you begun evacuation of the affected area yet?