HSEEP Exercise
1 / 62

- PowerPoint PPT Presentation

  • Uploaded on

HSEEP Exercise Evaluation and Improvement. ODP’s Mission. Primary responsibility within the executive branch to build and sustain the preparedness of the US to reduce vulnerabilities, prevent, respond to, and recover from acts of terrorism (Homeland Security Act). ODP’s Responsibilities.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about '' - sandra_john

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Slide1 l.jpg

HSEEP Exercise

Evaluation and Improvement

Odp s mission l.jpg
ODP’s Mission

  • Primary responsibility within the executive branch to build and sustain the preparedness of the US to reduce vulnerabilities, prevent, respond to, and recover from acts of terrorism (Homeland Security Act).

Odp s responsibilities l.jpg
ODP’s Responsibilities

  • Grant programs for planning, equipment, training and exercises

  • National training program

  • National exercise program

Grant programs l.jpg
Grant Programs

  • State Homeland Security Program

  • Law Enforcement Terrorism Prevention Program

  • Citizen Corps Program

  • Urban Areas Security Initiative Program

  • Fire Fighter Assistance Program

State homeland security program l.jpg
State Homeland Security Program

  • Purpose: to enhance capacity of states and locals to prevent, respond to, and recover from terrorism

  • Provides funds for

    • Homeland security and emergency operations planning

    • The purchase of specialized equipment

    • CBRNE and cyber security training programs

    • CBRNE and cyber security exercises

    • State Homeland Security Assessments and Strategies

Law enforcement terrorism prevention program l.jpg
Law Enforcement Terrorism Prevention Program

  • Provide law enforcement communities with funds to support the following prevention activities:

    • Information sharing to preempt terrorist attacks

    • Target hardening

    • Recognition of potential or developing threats

    • Interoperable communications

    • Intervention of terrorists before they can execute a threat

    • Planning, organization, training, exercises, and equipment

Citizen corps program l.jpg
Citizen Corps Program

  • Provides funds to support Citizen Corps Councils with planning, outreach, and management of Citizen Corps program and activities

    • Form and sustain a Citizen Corp Council

    • Engage citizens in homeland security

    • Conduct public education and outreach

    • Develop and implement Citizen Corps programs

    • Coordinate Citizen Corps activities with other DHS funded programs and other federal initiatives

Urban areas security initiative program l.jpg
Urban Areas Security Initiative Program

  • Address the unique needs of large urban areas – 50 cities

  • Conduct jurisdictional assessment and develop Urban Area Homeland Security Strategy.

  • Funds for planning, equipment, training, exercise, and administration and operational activities related to heightened threat levels

Fire fighter assistance l.jpg
Fire Fighter Assistance

  • Protect public and fire fighters against fire and fire-related hazards

    • Fire fighting Operations and Safety

    • Fire Prevention

    • Fire fighting Vehicles

Strategy process overview l.jpg
Strategy Process Overview




Statewide Homeland

Security Strategy

State Assistance



Conducted at the

local and state levels

Created at the regional and state level

Created by ODP

State and Urban Area use strategy to identify & allocate all HS resources

END RESULT = Capability Improvements

Strategy participants l.jpg
Strategy Participants

  • State and local jurisdictions

  • All First Responder Disciplines

  • Fire Service

  • HazMat

  • Emergency Medical Services

  • Law Enforcement

  • Emergency Management

  • Public Safety Communications

  • Public Health

  • Health Care

  • Public Works

  • Government Administrative

  • Private Sector

  • Non-Profit/Voluntary Sector

Assessment overview l.jpg

Risk Assessment

Needs Assessment





Assessment Overview





Planning Factors

CBRNE* Scenarios





Agricultural Vulnerability Assessment

Shortfalls or “Gaps”

* CBRNE: Chemical, Biological, Radiological, Nuclear, Explosive

Threat assessment l.jpg

Risk Assessment

Threat Assessment

  • Who:

  • Local, state, and federal law enforcement officials

  • What:

    • Identify number of Potential Threat Elements (PTEs)

    • Identify threat factors (existence, violent history, intentions, WMD capability, and targeting)

    • Identify motivations (political, religious, environmental, racial, or special interest)

    • Identify WMD capabilities (CBRNE)





Agricultural Vulnerability Assessment

Vulnerability assessment l.jpg



Vulnerability Assessment

  • Who:

  • All response disciplines at local, state, and federal levels

  • What:

  • Identify critical infrastructure/ potential targets

  • Evaluate targets for:

    • Level of visibility

    • Criticality of target site

    • Impact outside of jurisdiction

    • Access to target

    • Target threat of hazard

    • Target site population capacity

    • Potential collateral mass casualties

Risk Assessment



Agricultural Vulnerability Assessment

Capabilities and needs planning l.jpg
Capabilities and Needs: Planning

  • The results from the risk assessment process (threat and vulnerability) provide a link to the capabilities and needs assessment process.

    • Planning

    • Organization

    • Equipment

    • Training

    • Exercises

State homeland security strategy l.jpg
State Homeland SecurityStrategy

  • Developed by State based on local needs

  • Provides blueprint for planning of homeland security efforts to enhance preparedness and for use of resources

State assistance plans l.jpg
StateAssistance Plans

  • ODP uses the strategies and needs assessment data to tailor and formulate a State/Metro Assistance Plan (SAP/MAP) for each state

  • A SAP/MAP is a blueprint for the delivery of ODP training, exercise, technical assistance and equipment services

National training program l.jpg
National Training Program

  • Training for federal, state and local homeland security professionals

  • Based on critical tasks to prevent, respond to or recover from a terrorist incident

  • Over 40 courses available

Odp training program l.jpg
ODP Training Program

  • ODP offers more than 40 courses (Examples)

    • Live chemical agents training – Center for Domestic Preparedness

    • Live explosives training – New Mexico Institute of Mining and Technology

    • Radiological and nuclear agents training – Nevada Test Site

    • Advanced emergency medical training using human patient simulators – Texas A&M

    • Training on bioterrorism – Louisiana State University

National exercise program l.jpg
National Exercise Program

  • Responsible for National Exercise Program

  • Threat and performance-based excises at federal, state, local, and international levels

  • Strategy and Exercise Planning Workshops to define exercise needs and plan for each state

Assess program success through exercises l.jpg
Assess Program Success Through Exercises

  • Performance measures for ODP’s grant, training, and exercise programs are tied to performance of critical tasks

  • Percent of jurisdictions that can perform critical tasks as demonstrated through exercises

    • 500,000+ population

    • 100,000+ population

    • 50,000+ population

Overview of hseep l.jpg
Overview of HSEEP

  • Threat- and Performance-based Exercises

  • Cycle of exercises

  • Increasing complexity

  • To improve preparedness

Hseep manuals l.jpg
HSEEP Manuals

  • Volume I: Program Overview and Doctrine

  • Volume II: Exercise Evaluation and Improvement

  • Volume III: Exercise Development

  • Volume IV: Sample Exercise Documents and Formats

Vol i hseep overview and doctrine l.jpg
Vol I: HSEEP Overview and Doctrine

  • ODP’s exercise and evaluation doctrine

  • Uniform approach for exercise design, development, conduct, and evaluation

  • Exercise design and implementation process

  • Suite of common scenarios (TBD)

Vol ii exercise evaluation and improvement l.jpg
Vol II: Exercise Evaluation and Improvement

  • Defines exercise evaluation and improvement process

  • Provides uniform set of evaluation guides

  • Defines data analysis process

  • Includes standardized After-Action Report template

Vol iii exercise development l.jpg
Vol III: Exercise Development

  • Defines exercise planning and design process

  • Provides guidance for the development and conduct of various types of exercises

Vol iv sample documents l.jpg
Vol IV: Sample Documents

  • Provides sample letters, planning documents, checklists, scenarios, etc.

  • Reduces development time for exercise design team

Exercise evaluation l.jpg

Assess preparedness at federal, state and local levels

Validate strengths and identify improvement opportunities, resulting in improved preparedness

Provide guide for resource allocations

Exercise Evaluation

Evaluation enhancements l.jpg
Evaluation Enhancements

  • Focus on performance of critical tasks and mission outcomes

  • Use of uniform evaluation tools

  • Enhanced data analysis

  • Debriefing meeting with key officials

  • Improvement Plan

  • Track implementation of improvements

  • Suite of common scenarios (TBD)

Exercise evaluation methodology development l.jpg
Exercise Evaluation Methodology Development

  • Exercise Evaluation Working Group

  • Builds on

    • Responder Guidelines

    • ODP exercise experience

    • CSEP and other programs

  • Will continue to enhance and improve

Exercise evaluation and improvement process l.jpg
Exercise Evaluation and Improvement Process

Evaluation Planning, Observation, and Analysis

Exercise Evaluation and Improvement Process Data Collection and Analysis

Step 1

Plan & Organize the Evaluation

Step 2

Observe the Exercise & Collect Data

Step 3

Analyze Data

Step 4

Develop After Action Report

Improving Preparedness

Step 5

Conduct Debriefing

Step 6

Identify Improvements

Step 7

Finalize After Action Report

Step 8

 Track Implementation

Levels of analysis l.jpg
Levels of Analysis

  • Performance is analyzed at three levels:

    • Task level

    • Agency/discipline/function level

    • Mission level (within and across communities)

Levels of analysis33 l.jpg

Task Level Performance

Answers the question: did the person or team do the right thing the right way at the right time?

Helps assess need for training, equipment, personnel, etc.

Task = work with measurable output that has utility

Levels of Analysis

Levels of analysis34 l.jpg

Agency/Discipline/Function Level Performance — Multiple teams

Answers the question: did the larger team or organization perform duties in accordance with plans and policies?

Levels of Analysis

  • Helps assess communication, coordination, planning budgets, etc.

Levels of analysis35 l.jpg
Levels of Analysis

  • Mission Level Performance

    • Answers the question: were the mission level outcomes achieved?

    • Addresses jurisdictional preparedness

      Outcomes = results

Mission outcomes l.jpg
Mission Outcomes

  • Prevention/Deterrence

  • Emergency Assessment

  • Emergency Management

  • Hazard Mitigation

  • Public Protection

  • Victim Care

  • Investigation/Apprehension

  • Recovery/Remediation


Emergency Response


Evaluation requirements l.jpg
Evaluation Requirements

  • Determine what outcomes will be evaluated, based on exercise objectives

  • Identify activities to be evaluated

  • Identify which functions should be observed

  • Determine where observations will take place

  • Identify the appropriate evaluation tools

Exercise evaluation guides l.jpg
Exercise Evaluation Guides

  • ODP has developed Exercise Evaluation Guides that:

    • Identify the activities that the evaluator should be observing

    • Provide consistency in tasks across exercises

    • Link individual tasks to disciplines and outcomes

The evalplan l.jpg

  • Exercise-specific information

  • Plans, policies, procedures, and agreements

  • Evaluator recruiting and assignments

  • Evaluator training and instructions

Recruiting and assigning evaluators l.jpg
Recruiting and Assigning Evaluators

  • Setting expectations – evaluators must be available for:

    • pre-exercise training and briefing

    • pre-exercise site visit

    • the entire exercise (hours to days)

    • post-exercise hot-wash

    • post-exercise data analysis (1 day)

    • contribution to the draft AAR

Recording observations l.jpg
Recording Observations

  • The emphasis is on Who? What? When? Where? How? Why?

  • Record observations through:

    • use of Evaluator Guides

    • blank sheets of paper

  • Collect exercise documents

Record significant activities l.jpg
Record Significant Activities

  • Initiating scenario events

  • Facility activities

  • Response actions

  • Key decisions made by Players

  • Deviations from plans and procedures

  • Completion time of events

Evaluator summary l.jpg
Evaluator Summary

  • Compile observations into chronological narrative of events

  • Describe outcomes achieved or not – use questions below and evaluation guides:

    • What happened?

    • What was supposed to happen?

    • If there is a difference, why?

    • What is the impact of that difference?

    • What should be learned from this?

    • What improvements might be recommended?

Data analysis l.jpg
Data Analysis

  • Conduct Hotwash

  • Develop timeline of significant events

  • Analyze performance:

    • Individuals

    • Teams/Functions

    • Outcomes

Hotwash l.jpg

  • Player Hotwash:

    • Usually held immediately following exercise play

    • Typically facilitated by the evaluator

  • Provides opportunity for:

    • Player self-assessment

    • An interactive discussion

    • Clarification of observed events

    • Assessment of exercise simulations

Timeline development l.jpg
Timeline Development

Make a team timeline of actions

Focus on significant


Identify the appropriate outcome for

each activity

Analysis of performance l.jpg
Analysis of Performance

  • Analysis of activities

    • What tasks were to be accomplished

    • How well were they performed

    • Root causes of differences between expected and actual performance

    • Recommendations

Root cause analysis l.jpg
Root Cause Analysis

1. Why did it happen?

2. Why did that happen?

3. Why was that?

4. And why was that?

5. And why was that?

6. And so on…

***Root Cause***

Each step must completely

explain the step above…

…down to the basic

underlying causal factor.

Integrated analysis l.jpg
Integrated Analysis

  • Allows further identification of:

    • Successes and best practices

    • New gaps and problems

    • Root causes

    • Recommendations for improvement

  • Compares observations from different locations and functions

Recommendations for improvement l.jpg
Recommendations for Improvement

  • Questions for identifying recommendations for improvement:

    • What training and/or equipment is needed?

    • What changes need to be made to plans and procedures, or organization structures?

    • What changes could be made to the management processes?

The after action report aar l.jpg
The After-Action Report (AAR)

  • Serves as feedback tool

  • Summarizes what happened

  • Identifies successes and recommendations for improvement

  • May include lessons learned to share with other jurisdictions

  • Help jurisdictions focus resources on greatest needs

After action report l.jpg
After-Action Report

  • Prepared in two stages:

    • Draft AAR – completed immediately after the exercise for review

      • Community adds improvement steps/corrective actions

    • Final AAR

Aar format l.jpg
AAR Format

  • Executive Summary

  • Part 1: Exercise Overview

  • Part 2: Exercise Goals and Objectives

  • Part 3: Exercise Events Synopsis

  • Part 4: Analysis of Mission Outcomes

  • Part 5: Analysis of Critical Task Performance

  • Part 6: Conclusion

  • Appendix A: Improvement Plan Matrix

Improvement process l.jpg
Improvement Process

  • Improving preparedness activities:

    • Conduct exercise debrief

    • Identify improvements

    • Finalize AAR

    • Track implementation

Step 5

Conduct Debriefing

Step 6

Identify Improvements

Step 7

Finalize After Action Report

Step 8

Track Implementation

Exercise debrief l.jpg
Exercise Debrief

  • Provides a forum for jurisdiction officials to:

    • Hear the results of the analysis

    • Validate the findings and recommendations in draft AAR

    • Begin development of Improvement Plan

Improvement plan l.jpg
Improvement Plan

  • Developed by local jurisdiction during debrief

  • Identifies how recommendations will be addressed:

    • What actions

    • Who is responsible

    • Timeline for completion

Finalize aar l.jpg
Finalize AAR

  • Improvement Plan is included in final AAR

  • Final AAR submitted to ODP through State Administrative Agency

Monitor implementation l.jpg
Monitor Implementation

  • ODP Exercise Management System (under development) will provide:

    • Centralized calendar of exercises across the country

    • Electronic submission of AAR/IPs to the SAA and ODP

    • Monitoring of Improvement Plan implementation

Sharing lessons learned l.jpg
Sharing Lessons Learned

  • Ready-Net – Web-based, secure information network

    • National repository for best practices and lessons learned

    • Accessible to approved users within the response community

    • Administered by the Memorial Institute for the Prevention of Terrorism

Benefits of hseep approach l.jpg
Benefits ofHSEEP Approach

  • Nationwide consistency

  • More useful after action reports and improvement plans

  • Ability of jurisdictions to focus resources on greatest needs


Exercise evaluation training course l.jpg
Exercise EvaluationTraining Course

  • 2 ½ days - Exercise Evaluation methodology

  • 6 sessions to train ODP staff and contractors as change agents (225 people)

  • Training for SAAs Feb-May 2004

  • ODP Exercise Design Course being revised to deliver consistent message

Goal for working group l.jpg
Goal for Working Group

  • Review and modify Exercise Evaluation Guides for Radiological and Biological attacks

    • Are the right tasks identified?

    • Do other tasks need to added?

    • Are the conditions and typical steps logical and complete?

    • Are the followup analysis questions the right questions to assess performance?