Evaluating Tabletop Exercise Effectiveness

Evaluating Tabletop Exercise Effectiveness

by | Sep 13, 2018 | Resources

Despite the number of years tabletop exercises have been utilized for preparedness, only unreliable formal evaluation systems have existed until the creation of the DHS’s/FEMA Homeland Security Exercise and Evaluation Program (HSEEP) in 2005. This has been problematic and is characterized in a statement by Department of Homeland Security’s Office of Inspector General in an Executive Summary regarding DHS Efforts to Address Lessons Learned written in the Aftermath of Top Officials Exercises:

“Since the first Top Officials exercises in 2000, neither a process for tracking weaknesses and how those weaknesses were resolved, nor a method for identifying and analyzing trends in corrective actions or significant lessons learned has been established. As a result, federal, state, local, and territorial agencies were unclear regarding the implementation of suggested improvements following preparedness exercises.”

FEMA developed a standardized program that includes common terminology for exercise design, development, conduct, evaluation, and improvement planning.

One of benefit from this program is support organizations can achieve objective assessments of their capabilities. Strengths and areas for improvement are identified, corrected, and shared appropriately prior to a real incident.  This can be a strength of the program but must still be monitored and utilized effectively.  

What’s the alternative?

Even in the post-HSEEP era, tabletop exercises that are no-fault or are not considered an HSEEP exercise, include evaluations that are conducted in the form of collecting qualitative metrics through the use of questionnaires, narratives, hotwashes (post exercise discussions), lessons learned, logs, checklists, or surveys. A lot of progress has been made over the last two decades to improve these techniques and forms, and great data has been gathered from these methods. But do these techniques and forms answer the question on whether the exercise was effective? Do they help identify any expected results from exercise objectives? Unexpected results? This analysis will be looked at further in the evidence section.

Because the point of tabletop exercises is to evaluate policies, plans, and procedures, it is vital that metrics are collected. Questionnaires or surveys are able to gather response from participants or players, but do they provide the end user any meaningful data that determines if the exercise was effective?

Common Evaluation

Common to all tabletop exercises is the debriefing portion of the event, commonly called “the hotwash”. A hotwash is an opportunity for participants to provide their inputs on how well the exercise went, what plans or procedures should be changed, lessons learned, and make commitments on changes they see are appropriate.  It is common for an exercise to be followed by an evaluation meeting and may include an after-action report stating findings of the evaluation team and the effectiveness of the exercise. It serves as the basis for planning future exercises, upgrading contingency plans, and taking corrective actions.

One of the greatest qualitative metrics gained during the debriefing or hotwash are the lessons learned. But these lessons learned could be irrelevant such as the lessons learned from the “Hurricane Pam” exercise conducted in 2005 prior to Hurricane Katrina, if they have not been properly implemented and communicated. From the failures in information gathering and sharing, progress has been made in evaluating exercise effectiveness with the implementation of HSEEP.

Next Steps Toward Identifying Effectiveness

Substantial progress has been made in evaluating tabletop exercises, but more could be done.  For one, it would be beneficial to gather more than just qualitative data. Quantitative data would be extremely beneficial for both private and public organizations. This data could be used to forecast improvement in response times and effort, identifying resource needs and training when gaps are identified, as well as aligning training budget concerns.

Secondly, the HSEEP program only applies to DHS-funded exercises and is not necessarily utilized for no-fault or exercises conducted outside of HSEEP.  Furthermore, HSEEP may not be the solution.  Exercises that are mandated to be evaluated such as HSEEP, have seen their share of problems which are highlighted in the Congressional Research Service Report for Congress titled “Homeland Emergency Preparedness and the National Exercise Program: Background, Policy Implications, and issues for Congress”.  Noted in the report was the following:

The identification of capabilities on which to build through a public AAR, as required by the HSEEP method, may raise challenges if exercise participants have not adequately exercised their plans, or are concerned about potential consequences as a result of negative evaluations. As a result, there may be incentives for some exercise planners to understate exercise objectives, overstate the extent to which those objectives are met, or to downplay or omit deficiencies that are identified.  Any of those approaches arguably undermines the effectiveness of the exercise as tools to prepare for an incident, or to evaluate an entity’s capacity to respond to an incident. 

This report does point out the possibility of a potential bias or incentive for planners and participants to not fully identify problems when an exercise is evaluated.  This indication of bias may provide explanation for the need of no-fault exercises which essentially allows all participants and players to work through the exercise in an open environment without penalty.  The report further indicated:

The HSEEP method does not provide common benchmarks or metrics to apply in the evaluation of an exercise.  Moreover, under the HSEEP method, exercises are typically evaluated by the same group that designs the exercise.  This approach, which extends beyond the National Evaluation Program to any entity that uses the HSEEP method, may be problematic if the evaluators fail to critically asses their own program.

So is no evaluation the solution? One reasonable answer appears to be allowing for self assessments to be conducted similar to what is done in no-fault tabletop exercises, but can these types of evaluations be done effectively?  To determine if this is effective, it is necessary to analyze the current forms of evaluation for no-fault tabletop exercises.

Resources

PIO Go Kit

PIO Go Kit

Download this FREE resource to help prepare your Public Information Officer (PIO) Go Kit to be ready for a crisis at a moments’ notice.

CDC Plain Language Resources

CDC Plain Language Resources

Tools and resources from the Centers for Disease Control on communicating your message clearly and affectively to your audience the first time.

The Importance of Conducting Exercises

The Importance of Conducting Exercises

by | Aug 2, 2018 | Resources

Significant events over the past three decades have put pressure on the United States government to exercise response and preparedness.  Attacks both domestically and internationally have pushed legislation into domestic resilience, combating terrorism, incident command, and disaster preparedness.  In the 1980’s, the U.S. saw a rise of state-sponsored terrorism that included the following:

  1. In 1983, the United States embassy in Beirut, Lebanon was bombed, killing 63 people, mostly embassy and CIA staff members, as well as several soldiers and one Marine. 17 of the dead were Americans.

  2. In 1984, the Achille Lauro cruise ship was seized by a terrorist group known by the Palestine Liberation Front (PLF), holding 700 hostages, mostly Americans; 1 death.

  3. In 1985, TWA 847 flight hijacked; 1 death when a U.S Navy Diver was killed and tossed on to the Tarmac in Beirut, Lebanon.

  4. In December 1988, Pan Am 103 was bombed and exploded over Lockerbie, UK, killing all 270 people on board residing from 21 different countries.

In 1985 Vice President George Bush created “Vice President’s Task Force on Combating Terrorism” as the first end-to-end review focused on joint/interagency CT response assets coordination.  This review resulted in the passing of National Security Decision Directive 207 on Jan 20, 1986, where the U.S. National terrorism program first defined:  “U.S. Program on Combating Terrorism” crisis response to incidents overseas, ultimately guiding U.S. strategy for the next decade. Terrorism and other disasters continued to be the catalyst to push legislation and directives over the next 25 years, including the passing of several presidential policy directives that identified the need for better response and preparedness by first responders and follow-on response during a crisis.  Some of the legislation and policy directives specifically site the need for conducting tabletop exercises to help prepare for similar disasters.

Terrorism

In 1996, in response to the Tokyo sarin gas attacks and the Oklahoma City bombing, President Bill Clinton passed Presidential Policy Directive (PPD) 39.  Essentially, it stated it is the policy of the United States to “deter, defeat and respond vigorously to all terrorist attacks on our territory and against our citizens, or facilities, whether they occur domestically, in international waters or airspace or on foreign territory… the U.S. shall pursue vigorously efforts to deter and preempt, apprehend and prosecute, or assist other governments to prosecute, individuals who perpetrate or plan to perpetrate such attacks.”

Around that same time, many Congressional committees and subcommittees met on topics such as National Security and Combating Terrorism.  One of the programs discussed that demonstrated the expanded roll of the government in domestic preparedness is the Nunn-Lugar-Domenici Preparedness Program.  This program rose from the Defense Against Weapons of Mass Destruction Act of 1996 signed by President Clinton. This program looked to provide training for possible incidents involving terrorists using Weapons of Mass Destruction (WMD).  From this, the Department of Defense and Federal agencies were provided millions of dollars to conduct training, including tabletop exercises, in140 cities across the U.S.

Disasters and National Level Exercises

In 2005, Hurricane Katrina was considered the most destructive and costly hurricane to hit the United States, with damage estimated at $125 billion and 1,500 deaths across four states. Part of the blame lies on what appeared to be poor communication and response to this disaster, the Committee on Homeland Security and Governmental Affairs cited the failure of government at all levels to plan, prepare for, and respond aggressively to the storm.  Specifically, they stated that four overarching factors contributed to the failures.  These are:

  1. Long-term warnings went unheeded, and government officials neglected their duties to prepare for a forewarned catastrophe;

  2. Government officials took insufficient actions or made poor decisions in the days immediately before and after landfall;

  3. Systems on which officials relied on to support their response efforts failed; and

  4. Government officials at all levels failed to provide effective leadership.

The report stated that preparation plans and response were inefficient and not well devised.  Additionally, the report stated that the government had been insufficiently conducting training and exercises.  In 2005, DHS assumed full responsibility for planning, conducting, and after-action reporting of the National Exercise Program, known then as the Top Officials exercises or TOPOFF exercises.  In April 2005, DHS had implemented TOPOFF3 or the third tabletop exercise in the series which was designed to identify vulnerabilities in the Nation’s domestic incident management capability including the structure of the National Response Plan (NRP).  The NRP originated from Homeland Security Presidential Directive – 5 and was directed by President Bush to align Federal coordination structures, capabilities, and resources into a unified, all-discipline, and all-hazards approach to domestic incident management.

Exercises needed if done effectively

In response to the exercise, the Department of Homeland Security (DHS) Inspector General stated in November 2005: “the exercise highlighted – at all levels of government – a fundamental lack of understanding for the principles and protocols set forth in the NRP and the [National Incident Management System] NIMS.”  This identified confusion provoked discussion and demonstrated the importance of conducting exercises.  The absence of exercises in the NRP meant that there were no further formal opportunities to understand potential problems and to incorporate lessons learned into the NRP.

From these gaps and failures, DHS through the Federal Emergency Management Agency (FEMA) dropped the NRP in exchange for the National Response Framework (NRF). The National Response Framework (NRF) is a guide to how the Nation responds to all types of disasters and emergencies. It is built on scalable, flexible, and adaptable concepts identified in the National Incident Management System to align key roles and responsibilities across the Nation. This Framework describes specific authorities and best practices for managing incidents that range from the serious but purely local to large-scale terrorist attacks or catastrophic natural disasters. The National Response Framework describes the principles, roles and responsibilities, and coordinating structures for delivering the core capabilities required to respond to an incident and further describes how response efforts integrate with those of the other mission areas.

 The NRP was cited as being “insufficiently national in its focus… and …should speak more clearly to the roles and responsibilities of all parties involved in response.” This NRF is further covered under the signing of PPD 8 by President Obama, which in essence focused on an integrated, all-of-Nation, capabilities-based approach to all-hazard preparedness which can include, but are not limited to, the use of tabletop exercises as a form of preparation.

One thing Katrina did was reveal the impact that a lack of an effectively trained and exercised plan, as well as not practicing the interoperability of communications will further undermine the response. As part of the Committee on Homeland Security and Governmental Affairs, several recommendations were made to improve response, coordination and preparedness.  Of those, the committee recommended that “Federal departments and agencies should be required to conduct exercises to ensure that their plans are continually revised and updated,” as well as “emergency agencies at the federal, state, and local levels of government, as well as first-responder groups outside of government, should receive regular training on NRP and NIMS.” It is important to note that the NRP is considered the foundation of the NRF and it built upon, not entirely rebuilt, the national framework previously established.

Despite these failures, not all was a loss when, in 2005, DHS developed the Universal Task List (UTL) and Target Capabilities List (TCL).  The UTL helps exercise participants and planners by describing incident management tasks to be performed and provide them with a standardized reference for all levels of government and the private sector. The TCL contains capabilities that various levels of government need to develop and maintain to prevent, respond to, and recover from a terrorist attack or major disaster.  These two lists have proven useful to this day in all types and levels of exercise.

Exercise Progress

Through the lessons learned in the 1990s and the first decade of this century, exercises have been identified as a valuable tool for preparedness.  These lessons have shown the importance of addressing command and control during a crisis situation and working through scenarios such as a terrorist WMD incident or a major storm.  This progress has not always been so positive.  The final after-action report recommendations from the TOPOFF3 exercise failed to include improvement planning to address remedial needs and corrective action procedures which were not a part of the original evaluation.  It only informed participating departments and agencies of existing problems, and encouraged improvements in agency prevention, response, and recovery capabilities.  Furthermore, after-action reports, best practices, and lessons learned from the TOPOFF3 exercise have not been disseminated to a broad national audience.

Despite some of the problems stated, there is still great need for response assets to exercise their plans and prepare for disasters. For the response assets to take part in these exercises, it comes at the cost of government spending.  As Sequestration is taking effect and budgetary uncertainty lingers, training value needs to be maximized for ultimate effectiveness.  One cost-effective solution to providing an exercise that helps identify gaps or vulnerabilities in plans and preparedness are tabletop exercises. Tabletop exercises have evolved to incorporate the most important objectives and have been proven to be effective at training personnel without a large cost or resource commitment.  The difference in costs between a field exercise that requires the deployment of assets and a tabletop exercises can be as much as ten times. Despite this, field exercises are still necessary to ensure front line response and command post officials are able to handle their tasks and responsibilities in a stress added situation. So, can tabletop exercises provide the training and exercising of plans and procedures needed for response assets to prepare for a disaster?  Furthermore, the question that ultimately surfaces is how do we know this to be true from a discussion-based exercise?

Resources

PIO Go Kit

PIO Go Kit

Download this FREE resource to help prepare your Public Information Officer (PIO) Go Kit to be ready for a crisis at a moments’ notice.

CDC Plain Language Resources

CDC Plain Language Resources

Tools and resources from the Centers for Disease Control on communicating your message clearly and affectively to your audience the first time.

Types of Exercises and Training

Types of Exercises and Training

by | Jun 4, 2018 | Resources

As seen in human-caused events such as the 9/11 attacks, the Virginia Tech, Columbine, and Aurora, CO. shootings, as well as natural events such as “Snowmageddon,” the name used to describe the various blizzards impacting parts of the world in the last four years, hurricanes, and earthquakes, response assets are tasked with handling the crisis and consequence results.  To prepare for numerous scenarios, response assets take part in all forms of training that range from Discussion-Based to Operations-Based exercises which include the actual deployment of resources.  Each form of training exercise has unique goals and characteristics of conduct.  Below, table 1 describes discussion-based exercises and table 2 describes operations-based exercises.

Discussion-Based Exercises

Operations-Based Exercises

Exercises play a vital role in preparedness by enabling stakeholders to test and validate plans and capabilities, and identify both capability gaps and areas for improvement. A well-designed exercise provides a low-risk environment to test capabilities, familiarize personnel with roles and responsibilities, and foster meaningful interaction and across organizations. Exercises bring together and strengthen the responders in their efforts to prevent, protect/deter, mitigate, respond to, and recover from all hazards. Overall, exercises are cost-effective and useful tools that help the nation or an organization practice and refine our collective capacity to achieve the core capabilities preparedness goals.

Because most of the expertise utilized in the contribution to this paper originates from tabletop exercise experience, this paper will focus on tabletop exercises and pose the question of whether they are an effective form of preparing response assets. Tabletop exercises can be as much as 10 times less expensive than a full-scale exercise and may be able to address issues and plans with stakeholders that have the ability to make appropriate organizational changes.  Additionally, by solely focusing on tabletops, the successes and limitations of these exercises can be thoroughly examined and evaluated for their effectiveness.