SummitET Appoints Pamela Piersanti as Vice President of Programs and Strategy

SummitET Appoints Pamela Piersanti as Vice President of Programs and Strategy

September 23, 2019 – ST, PETERSBURG, FL.  Summit Exercises and Training® (SummitET®), a preparedness solutions company®, announces the appointment of Pamela Piersanti as Vice President of Programs and Strategy.  

Pamela’s appointment will provide SummitET’s private sector and governmental customers with the highest level of focus to help them build and improve capabilities necessary to prevent, protect against, mitigate the effects of, respond to, and recover from all-hazard threats. In her new role, Pamela will oversee SummitET programs, delivering cutting edge tailored preparedness solutions.

“Our programs and strategic focus are at the heart of everything we do at SummitET and provide our customers immense value,” says John Duda, Chief Executive Officer. “The appointment of Pamela to this new role demonstrates our commitment to our customers to meet and exceed our customer’s expectations and

continue operating our programs at the highest level. Pamela’s diverse expertise in law enforcement and managing national preparedness programs across critical infrastructure and threats involving Weapons of Mass Destruction make her the ideal person to lead our programs.” 

With twenty-seven years of experience, Pamela recently retired from the FBI as a Supervisory Special Agent and Manager for the FBI’s Weapons of Mass Destruction Directorate. Her key accomplishments included significant contributions to Presidential counterterrorism national policy and leading the development of the National Prevention Framework and Federal Interagency Operational Plan that describes what the whole community – from community members to senior leaders in government – should do upon credible, specific, and imminent terrorist threats or ongoing attacks against the United States. Before joining the FBI, Pamela served as an Assistant Special Agent in Charge of the Naval Criminal Investigative Service and as a Special Agent of the U.S. Bureau of Alcohol, Tobacco, Firearms, and Explosives.  She also has an M.A. degree in Criminology with Leadership in WMD, and a B.S. degree in Forensic Science.  

“I am eager to take on this new role and lead an amazing team of preparedness experts,” commented Pamela. “One of our greatest strengths is the experience and expertise of our employees comprised of former members of U.S. departments and agencies, and state and local governments responsible for reducing risk, building preparedness capability, and affording an effective threat response. We pride ourselves on our proactive approaches to build a system of preparedness that is designed to mitigate threats we face today and those we will undoubtedly face tomorrow.”

About SummitET
Summit Exercises and Training® is a U.S. based, veteran-owned preparedness solutions company that provides proven full spectrum solutions to threats facing the U.S. and international governmental and non-governmental organizations, as well as private entities through industry-leading experts and impeccable customer service. Our diverse and highly trained team support our corporate and governmental partners in developing solutions that address issues of critical infrastructure, radiological and emergency preparedness, cybersecurity, workplace safety and active shooter programs, counterterrorism, and other disaster-related training.

Media Contact
Ron Edmond
RonEdmond@summitet.com
Phone: 865-567-2248

SummitET News

SummitET Celebrates 10 Years of Success and Innovation

SummitET Celebrates 10 Years of Success and Innovation

St. Petersburg, FL – Summit Exercises and Training LLC (SummitET®), a Preparedness Solutions Company®, is proud to announce the celebration of its 10th anniversary this month. Since 2013, SummitET has been a driving force in the preparedness services industry,...

read more
HPS Specialist in Radiation Protection

HPS Specialist in Radiation Protection

HPS Specialist in Radiation Protection

If you are a member of the public or the media, a health physicist, or a scientist in a related field, you can either find answers or submit questions to HPS Ask the Experts.

HSP Radiation Fact Sheets

Health Physics Society Fact Sheets are available as portable document format (PDF) files.

Resources

Natural Disasters Take a Toll on the US Economy: and it’s Only Getting Worse

Natural Disasters Take a Toll on the US Economy: and it’s Only Getting Worse

Bridging the Gap Episode 4: Strategic Communications for Higher Education: Sensitive and Controversial Issues

In this podcast episode, SummitET’s team of Strategic Communications experts discuss crisis communications fundamentals that higher education institutions can incorporate when addressing controversial and sensitive issues with their stakeholders.

Bridging the Gap Episode 3, Part 1: Improving Exercise Design and Evaluation; the XF Score

In this podcast episode, SummitET preparedness experts and former DHS Acting Secretary and FEMA Administrator discuss the XF Score™️, a rubric for analyzing the effectiveness of tabletop exercises and trainings for crises.

Active Shooter Incidents Over Time

“Active shooters” were not always considered cause for alarm in the United States. This post examines the trend of active shooter incidents over time and offers preparedness solutions to mitigate the threat.

Bridging the Gap Episode 2: Stolen Radiation Sources

SummitET Subject Matter Experts (SMEs) discuss the emerging threat of stolen radiation sources.

Bridging the Gap Episode 1: Bioterrorism and Ricin

SummitET Subject Matter Experts (SMEs) discuss the possible implications of a ricin bomb as a Weapon of Mass Destruction (WMD).

What Your Radiological Communications May Be Missing

When communicating effects of a radiological incident, it’s important to have this one thing to effectively communicate to the public.

Build Organizational Resiliency Within Strategic Communications

The APSTER Process is a SummitET proprietary process for building organizational resiliency and preparedness within strategic communications.

Integrating Preparatory Consequence Management and Law Enforcement Operations During “Left of Boom” Terrorist Threats

A new planning approach enabling all levels of government to integrate their operations to maximize their ability to stop the terrorist.

Evaluating Tabletop Exercise Effectiveness

Evaluating Tabletop Exercise Effectiveness

Despite the number of years tabletop exercises have been utilized for preparedness, only unreliable formal evaluation systems have existed until the creation of the DHS’s/FEMA Homeland Security Exercise and Evaluation Program (HSEEP) in 2005. This has been problematic and is characterized in a statement by Department of Homeland Security’s Office of Inspector General in an Executive Summary regarding DHS Efforts to Address Lessons Learned written in the Aftermath of Top Officials Exercises:

“Since the first Top Officials exercises in 2000, neither a process for tracking weaknesses and how those weaknesses were resolved, nor a method for identifying and analyzing trends in corrective actions or significant lessons learned has been established. As a result, federal, state, local, and territorial agencies were unclear regarding the implementation of suggested improvements following preparedness exercises.”

FEMA developed a standardized program that includes common terminology for exercise design, development, conduct, evaluation, and improvement planning.

One of benefit from this program is support organizations can achieve objective assessments of their capabilities. Strengths and areas for improvement are identified, corrected, and shared appropriately prior to a real incident.  This can be a strength of the program but must still be monitored and utilized effectively.  

What’s the alternative?

Even in the post-HSEEP era, tabletop exercises that are no-fault or are not considered an HSEEP exercise, include evaluations that are conducted in the form of collecting qualitative metrics through the use of questionnaires, narratives, hotwashes (post exercise discussions), lessons learned, logs, checklists, or surveys. A lot of progress has been made over the last two decades to improve these techniques and forms, and great data has been gathered from these methods. But do these techniques and forms answer the question on whether the exercise was effective? Do they help identify any expected results from exercise objectives? Unexpected results? This analysis will be looked at further in the evidence section.

Because the point of tabletop exercises is to evaluate policies, plans, and procedures, it is vital that metrics are collected. Questionnaires or surveys are able to gather response from participants or players, but do they provide the end user any meaningful data that determines if the exercise was effective?

Common Evaluation

Common to all tabletop exercises is the debriefing portion of the event, commonly called “the hotwash”. A hotwash is an opportunity for participants to provide their inputs on how well the exercise went, what plans or procedures should be changed, lessons learned, and make commitments on changes they see are appropriate.  It is common for an exercise to be followed by an evaluation meeting and may include an after-action report stating findings of the evaluation team and the effectiveness of the exercise. It serves as the basis for planning future exercises, upgrading contingency plans, and taking corrective actions.

One of the greatest qualitative metrics gained during the debriefing or hotwash are the lessons learned. But these lessons learned could be irrelevant such as the lessons learned from the “Hurricane Pam” exercise conducted in 2005 prior to Hurricane Katrina, if they have not been properly implemented and communicated. From the failures in information gathering and sharing, progress has been made in evaluating exercise effectiveness with the implementation of HSEEP.

Next Steps Toward Identifying Effectiveness

Substantial progress has been made in evaluating tabletop exercises, but more could be done.  For one, it would be beneficial to gather more than just qualitative data. Quantitative data would be extremely beneficial for both private and public organizations. This data could be used to forecast improvement in response times and effort, identifying resource needs and training when gaps are identified, as well as aligning training budget concerns.

Secondly, the HSEEP program only applies to DHS-funded exercises and is not necessarily utilized for no-fault or exercises conducted outside of HSEEP.  Furthermore, HSEEP may not be the solution.  Exercises that are mandated to be evaluated such as HSEEP, have seen their share of problems which are highlighted in the Congressional Research Service Report for Congress titled “Homeland Emergency Preparedness and the National Exercise Program: Background, Policy Implications, and issues for Congress”.  Noted in the report was the following:

The identification of capabilities on which to build through a public AAR, as required by the HSEEP method, may raise challenges if exercise participants have not adequately exercised their plans, or are concerned about potential consequences as a result of negative evaluations. As a result, there may be incentives for some exercise planners to understate exercise objectives, overstate the extent to which those objectives are met, or to downplay or omit deficiencies that are identified.  Any of those approaches arguably undermines the effectiveness of the exercise as tools to prepare for an incident, or to evaluate an entity’s capacity to respond to an incident. 

This report does point out the possibility of a potential bias or incentive for planners and participants to not fully identify problems when an exercise is evaluated.  This indication of bias may provide explanation for the need of no-fault exercises which essentially allows all participants and players to work through the exercise in an open environment without penalty.  The report further indicated:

The HSEEP method does not provide common benchmarks or metrics to apply in the evaluation of an exercise.  Moreover, under the HSEEP method, exercises are typically evaluated by the same group that designs the exercise.  This approach, which extends beyond the National Evaluation Program to any entity that uses the HSEEP method, may be problematic if the evaluators fail to critically asses their own program.

So is no evaluation the solution? One reasonable answer appears to be allowing for self assessments to be conducted similar to what is done in no-fault tabletop exercises, but can these types of evaluations be done effectively?  To determine if this is effective, it is necessary to analyze the current forms of evaluation for no-fault tabletop exercises.

Resources

Analyzing Tabletop Exercise Effectiveness

Analyzing Tabletop Exercise Effectiveness

To assess the current forms of evaluating the effectiveness of no-fault exercises and offer alternatives, several individuals with a great deal of experience participating in and planning tabletop exercises were interviewed.  This cadre includes former senior representatives with the Department of State, a Senior Policy Advisor to the White House and Department of Energy, a Deputy Under Secretary of the National Nuclear Security Administration, an Exercise Director with the Federal Bureau of Investigations, a former Navy SEAL, an Assistant Administrator for Protective Services and Security with the National Aeronautics and Space Administration, a hospital Emergency Manager, and a Senior Exercise Planner.  These interviews yielded a broad and detailed view on how tabletop exercises are currently evaluated and how they could improve.  Moreover, over 100 articles, governmental committee and subcommittee notes/reports, journals, after-action reports, and thesis were reviewed to gain a strong knowledge of how tabletop exercises are evaluated and whether they are effective.

Are qualitative assessments good enough?

Although no numeric rating may result from qualitative assessments and responses, one can certainly provide solid qualitative evidence that an exercise was effective through subjective post exercise evaluations/critiques provided by exercise participants.  Though it can be subjective, the individuals interviewed also provided examples of how one can determine that an exercise was effective solely through qualitative response.  Common qualitative examples discussed in the interviews as well as actual results from tabletop experience include:

  • Team-building and familiarity among response assets and leadership –The tabletop exercise provides an opportunity for first responders and follow-on response to meet one another, sometimes for the first time, and get to know/build trust among one another. “Almost impossible to measure, but a tabletop exercise is invaluable because it is the relationships built between responders in an emergency.  [A tabletop exercise] builds a trust between response.”

  • Knowledge gained of roles, responsibilities, and assets among responding parties – Deputy Under Secretary for the National Nuclear Security Administration Dr. Steve Aoki stated that taking part in tabletop exercises helped him in response guidance to Fukushima.  The exercises he participated in provided him with the knowledge on the various response assets available and what assets could be called upon during a crisis/disaster.  Additionally, the tabletop provided him with a venue to work through various scenarios.  Furthermore, the Assistant Administrator at NASA, Mr. Mahaley, stated that he took part in a tabletop exercise that involved him contacting the White House during a disaster.  This experience prepared him for an actual call that was needed while he was acting Director of Security for Energy during the blackout of 2003 that impacted much of the Northeast United States.  He stated that experience provided him the opportunity to work through how a phone call to the White House would take place and understand who needed to be included in the call.

  • Post-exercise lessons learned – things learned that were otherwise not know prior to the exercise.  “For the after action review to be effective, the opportunity to incorporate recommended changes to site response plans and procedures should be a goal.”A gap in a current security plan or procedure or a lack of understanding of a particular substance/organism often is identified through the course of tabletop exercise.  This often results in a change in a plan/procedure or a group of people being more comfortable with response.

  • Knowledge of a particular threat – i.e. group, source/material, attack.  As stated from the conclusion of Maryland’s pandemic influenza preparedness exercise – It [the tabletop exercise] served to engage the emergency response community and address the issues of incident command and how pandemic planning fits with the “all hazards” approach.  The exercise also educated key partners and stakeholders, through an experiential approach, about the potential severe consequences of pandemic influenza, and it provided a forum to “drill down” beyond the current state plan and identify additional critical local planning activities that are needed.  Instructive insights and lessons were gained from the exercise that should bolster further planning efforts in Maryland, not only for pandemic influenza, but also for bioterrorism and other public health disasters.

  • Exercising plans in place – a tabletop exercise provides a venue for response to actually practice the plans and procedures in place to ensure they fully understand said plan/procedure and/or response to a disaster. Mr. Mahaley stated “You do what you are trained to do.  In real life, you are going to react how you are trained.  In my 40 years of experience, tabletop exercises provide the most effective form of training.”

This qualitative information is extremely vital and shows a tabletop exercise is effective, but this information is not always easy to gather.  In almost every interview that was conducted, a common theme regarding the best way of obtaining great qualitative response was if the exercised remained no-fault or non-attributional, allowing an open, honest environment.  The reasons stated included that assets are more likely to admit faults, vulnerabilities, or lack of understanding or a shortfall in a plan/procedure if they are not worried about their job. This makes sense considering that participants may feel more comfortable speaking if they are not being graded.  So, if a no-fault tabletop exercise yields the best qualitative responses, can it also provide quantitative results to determine effectiveness?

Quantitative assessments

Raw, not well developed, quantitative assessments currently exist in the tabletop community but they are not well known and there is no standard.  It would be useful to have standardized quantitative assessments to assist public and private organizations determine if the money being spent by their organization is going to good use and the tabletop exercise is worth attending.  With a lack of available quantitative metrics, it is prudent to look at ways to quantify the results of a tabletop exercise to compliment the qualitative data.  Furthermore, for the purpose of this paper, it was stated that exercises may be more effective being non-attributional, so we will also mull over this as the type of tabletop exercise being considered.

Suggested Metrics

Three forms of quantitative assessments should be considered to assist government agencies and private organizations with determining effectiveness for a no-fault tabletop exercise.  These include a pre/post test combination to help identify the percentage of improvement, a numeric count of observations during and post exercise, and a rubric as an assessment tool.

Conducting a pre and post test among players and observers (observers typically consist of other invited responders not sitting at the player’s table) is a way to gauge a level of improvement in understanding, knowledge, and collaboration.  Participants would take a test to indicate their understanding of response to a disaster, level of knowledge on the particular threat, and how well they know who would be responding/in charge of a particular incident.  Then following the tabletop exercise, the participants would take the same test and the results would be compared between the two.  From that, a level of improvement could be gathered providing some quantitative gauge of exercise effectiveness.

Another potential way to gather quantitative data from a tabletop exercise could be in the form of counting the number of observations either during an exercise or post exercise.  During a no-fault exercise, an unbiased observer could be included not to grade or place fault, but to instead count the number of observations that a participant learned something, a vulnerability was identified, a gap in a plan or procedure was identified, or an agency stated they were unaware of a particular response or response asset.  The observations could then be sorted and tallied.  This could also be done post- exercise counting the number of changes made to plans, policies, or procedures, as well as anything else that may have resulted from the exercise experience.

The final suggestion that should be considered is an original contribution from the research for this paper.  This rubric was designed considering the great number of tasks and objectives that may be included in a tabletop exercise. Table 3 is shown in its full capacity below.

Scoring Effectiveness

To utilize the rubric, exercise evaluators would read the tasks listed on the left and employ the descriptions listed beside the tasks to determine a score for the specific task or question listed.  Each task would receive a 1 (lowest quality), 2 (average quality) or a 3 (highest quality) based on how well the tabletop exercise fulfilled the task.  The culmination of the tasks/questions listed in the rubric should fulfill the purpose and goals of the exercise.  Hence, a score toward the higher end of the max scoring should indicate exercise effectiveness.  In the example in Appendix 1, the minimum score would be a 20 with a maximum value of 60.  Considering the range, a median score of at least 30 may indicate an effective exercise, but it would be up to the designer of the rubric to identify the threshold based on the number of tasks/questions listed and their respective values.   One other suggestion to the rubric might be assigning a greater weighted value range for tasks/questions that have great importance.

Who should complete the rubric?

In this assessment, it would make sense to utilize two groups to fill out this form following the exercise.  In the first group, the players should be considered the primary responder to the rubric.  They will be the main focus and it will be their response to the exercise scenario that will be gauged.  The rubric could remain anonymous since the exercise in no-fault and will not be a factor in gauging the results.  Additionally, the tabletop exercise remaining no-fault may foster more honest response from the players.  The second group that would complete the rubric consists of site agents that have expert knowledge and experience with tabletop exercises.  They may be individuals that assist in the setup and reality design of the exercise, but are not a part of the exercise planning/facilitation team and have no stake in how well the exercise performs.  Ideally this would be someone who can observe the exercise, but one who is not a player or providing response during the exercise.  Lastly, a combination of the two may be the best model.  By obtaining a score from both the players and the site, one could compare the averages between the two sets to see any deviation of appeared effectiveness.

Considering the potential alternatives to the current quantitative metrics available, the rubric may provide the most value to gain a quantitative insight from a no-fault exercise.

Resources