Creating a New Standard for Evaluating Tabletop Exercises

Creating a New Standard for Evaluating Tabletop Exercises

By John Duda and Scott J. Glick

Even though tabletop exercises (TTXs) have been used for decades, an industry standard has not emerged on how to evaluate their effectiveness. Since 2012, John Duda, CEO of SummitET, has noted the lack of an industry standard for quantitative assessments of TTXs, which prompted him to develop a rubric for analyzing and measuring exercise effectiveness. Based on their extensive exercise experience in both the government and the private sector, Mr. Duda and Scott J. Glick, SummitET’s General Counsel, have refined the rubric and its scoring of various exercise factors into what they call the XF ScoreTM, which they discuss in their article published by the Domestic Preparedness Journal entitled Creating A New Standard for Evaluating Tabletop Exercises.

Access the full article published by DomesticPreparedness.com.

Tabletop exercise design checklist

Go to our checklist for organizations to use to ensure that the critical design elements for a successful TTX are considered.

John Duda, Chief Executive Officer and founding partner at SummitET

John Duda

CEO

Scott Glick VP and General Counsel

Scott Glick

General Counsel

Resources

Checklist for Designing a Successful Tabletop Exercise

Checklist for Designing a Successful Tabletop Exercise

John Duda and Scott J. Glick

The manner in which a tabletop exercise (TTX) is designed will have a profound impact on the success of the TTX. For example, if an organization is seeking to exercise a policy, plan, or procedure during the TTX, it is essential that the policy, plan, or procedure have sufficient details and is ready to be tested. If the organization’s policies, plans, and procedures have not reached that level of development, then the organization should consider using a seminar or workshop to educate participants and fill in the necessary details before conducting the TTX. A TTX should also be designed to ensure that specific “big picture” goals and “SMART” – specific, measurable, achievable, relevant, and time-bound –objectives can be achieved.

Most important, a TTX should be designed and facilitated in ways that consider how adults learn best. Dr. Malcolm Knowles, who is known for his groundbreaking research concerning adult learners, and other scholars who have studied how to effectively engage adults, have emphasized that adults learn differently and are motivated differently. As a result, to maximize the learning that takes place during a TTX: (1) TTX participants should be represented during the planning and involved with how their training is delivered; (2) TTXs should be designed to take into account the impact of experiential learning on adults and draw upon their prior experiences and knowledge; (3) TTX participants should be asked to solve problems from the information that is presented; and (4) TTXs should be designed to have immediate relevance and impact to participants’ jobs.  When TTXs are properly designed and executed, their use meets the principles which underlie adult learning.

Based on the foregoing, we recommend that organizations use the following checklist to ensure that the critical design elements for a successful TTX are considered.

  • The exercise design team is small, manageable, experienced, and includes trusted agents from relevant participating organizations.
  • The exercise design team has the necessary resources and support from senior leadership.
  • The policy, plan, or procedure to be evaluated has sufficient details to be tested or exercised.
  • Specific “big picture” goals have been identified (e.g., whether the policies, plans, and procedures have any gaps, and establishing relationships among responding organizations).
  • “SMART” – specific, measurable, achievable, relevant, and time-bound –objectives have been identified.
  • The exercise scenario is tied to the goals and objectives, as well as identified learning outcomes.
  • The exercise scenario is realistic and includes appropriate script and data injects.
  • The exercise scenario is designed to maximize the involvement of all participants.
  • The exercise has been designed and includes facilitation techniques that draw upon the way that adults learn best (e.g., the appropriate use of multi-media).
  • Sufficient time will be allocated for the exercise given the nature of the topics to be discussed, the number of participants, and how the facilitator intends to engage the participants.
  • Exercise participants will be from all the appropriate agencies or organizations based on the exercise scenario and its goals, objectives, and learning outcomes.
  • Exercise participants will be provided with short summaries of key exercise-related information (“one-pagers”) that they can use during the exercise.
  • The exercise has an experienced and skilled facilitator with extensive knowledge about the organization, its mission, the subject matter, and the exercise objectives.
  • Exercise planners have a detailed logistical checklist for the delivery of the exercise (e.g., space and seating requirements, audio and video equipment, microphones to enable all participants to be heard).
  • Technology has been tested in the location of the exercise and is functioning as intended.
  • There are enough skilled and knowledgeable exercise evaluators who can identify key takeaways for use during any immediate “hot wash” and who can record useful data for an after-action report.
John Duda, Chief Executive Officer and founding partner at SummitET

John Duda

CEO

Scott Glick VP and General Counsel

Scott Glick

General Counsel

Creating a New Standard for Evaluating Tabletop Exercises

By John Duda and Scott J. Glick

Even though tabletop exercises (TTXs) have been used for decades, an industry standard has not emerged on how to evaluate their effectiveness. Since 2012, John Duda, CEO of SummitET, has noted the lack of an industry standard for quantitative assessments of TTXs, which prompted him to develop a rubric for analyzing and measuring exercise effectiveness. Based on their extensive exercise experience in both the government and the private sector, Mr. Duda and Scott J. Glick, SummitET’s General Counsel, have refined the rubric and its scoring of various exercise factors into what they call the XF ScoreTM, which they discuss in their article published by the Domestic Preparedness Journal entitled Creating A New Standard for Evaluating Tabletop Exercises.

Access the full article published by DomesticPreparedness.com.

Resources

Radiation Communication Resources

Radiation Communication Resources

Find below an expertly curated list of government radiation emergency information, tools, and resources.

Communicating Radiation Emergencies

Radiation Emergency Resources

In the event of a radiological emergency, medical professionals can find references and job aids to assist in the management of individuals injured by ionizing radiation.

Oak Ridge Institute for Science and Education (ORISE) / U.S. Department of Energy (DOE)

orise.orau.gov

“Do I Need to Take Potassium Iodide (KI)?”

This document answers frequently asked questions related to taking potassium iodide (KI) during a radiological release, as well as dosing recommendations from the United States Food and Drug Administration (FDA).

Oak Ridge Institute for Science and Education (ORISE) / U.S. Department of Energy (DOE)

orise.orau.gov

How to Stay Safe in the Event of a Radiation Emergency

Federal Emergency Management Agency (FEMA)

Ready.gov/radiation

Radiation Monitoring

EPA’s RadNet system monitors the nation’s air, precipitation, and drinking water for radiation.

United States Environmental Protection Agency (EPA)

epa.gov/radnet

Radiation Protection and Health Effects

United States Environmental Protection Agency (EPA)

epa.gov/radiation

Radiation Emergencies

Tools and resources for clinicians, public health professionals, and the public

Centers for Disease Control and Prevention (CDC)

cdc.gov

Radiation Thermometer

The purpose of the radiation thermometer is to put common radiation doses in perspective. This tool can help people assess their own risk in a radiation emergency.

Centers for Disease Control and Prevention (CDC)

cdc.gov

Radiation Hazard Scale

The Centers for Disease Control and Prevention has developed the Radiation Hazard Scale as a tool for communication in emergencies.

Centers for Disease Control and Prevention (CDC)

cdc.gov

Find a U.S. Embassy

Websites of U.S. embassies, consulates, diplomatic missions, and offices providing consular services

U.S. Department of State

usembassy.gov

Nuclear and Radiological Incidents

Learn the steps to follow to keep you and your family safe before, during, and after a nuclear incident.

U.S. Department of State

travel.state.gov

Radiological and Nuclear Emergency Preparedness Information from FDA – Food Safety

Food and Drug Administration (FDA)

fda.gov

Food Emergency Response Network (FERN)

FERN integrates the nation’s food-testing laboratories at the local, state, and federal levels into a network that is able to respond to emergencies involving biological, chemical, or radiological contamination of food.

U.S. Department of Agriculture (USDA)

fernlab.org

Nuclear Materials Safe Use

The U.S. Nuclear Regulatory Commission is an independent regulator created to ensure the safe use of radioactive materials in the United States, including commercial nuclear power plants and the use of nuclear materials in industry, academia, and medicine.

U.S. Nuclear Regulatory Commission (NRC)

nrc.gov

Radiation Communications Resources

Resources

Initial Dose Magnitude Estimation for Individuals Involved in a Radiological Incident/Accident 

Initial Dose Magnitude Estimation for Individuals Involved in a Radiological Incident/Accident 

Steve Sugarman originally wrote Early Internal and External Dose Magnitude Estimation in 2008 when serving as the Health Physics Project Manager at the Radiation Emergency Assistance Center/Training Site (REAC/TS), a response asset of the US Department of Energy, to provide general guidance for early estimation of radiation dose magnitude. Although not always possible, every effort was made to write in understandable terms so that the guidance could be used by a wide range of people. The document was posted on the REAC/TS website, and during my time at REAC/TS the document was periodically updated in an effort to keep it current and improve its usability – the most recent update to Early Internal and External Dose Magnitude Estimation being 2017.

In late 2017 Steve left REAC/TS to join SummitET where he is the Vice President of Operations and Corporate Health Physicist. Upon revisiting Early Internal and External Dose Magnitude Estimation, Steve realized new information that may be pertinent to the topic had become available. He then incorporated that information into an undated document: Initial Dose Magnitude Estimation for Individuals Involved in a Radiological Incident/Accident.

Initial Dose Magnitude Estimation for Individuals Involved in a Radiological Incident/Accident isn’t intended to provide methods for definitive dose calculation, but to provide methods one may consider using for initial dose estimation when trying to determine the potential magnitude of the radiation doses to individuals involved in a radiation incident/accident. As with any job, it’s advantageous to have multiple tools available to help with the task, but it’s up to the user to determine if the proper tool is being selected and to apply that tool correctly to any given situation. The tools/methods described in this document are in no way intended to take the place of established/validated internal dose assessment (urinalysis, whole body counting, etc.) or external dose assessment (selection of proper dosimetry, in-depth reconstructions, etc.) techniques, nor are they to be used for regulatory and/or occupational dose assignment. Each situation should be evaluated for the applicability of the described tools with an understanding of the strengths and weaknesses that are inherent in each of them.

This document is intended to provide general guidance and is not a peer-reviewed publication. 

Early Dose Magnitude Estimation

Download this free resource document created by Steve Sugarman
Steve Sugarman

Steve Sugarman

VP of Operations & Corporate Health Physicist

Note: The following document builds upon and updates information contained in a document written by Steve Sugarman when he was the Health Physics Project Manager at the Radiation Emergency Assistance Center/Training Site (REACT/TS) entitled “Early Internal and External Dose Magnitude Estimation.” The technical information contained in this update can be used to guide emergency responders, medical personnel, and others in occupational settings to conduct early radiation dose estimations. 

Resources

Meeting Strategic Communications Virtual Solutions Requirements during COVID-19 Response For Nuclear Power Plants and Offsite Response Organizations

Meeting Strategic Communications Virtual Solutions Requirements during COVID-19 Response For Nuclear Power Plants and Offsite Response Organizations

Introduction

 A global pandemic, civil unrest, and a volatile political climate have exposed the need for more advanced and strategic communication in uncertain times. Summit Exercises and Training LLC (SummitET®) recognizes the uniqueness of strategic communications, particularly when communicating crisis and emergency risk communication (CERC). Radiation is a topic that is widely misunderstood and requires specialized CERC skills to address pre-crisis, crisis, and post-crisis messaging in order to meet stakeholder informational needs.

It is important for nuclear power plants and Offsite Response Organizations (OROs) to develop and practice effective strategic communication to meet training and exercise requirements such as:

  • NUREG-0654 FEMA-REP-1, Criteria for Preparation and Evaluation of Radiological Emergency Response Plans and Preparedness in Support of Nuclear Power Plants;
  • Federal Emergency Management Agency (FEMA) Radiological Emergency Preparedness Exercise Criteria identified in the Federal Registry Radiological Emergency Preparedness: Exercise Evaluation Methodology (66 FR 47526-47546); and
  • State, local, and tribal requirements.

Response to COVID-19 has presented challenges in meeting training and exercise requirements since many in-person activities are not an option, however, SummitET has successfully made the pivot from in-person to virtual engagements to meet its customer’s mission requirements. SummitET’s Strategic Communications Program offers exercise solutions and training presented through The Strategic Communications Institute for Preparedness® (SCIP), delivered by experts, including a Certified Health Physicist, with extensive knowledge of the unique challenges presented by a radiation emergency. This white paper describes SummitET’s virtual and hybrid training exercise solutions for nuclear power plants and OROs. These training and exercise solutions enhance strategic communications and address CERC principles such as effectiveness, consistency, and proficiency for a FEMA-evaluated exercise and when communicating technical information to non-technical audiences.

Challenge

To meet regulatory requirements as outlined in 10 CFR 50.47(b)(7); 44 CFR 350.5(a)(7), training and exercise activities are conducted to ensure public education and information meet planning standards and evaluation criteria. Due to the inability of many nuclear power plants and OROs to hold in-person training and exercises, this requirement calls for an alternative approach. According to NUREG-0654/FEMA-REP-1, Rev. 2, “Alternative approaches provide an opportunity for state, local, and tribal governments, applicants, and licensees to meet the planning standards in a manner that is different from what the evaluation criteria recommend…” SummitET offers alternative approaches to meet training and exercise requirements using virtual and hybrid learning and exercise environments.

Strategic communications and CERC are an essential part of managing radiation emergencies. CERC fosters a common understanding of issues, protects public health and safety, and preserves organizational efficacy. Evidence-based CERC research and technical expertise make a critical difference in mitigating loss, maintaining trust and credibility, and strengthening relationships with stakeholders. All emergencies have a crisis and risk management component, yet there are few existing courses that teach both crisis AND risk communication. Scholars define risk communication as a science-based approach for communicating effectively in situations of high stress, high concern, or controversy. The Centers for Disease Control defines crisis communication as the process of providing facts to the public about an emergency, beyond an organization’s control, that involves the organization and requires an immediate response. The crisis may cause harm to an organization’s reputation or viability.  In real-world emergencies and training exercises, effective communication continues to be identified as a gap. Particularly, communicating scientific information of science-related information can be an even greater challenge. Organizations should integrate CERC into incident response. CERC integration improves outcomes by effectively engaging stakeholders and promoting a coordinated communication response.

In a high-consequence scenario, both internal and external communications should be addressed. The ability to work with stakeholders such as transient populations, those with access and functional needs, regulatory agencies, and employees is critical for an effective communication response. Failure to adequately communicate risk serves to exacerbate the potential for miscommunication. Consequently, it is critical for nuclear power plants and OROs to collaborate with subject-matter experts to develop strategic communication and integrate CERC skills and processes for planning and performance in a crisis scenario. Particularly during COVID-19 response, the ability to deliver in-person training and conduct exercises has been drastically hampered, although training and exercise must continue to be conducted.

 

Solution

SummitET experts are recognized as leading professionals in the development of strategic communications programs and hosts the SCIP, which offers in-person, virtual and hybrid training programs that extend beyond typical baseline training offered for those responsible for communicating prevention, protection, mitigation, response, and recovery information for radiation emergencies. Our subject-matter experts are highly experienced, trained, and internationally known, current, and former communications professionals. Our team is comprised of a Certified Health Physicist with over 20 years’ experience responding to domestic and international radiation emergencies. Other subject-matter experts are experienced in radiation emergency response, strategic communications, state/local government and private sector emergency management, preparedness planning, federal law enforcement, and hazardous materials. They are skilled in helping organizations assess, plan, and build capacity to effectively manage and message internal and external communications for all-hazards and threats. This includes developing strategies for CERC communications, reputation management, digital and social media, internal communications, community education, and media relations practice. SummitET’s programs include a wide range of trainings, drills, tabletops (TTXs) and practical exercises designed to prepare communicators and senior leaders through informed, innovative, and dynamic instruction offered nationally and internationally.

SummitET Strategic Communications Institute for Preparedness
SummitET currently leverages virtual training in an online, interactive, and participatory virtual classroom for the Department of Energy/National Nuclear Security Administration (DOE/NNSA). We assist sites across the DOE/NNSA enterprise to effectively integrate CERC into their communication strategies. This training delivery format has successfully positioned the DOE/NNSA enterprise to follow CDC’s COVID-19 guidelines by protecting its workforce, customers, and meet mission requirements. Using virtual training environments, SummitET can transition and customize nuclear power plants and OROs’ training requirements, enhance strategic communications, and employ decision-makers and communicators with CERC proficiency.

SummitET’s SCIP also offers a hybrid training model where a segment of participants attends in-person classes, at a facility with appropriate social distancing and safety measures, while other participants join the workshop virtually. This model combines traditional classroom experiences, experiential learning objectives, and digital course delivery using the best option(s) for each learning objective.

SummitET’s virtual and hybrid trainings and exercises provide a learning environment to promote strategic communication and CERC skill development among nuclear power plants and OROs to:

  • Improve internal CERC skills and standardize response procedures across the spectrum of nuclear power plants and OROs to address stakeholder communities, transient populations, and those with access and functional needs.
  • Develop innovative processes grounded in evidence-based research and industry best practices.
  • Deliver strategic communications and CERC training and workshops using virtual and hybrid learning environments to meet requirements.

About Us

SummitET is a veteran-owned small business specializing in preparedness solutions for organizations, businesses, all levels of government agencies, as well as an extensive list of international clients. Our personnel come from the military, emergency management, CERC, fire rescue, law enforcement, federal service, and health physics communities. Our customized solutions and services preserve lives and property through planning, preparation, prevention, mitigation, and response to technical, manmade, and natural disasters. SummitET provides training, exercises, and technical consulting in the following core areas:

  • Radiation Emergency Crisis and Emergency Risk Communication
  • Workplace Violence and Active Shooter
  • CBRNE, WMD, or Terrorism
  • Critical Infrastructure, Red Cell, and Emerging Threats
  • Cybersecurity
  • Business Continuity

SCIP Radiation Safety Communication Workshops

See our full catalog of Radiation Safety Communication workshops from the Strategic Communications Institute for Preparedness.

Call to Action

Given SummitET’s expertise in radiation emergency strategic communications, CERC, and health physics and the SCIP, which provides in-person, virtual and hybrid learning environments for advanced training programs that extend beyond typical baseline training for communicators, we offer the opportunity to guide nuclear power plants and OROs through a variety of scenarios unique to the community. The SummitET team has cultivated relationships with the private sector, federal, state, local, and tribal governments and is positioned to design, deliver, and conduct training and exercises with an interagency, intragovernmental, and cross-sector approach. Our technical subject-matter expertise and real-world experience in radiation emergency strategic communication and CERC effectively prepare individual participants and their organizations to meet training and exercise requirements.

Our approach engages nuclear power plants and OROs in a process using multiple, active learning techniques, blended learning theories, and leveraging instruction and facilitation by experienced staff. SummitET caters to the unique needs of each audience, providing the appropriate introductory, theoretical, and practical knowledge and performance-based skill development for capacity building. We provide participants with customized materials to guide them through their learning experience in a virtual or hybrid learning environment. Strategic communication offerings can be coordinated with virtual TTXs (discussion-based) exercises to help integrate this important capability into the overall response.

Contacts

Additional information regarding SummitET’s expertise can be obtained by contacting individuals listed here through the form below.

Holly Hardin Strategic Communications

Holly Hardin

Strategic Communications Program Manager

Steve Sugarman

Steve Sugarman

VP of Operations & Corporate Health Physicist

Download White Paper PDF

Get in touch with our experts

SummitET News

SummitET Celebrates 10 Years of Success and Innovation

SummitET Celebrates 10 Years of Success and Innovation

St. Petersburg, FL – Summit Exercises and Training LLC (SummitET®), a Preparedness Solutions Company®, is proud to announce the celebration of its 10th anniversary this month. Since 2013, SummitET has been a driving force in the preparedness services industry,...

read more

Coronavirus Information Sources

Coronavirus Information Sources

Getting information from trusted sources regarding the Coronavirus (COVID-19) is essential for preparation. Find the most up-to-date information from the resource links below.

Travel

Coronavirus Disease 2019 Information for Travel:
https://www.cdc.gov/coronavirus/2019-ncov/travelers/index.html

Personal Protection

Personal Protective Equipment (PPE) Shortages:
https://astho.org/COVID-19/PPE-Shortages/

How to Make Your Own Face masks:
https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/diy-cloth-face-coverings.html

PPE Decontamination and Reuse:
https://www.astho.org/COVID-19/Optimizing-PPE-Supplies/

Information for Businesses

Free Strategic Communication Consultation:
https://summitet.com/strategic-communications-covid19

Free COVID Stock Images and Videos:
https://www.shutterstock.com/explore/covid19-resources

Interim Guidance for Business:
https://www.cdc.gov/coronavirus/2019-ncov/specific-groups/guidance-business-response.html

Risk Communication and Community Engagement
https://www.who.int/emergencies/diseases/novel-coronavirus-2019/technical-guidance/risk-communication-and-community-engagement

Canned Advertising Messages
http://coronavirus.adcouncilkit.org/

Health Department Communication Resources
https://astho.informz.net/ASTHO/data/images/CRC_brochure_1.19.pdf

Risk Communications Guide: Questions and Key Messages
https://www.astho.org/COVID-19/Q-and-A/

Cybersecurity

Cybersecurity Best Practice for Remote Workers:
https://staysafeonline.org/press-release/ncsa-encourages-coronavirus-vigilance/

Higher Education

Crisis Leadership and the Higher Education Response
https://evolllution.com/managing-institution/operations_efficiency/crisis-leadership-and-the-higher-education-response-to-the-covid-19-outbreak/

Resources