Jump to ContentJump to Main Navigation
Show Summary Details
More options …

Journal of Homeland Security and Emergency Management

Editor-in-Chief: Renda-Tanali, Irmak

Managing Editor: McGee, Sibel

4 Issues per year

IMPACT FACTOR 2017: 0.712

CiteScore 2017: 0.92

SCImago Journal Rank (SJR) 2017: 0.242
Source Normalized Impact per Paper (SNIP) 2017: 0.615

See all formats and pricing
More options …

A Capabilities-Based Framework for Disaster Response Exercise Design and Evaluation: Findings from Oil Spill Response Exercises

Brandon Greenberg
  • George Washington University School of Engineering and Applied Science – Engineering Management and Systems Engineering, Washington, DC 20052, USA
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
/ Paule Voevodsky
  • George Washington University School of Engineering and Applied Science – Engineering Management and Systems Engineering, Washington, DC 20052, USA
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
/ Erica Gralla
  • Corresponding author
  • George Washington University School of Engineering and Applied Science – Engineering Management and Systems Engineering, 800 22nd Street NW, Washington, DC 20052, USA
  • Email
  • Other articles by this author:
  • De Gruyter OnlineGoogle Scholar
Published Online: 2017-06-20 | DOI: https://doi.org/10.1515/jhsem-2016-0034


The responder community must be ready to respond quickly and effectively in the event of a disaster. In order to maintain readiness, many disaster response communities exercise their response capabilities on a regular basis. The critical challenge is to design, conduct, and evaluate exercises in a manner that effectively tests responders’ readiness and generates lessons that can improve readiness. This paper describes a framework to enable assessment of response readiness through evaluation of critical capabilities in exercises. It was developed for oil spill response based on the observation and analysis of four response exercises. The framework (1) identifies critical capabilities that lead to readiness for spill response, and maps them to (2) exercise design components that test each capability and (3) evaluation measures to evaluate each capability within an exercise. The framework enables continuous improvement by linking the evaluation of exercises to the critical capabilities required of an oil spill response organization; by evaluating the performance of specific capabilities, areas for improvement are clearly identified and can be re-tested in a future exercise. While the findings are necessarily specific to oil spill response, the principles apply to any disaster response context.

Keywords: capabilities; evaluation; exercises; learning; preparedness; response

  • Andersson, Annika, Eric D. Carlstrom, Bengt Ahgren, and Johan M. Berlin. 2014. “Managing Boundaries at the Accident Scene – A Qualitative Study of Collaboration Exercises.” International Journal of Emergency Services 3 (1): 77–94.CrossrefGoogle Scholar

  • Arnold, Jacqueline J., LeAnn M. Johnson, Sharon J. Tucker, James F. Malec, Sarah E. Henrickson, and William F. Dunn. 2009. “Evaluation Tools in Simulation Learning: Performance and Self-Efficacy in Emergency Response.” Clinical Simulation in Nursing 5 (1): e35–e43.Google Scholar

  • Babus, Sylvia, Kathryn Hodges, and Erik Kjonnerod. 1997. “Simulations and Institutional Change: Training US Government Professionals for Improved Management of Complex Emergencies Abroad.” Journal of Contingencies and Crisis Management 5 (4): 231–240.CrossrefGoogle Scholar

  • Bakacsi, Gyula 2010. “Managing Crisis: Single-Loop or Double-Loop Learning?” Strategic Management 15 (3): 003–009.Google Scholar

  • Beamon, Benita M., and Burcu Balcik. 2008. “Performance Measurement in Humanitarian Relief Chains.” International Journal of Public Sector Management 21 (1): 4–25.CrossrefGoogle Scholar

  • Berlin, Johan M., and Eric D. Carlström. 2013. “The Dominance of Mechanistic Behaviour: A Critical Study of Emergency Exercises.” International Journal of Emergency Management 9 (4): 327–350.CrossrefGoogle Scholar

  • Berlin, Johan M., and Eric D. Carlström. 2015. “The Three-Level Collaboration Exercise – Impact of Learning and Usefulness.” Journal of Contingencies and Crisis Management 23 (4): 257–265.Web of ScienceCrossrefGoogle Scholar

  • Biddinger, Paul D., Rebecca O. Cadigan, Bruce S. Auerbach, Jonathan L. Burstein, Elena Savoia, Michael A. Stoto, and Howard K. Koh. 2008. “On Linkages: Using Exercises to Identify Systems-Level Preparedness Challenges.” Public Health Reports 123 (1): 96–101.CrossrefWeb of ScienceGoogle Scholar

  • Boin, Arjen, Celesta Kofman-Bos, and Werner Overdijk. 2004. “Crisis Simulations: Exploring Tomorrow’s Vulnerabilities and Threats.” Simulation & Gaming 35 (3): 378–393.CrossrefGoogle Scholar

  • Brady, Thomas F (2003)."Emergency Management: Capability Analysis of Critical Incident Response," Winter Simulation Conference. 1863–1867 http://dl.acm.org/citation.cfm?id=1031075.

  • Cashman, Dennis 2011. “Designing Operations-Based Exercises Based on Artificialities and Realism.” In 2011 International Oil Spill Conference. American Petroleum Institute http://www.ioscproceedings.org/doi/abs/10.7901/2169-3358-2011-1-42.

  • Cashman, Dennis, Jackie Stephens, and LT. Thomas Boyles. 2003. “Designing, Executing, and Evaluating a US Government-Led National Preparedness for Response Exercise Program (PREP) Drill.” In 2003 International Oil Spill Conference., 597–602. American Petroleum Institute http://ioscproceedings.org/doi/abs/10.7901/2169-3358-2003-1-597.

  • Chi, Chih-Hsien, Wen-Hsin Chao, Chia-Chang Chuang, Ming-Che Tsai, and Liang-Miin Tsai. 2001. “Emergency Medical Technicians’ Disaster Training by Tabletop Exercise.” The American Journal of Emergency Medicine 19 (5): 433–36.CrossrefGoogle Scholar

  • Cooper, Simon, Robyn Cant, Joanne Porter, Ken Sellick, George Somers, Leigh Kinsman, and Debra Nestel. 2010. “Rating Medical Emergency Teamwork Performance: Development of the Team Emergency Assessment Measure (TEAM).” Resuscitation 81 (4): 446–52.Web of ScienceCrossrefGoogle Scholar

  • Dausey, David J., James W. Buehler, and Nicole Lurie. 2007. “Designing and Conducting Tabletop Exercises to Assess Public Health Preparedness for Manmade and Naturally Occurring Biological Threats.” BMC Public Health 7: 92.CrossrefWeb of ScienceGoogle Scholar

  • Descatha, Alexis, Thomas Loeb, François Dolveck, Nathalie-Sybille Goddet, Valerie Poirier, and Michel Baer. 2009. “Use of Tabletop Exercise in Industrial Training Disaster.” Journal of Occupational and Environmental Medicine 51 (9): 990–991.Web of ScienceCrossrefGoogle Scholar

  • Donahue, Amy K., and Robert V. Tuohy. 2006. “Lessons We Don’t Learn: A Study of the Lessons of Disasters, Why We Repeat Them, and How We Can Learn Them.” Homeland Security Affairs II (2): 1–28.Google Scholar

  • Draft National Preparedness for Response Exercise Program Guidelines. 2015. USCG, EPA & BSEE http://www.cleangulfassoc.com/uploads/files/PREP Guidelines MASTER_March 2015.pdf.

  • Eisenhardt, Kathleen M 1989. “Building Theories from Case Study Research.” Academy of Management Review 14 (4): 532–550.CrossrefGoogle Scholar

  • Fautua, David, Sae Schatz, Emilie Reitz, and Patricia Bockelman. (2014)."Institutionalizing Blended Learning into Joint Training: A Case Study and Ten Recommendations," Interservice/Industry Training, Simulation, and Education Conference 2014 http://www.iitsec.org/about/PublicationsProceedings/Documents/BP_ED_14208_Paper.pdf.

  • Franks, Nicole, Scott Knutson, Heather A. Parker, and Fred LeJeune. (2011)."Plan-Holder Exercises – How to Recapture a Lost Opportunity for Area Plan Improvement," 2011 International Oil Spill Conference. American Petroleum Institute http://ioscproceedings.org/doi/abs/10.7901/2169-3358-2011-1-155.

  • Gagné, Robert Mills, Walter W. Wager, Katharine C. Golas, and John M. Keller. 2005. Principles of Instructional Design., 5th ed. Wiley Online Library. DOI:.CrossrefGoogle Scholar

  • Gebbie, Kristine M., Joan Valas, Jacqueline Merrill, and Stephen Morse. 2006. “Role of Exercises and Drills in the Evaluation of Public Health in Emergency Response.” Prehospital and Disaster Medicine 21 (3): 173–182.CrossrefGoogle Scholar

  • Gralla, Erica, Brandon Greenberg, Paule Voevodsky, John Harrald, Gregory Shaw, and Samuel Babbitt. (2016).A Capabilities-Based Framework for Designing and Evaluating Oil Spill Response Exercises. Technical ReportU.S. Bureau of Safety and Environmental Enforcement.Google Scholar

  • Harrald, John R 2006. Testimony for the Senate Homeland Security Government Affairs Committee on Hurricane Katrina: Recommendations for Reform. March.Google Scholar

  • Homeland Security Exercise and Evaluation Program. 2013. Policy. U.S. Department of Homeland Security https://www.preptoolkit.org/web/hseep-resources/home.

  • Hovden, Jan, Fred Storseth, and Ranveig Kviseth Tinmannsvik. 2011. “Multilevel Learning from Accidents - Case Studies in Transport.” Safety Science 49 (1): 98–105.Web of ScienceCrossrefGoogle Scholar

  • Jackson, Brian A., and Shawn McKay. 2011. “Preparedness Exercises 2.0: Alternative Approaches to Exercise Design That Could Make Them More Useful for Evaluating and Strengthening Preparedness.” Homeland Security Affairs 7. http://calhoun.nps.edu/handle/10945/24976.

  • Jönsson, Henrik, Marcus Abrahamsson, and Henrik Johansson. 2007. “An Operational Definition of Emergency Response Capabilities.” In Proceedings of Disaster Recovery and Relief: Current & Future Approaches (TIEMS 2007). The International Emergency Management Society http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:An+Operational+Definition+of+Emergency+Response+Capabilities#0.

  • Langley, Ann 1999. “Strategies for Theorizing from Process Data.” Academy of Management Review 24 (4): 691–710.CrossrefGoogle Scholar

  • Mendonça, David 2007. “Decision Support for Improvisation in Response to Extreme Events: Learning from the Response to the 2001 World Trade Center Attack.” Decision Support Systems 43 (3): 952–967.Web of ScienceCrossrefGoogle Scholar

  • Miles, Matthew B., and A. Michael Huberman. 1984. Qualitative Data Analysis: A Sourcebook of New Methods. Sage Publications https://lccn.loc.gov/84002140.Google Scholar

  • National Incident Management System. 2008. U.S. Department of Homeland Security https://www.fema.gov/pdf/emergency/nims/NIMS_core.pdf.

  • National Preparedness for Response Exercise Program Guidelines. 2002.USCG, EPA, & DOI.Google Scholar

  • National Response Framework. 2013. U.S. Department of Homeland Security May. https://www.fema.gov/media-library-data/20130726-1914-25045-1246/final_national_response_framework_20130501.pdf.

  • Ott, Gary L., David C. Haynes, and Robert G. Pond. 1999. “Considering ‘Best Response’ Capabilities in Area Contingency Plans.” In 1999 International Oil Spill Conference., 7–13. American Petroleum Institute http://ioscproceedings.org/doi/abs/10.7901/2169-3358-1999-1-7.

  • Rosen, Michael A., Sallie J. Weaver, Elizabeth H. Lazzara, Eduardo Salas, Teresa Wu, Salvatore Silvestri, Nicola Schiebel, Sandra Almeida, and Heidi B. King. 2010. “Tools for Evaluating Team Performance in Simulation-Based Training.” Journal of Emergencies, Trauma, and Shock 3 (4): 353.CrossrefGoogle Scholar

  • Savoia, Elena, Marcia A. Testa, Paul D. Biddinger, Rebecca O. Cadigan, Howard Koh, Paul Campbell, and Michael A. Stoto. 2009. “Assessing Public Health Capabilities during Emergency Preparedness Tabletop Exercises: Reliability and Validity of a Measurement Tool.” Public Health Reports 124: 138–148.CrossrefWeb of ScienceGoogle Scholar

  • Silenas, Rasa, Ralitsa Akins, Alan R. Parrish, and Janine C. Edwards. 2008. “Developing Disaster Preparedness Competence: An Experiential Learning Exercise for Multiprofessional Education.” Teaching and Learning in Medicine: An International Journal 20 (1): 62–68.CrossrefGoogle Scholar

  • Smith, Eleanor M., Ford J. Kevin, and Steve W. J. Kozlowski. 1997. “Building Adaptive Expertise: Implications for Training Design Strategies.” In Training for a Rapidly Changing Workplace Applications of Psychological Research., edited by Meguel A. Quinones, and Ebrenstrin Addie, 89–118. Washington, DC, US: American Pschological Association.Google Scholar

  • Spradley, James P. 1980. “Participant Observation.” Qualitative Research 3: 1–16.Web of ScienceGoogle Scholar

  • Target Capabilities List. 2007. A Companion to the National Preparedness Guidelines U.S. Department of Homeland Security papers2://publication/uuid/B07AFDA1-93F0-40AC-BE67-5B1700D68031.Google Scholar

  • U.S. Coast Guard Incident Management Handbook. 2001. US Coast Guard.Google Scholar

  • Yin, Robert K. 2014. Case Study Research: Design and Methods., 5th ed. SAGE: Los Angeles.Google Scholar

  • Zhou, Bojia, Gang Sun, Xiaoyong Zhang, Jianhua Xu, Junyan Lai, Xiaoxia Du, Masafumi Hosokawa, Haruo Hayashi, and Reo Kimura. 2015. “Development of Web-Based Tabletop Emergency Earthquake Exercise System.” Journal of Disaster Research 10 (2): 217–224.CrossrefGoogle Scholar

About the article

Published Online: 2017-06-20

Citation Information: Journal of Homeland Security and Emergency Management, Volume 13, Issue 4, 20160034, ISSN (Online) 1547-7355, ISSN (Print) 2194-6361, DOI: https://doi.org/10.1515/jhsem-2016-0034.

Export Citation

©2016 Walter de Gruyter GmbH, Berlin/Boston.Get Permission

Comments (0)

Please log in or register to comment.
Log in