Skip to content
Licensed Unlicensed Requires Authentication Published by De Gruyter December 10, 2021

“Sick or not sick?” A mixed methods study evaluating the rapid determination of illness severity in a pediatric emergency department

Laura B. O’Neill, Priti Bhansali ORCID logo, James E. Bost, James M. Chamberlain and Mary C. Ottolini
From the journal Diagnosis



Experienced physicians must rapidly identify ill pediatric patients. We evaluated the ability of an illness rating score (IRS) to predict admission to a pediatric hospital and explored the underlying clinical reasoning of the gestalt assessment of illness.


We used mixed-methods to study pediatric emergency medicine physicians at an academic children’s hospital emergency department (ED). Physicians rated patients’ illness severity with the IRS, anchored by 0 (totally well) and 10 (critically ill), and shared their rationale with concurrent think-aloud responses. The association between IRS and need for hospitalization, respiratory support, parenteral antibiotics, and resuscitative intravenous (IV) fluids were analyzed with mixed effects linear regression. Area under the curve (AUC) receiver operator characteristic (ROC) curve and test characteristics at different cut-points were calculated for IRS as a predictor of admission. Think-aloud responses were qualitatively analyzed via inductive process.


A total of 141 IRS were analyzed (mean 3.56, SD 2.30, range 0–9). Mean IRS were significantly higher for patients requiring admission (4.32 vs. 3.13, p<0.001), respiratory support (6.15 vs. 3.98, p = 0.033), IV fluids (4.53 vs. 3.14, p < 0.001), and parenteral antibiotics (4.68 vs. 3.32, p = 0.009). AUC for IRS as a predictor of admission was 0.635 (95% CI: 0.534–0.737). Analysis of 95 think-aloud responses yielded eight categories that describe the underlying clinical reasoning.


Rapid assessments as captured by the IRS differentiated pediatric patients who required admission and medical interventions. Think-aloud responses for the rationale for rapid assessments may form the basis for teaching the skill of identifying ill pediatric patients.

Corresponding author: Dr. Laura B. O’Neill, MD, MS, Division of Hospital Medicine, Children’s National Hospital, 111 Michigan Ave NW, Suite 4800, Washington, DC 20010, USA; and George Washington University of Medicine and Health Sciences, Washington, DC, USA, Phone: +1 202 476 3664, E-mail:

  1. Research funding: None declared.

  2. Author contributions: All authors have accepted responsibility for the entire content of this manuscript and approved its submission.

  3. Competing interests: Authors state no conflict of interest.

  4. Informed consent: Informed consent was obtained from all individuals included in this study.

  5. Ethical approval: The study survey and protocol involving human subjects (Pro00011729) were submitted to the Institutional Review Board (IRB) at Children’s National Hospital and granted approval on July 24, 2019.

  6. Previous presentations: The quantitative data were presented in written and oral abstract presentation form at the following conferences: Pediatric Academic Society (PAS) Annual Conference, April 2020, Virtual; Pediatric Hospital Medicine (PHM) Annual Conference, July 2020, Virtual; Society for Diagnosis in Medicine (SIDM) Annual Conference, October 2020, Virtual.


1. Spaeder, MC, Moorman, JR, Tran, CA, Keim-Malpass, J, Zschaebitz, JV, Lake, DE, et al.. Predictive analytics in the pediatric intensive care unit for early identification of sepsis: capturing the context of age. Pediatr Res 2019;86:655–61. in Google Scholar

2. Topjian, AA, Berg, RA, Nadkarni, VM. Advances in recognition, resuscitation, and stabilization of the critically ill child. Pediatr Clin North Am 2013;60:605–20. in Google Scholar

3. Wright, JL, Patterson, MD. Resuscitating the pediatric patient. Emerg Med Clin North Am 1996;14:219–31. in Google Scholar

4. American Heart Association. 2005 American Heart Association (AHA) guidelines for cardiopulmonary resuscitation (CPR) and emergency cardiovascular care (ECC) of pediatric and neonatal patients: pediatric basic life support. Pediatrics 2006;117:e989–1004. in Google Scholar PubMed

5. Corrales, AY, Starr, M. Assessment of the unwell child. Aust Fam Physician 2010;39:270–5.Search in Google Scholar

6. Mandell, IM, Bynum, F, Marshall, L, Bart, R, Gold, JI, Rubin, S. Pediatric Early Warning Score and unplanned readmission to the pediatric intensive care unit. J Crit Care 2015;30:1090–5. in Google Scholar PubMed

7. Chamberlain, JM. The Pediatric Risk of Hospital Admission Score: a second-generation severity-of-illness score for pediatric emergency patients. Pediatrics 2005;115:388–95. in Google Scholar PubMed

8. Gold, DL, Mihalov, LK, Cohen, DM. Evaluating the Pediatric Early Warning Score (PEWS) system for admitted patients in the pediatric emergency department. Walthall J, editor. Acad Emerg Med 2014;21:1249–56. in Google Scholar PubMed PubMed Central

9. Sweney, JS, Poss, WB, Grissom, CK, Keenan, HT. Comparison of severity of illness scores to physician clinical judgment for potential use in pediatric critical care triage. Disaster Med Public Health Prep 2012;6:126–30. in Google Scholar PubMed

10. Bonafide, CP, Roberts, KE, Weirich, CM, Paciotti, B, Tibbetts, KM, Keren, R, et al.. Beyond statistical prediction: qualitative evaluation of the mechanisms by which pediatric early warning scores impact patient safety: Early Warning Score Qualitative Study. J Hosp Med 2013;8:248–53. in Google Scholar PubMed

11. Akre, M, Finkelstein, M, Erickson, M, Liu, M, Vanderbilt, L, Billman, G. Sensitivity of the pediatric early warning score to identify patient deterioration. Pediatrics 2010;125:e763–9. in Google Scholar PubMed

12. Fenix, J, Gillespie, CW, Levin, A, Dean, N. Comparison of pediatric early warning score to physician opinion for deteriorating patients. Hosp Pediatr 2015;5:474–9. in Google Scholar

13. Bradman, K, Borland, M, Pascoe, E. Predicting patient disposition in a paediatric emergency department: predicting patient disposition in paediatric emergency department. J Paediatr Child Health 2014;50:E39–44. in Google Scholar

14. Rohacek, M, Nickel, CH, Dietrich, M, Bingisser, R. Clinical intuition ratings are associated with morbidity and hospitalisation. Int J Clin Pract 2015;69:710–7. in Google Scholar

15. Cabrera, D, Thomas, J, Wiswell, J, Walston, J, Anderson, J, Hess, E, et al.. Accuracy of ‘my gut feeling:’ comparing system 1 to system 2 decision-making for acuity prediction, disposition and diagnosis in an academic emergency department. West J Emerg Med 2015;16:653–7. in Google Scholar

16. Wiswell, J, Tsao, K, Bellolio, MF, Hess, EP, Cabrera, D. “Sick” or “not-sick”: accuracy of System 1 diagnostic reasoning for the prediction of disposition and acuity in patients presenting to an academic ED. Am J Emerg Med 2013;31:1448–52. in Google Scholar

17. Stolper, E, Van de Wiel, M, Van Royen, P, Van Bokhoven, M, Van der Weijden, T, Dinant, GJ. Gut feelings as a third track in general practitioners’ diagnostic reasoning. J Gen Intern Med 2011;26:197–203. in Google Scholar

18. Sibbald, M, Sherbino, J, Preyra, I, Coffin-Simpson, T, Norman, G, Monteiro, S. Eyeballing: the use of visual appearance to diagnose ‘sick’. Med Educ 2017;51:1138–45. in Google Scholar

19. Baumann, MR. Evaluation of the emergency severity index (version 3) triage algorithm in pediatric patients. Acad Emerg Med 2005;12:219–24. in Google Scholar

20. Eitel, DR. The emergency severity index triage algorithm version 2 is reliable and valid. Acad Emerg Med 2003;10:1070–80. in Google Scholar

21. Battista, A, Konopasky, A, Ramani, D, Ohmer, M, Mikita, J, Howle, A, et al.. Clinical reasoning in the primary care setting: two scenario-based simulations for residents and attendings. MedEdPORTAL 2018;14:10773. in Google Scholar PubMed PubMed Central

22. Ohmer, M, Durning, SJ, Kucera, W, Nealeigh, M, Ordway, S, Mellor, T, et al.. Clinical reasoning in the ward setting: a rapid response scenario for residents and attendings. MedEdPORTAL 2019;15:10834. in Google Scholar PubMed PubMed Central

23. Harris, PA, Taylor, R, Thielke, R, Payne, J, Gonzalez, N, Conde, JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42:377–81. in Google Scholar PubMed PubMed Central

24. Feudtner, C, Feinstein, JA, Zhong, W, Hall, M, Dai, D. Pediatric complex chronic conditions classification system version 2: updated for ICD-10 and complex medical technology dependence and transplantation [Internet]. BMC Pediatr 2014;14:199. in Google Scholar PubMed PubMed Central

25. Saldaña, J. The coding manual for qualitative researchers, 2nd ed. Los Angeles: SAGE; 2013:303 p.Search in Google Scholar

26. Kahneman, D, Klein, G. Conditions for intuitive expertise: a failure to disagree. Am Psychol 2009;64:515–26. in Google Scholar PubMed

27. Van den Bruel, A, Thompson, M, Buntinx, F, Mant, D. Clinicians’ gut feeling about serious infections in children: observational study. BMJ 2012;345:e6144. in Google Scholar PubMed PubMed Central

28. Walsh, P, Thornton, J, Asato, J, Walker, N, McCoy, G, Baal, J, et al.. Approaches to describing inter-rater reliability of the overall clinical appearance of febrile infants and toddlers in the emergency department. PeerJ 2014;2:e651. in Google Scholar PubMed PubMed Central

29. Search in Google Scholar

30. Jauregui, J, Nelson, D, Choo, E, Stearns, B, Levine, AC, Liebmann, O, et al.. The BUDDY (Bedside Ultrasound to Detect Dehydration in Youth) study. Crit Ultrasound J 2014;6:15. in Google Scholar PubMed PubMed Central

31. Bowen, L, Shaw, A, Lyttle, MD, Purdy, S. The transition to clinical expert: enhanced decision making for children aged less than 5 years attending the paediatric ED with acute respiratory conditions. Emerg Med J 2017;34:76–81. in Google Scholar PubMed PubMed Central

32. Douw, G, Schoonhoven, L, Holwerda, T, Huisman-de Waal, G, van Zanten, ARH, van Achterberg, T, et al.. Nurses’ worry or concern and early recognition of deteriorating patients on general wards in acute care hospitals: a systematic review. Crit Care 2015;19:230. in Google Scholar PubMed PubMed Central

33. Fernández, A, Ares, MI, Garcia, S, Martinez-Indart, L, Mintegi, S, Benito, J. The validity of the Pediatric Assessment Triangle as the first step in the triage process in a pediatric emergency department. Pediatr Emerg Care 2017;33:234–8. in Google Scholar PubMed

34. Horeczko, T, Enriquez, B, McGrath, NE, Gausche-Hill, M, Lewis, RJ. The Pediatric Assessment Triangle: accuracy of its application by nurses in the triage of children. J Emerg Nurs 2013;39:182–9. in Google Scholar PubMed PubMed Central

35. Kestner, V, Vogeley, E, Pitetti, RD, Sun, Q, Parker, KN, Maloney, L. A comparison of parental and nursing assessments of level of illness or injury in a pediatric emergency department. Pediatr Emerg Care 2009;25:633–5. in Google Scholar PubMed

36. Dhaliwal, G. Going with your gut. J Gen Intern Med 2011;26:107–9. in Google Scholar PubMed PubMed Central

37. Vanstone, M, Monteiro, S, Colvin, E, Norman, G, Sherbino, J, Sibbald, M, et al.. Experienced physician descriptions of intuition in clinical reasoning: a typology. Diagnosis (Berl). 2019;6:259–68. in Google Scholar PubMed

38. Berry, JG, Hall, M, Hall, DE, Kuo, DZ, Cohen, E, Agrawal, R, et al.. Inpatient growth and resource use in 28 children’s hospitals: a longitudinal, multi-institutional study. JAMA Pediatr 2013;167:170. in Google Scholar PubMed PubMed Central

39. Murphy, NA, Alvey, J, Valentine, KJ, Mann, K, Wilkes, J, Clark, EB. Children with medical complexity: the 10-year experience of a single center. Hosp Pediatr 2020;10:702–8. in Google Scholar PubMed

40. Russell, CJ, Simon, TD. Care of children with medical complexity in the hospital setting. Pediatr Ann 2014;43:e157–62. in Google Scholar PubMed

41. Dosa, NP, Boeing, NM, Ms, N, Kanter, RK. Excess risk of severe acute illness in children with chronic health conditions. Pediatrics 2001;107:499–504. in Google Scholar PubMed

42. Chan, T, Rodean, J, Richardson, T, Farris, RWD, Bratton, SL, Di Gennaro, JL, et al.. Pediatric critical care resource use by children with medical complexity. J Pediatr 2016;177:197–203.e1. in Google Scholar PubMed

43. Rudolf, F, Oyama, LC, Schwartz, K, Fernandez, JA, Hayden, SR. Teaching rapid assessment skills in triage for the emergency medicine clerkship. J Emerg Med 2021;61:76–81.10.1016/j.jemermed.2021.02.005Search in Google Scholar PubMed

44. Seki, M, Otaki, J, Breugelmans, R, Komoda, T, Nagata-Kobayashi, S, Akaishi, Y, et al.. How do case presentation teaching methods affect learning outcomes?-SNAPPS and the One-Minute preceptor. BMC Med Educ 2016;16:12. in Google Scholar PubMed PubMed Central

45. Natesan, S, Bailitz, J, King, A, Krzyzaniak, S, Kennedy, S, Kim, A, et al.. Clinical teaching: an evidence-based guide to best practices from the council of emergency medicine residency directors. West J Emerg Med 2020;21:985–98. in Google Scholar PubMed PubMed Central

46. Sokol, K. Modifying the one-minute preceptor model for use in the emergency department with a critically ill patient. J Emerg Med 2017;52:368–9. in Google Scholar PubMed

47. Ravichandran, L, Sivaprakasam, E, Balaji, S, Swaminathan, N. Effectiveness of the 1-minute preceptor on feedback to pediatric residents in a busy ambulatory setting. J Grad Med Educ 2019;11:204–6. in Google Scholar PubMed PubMed Central

48. Richards, JB, Hayes, MM, Schwartzstein, RM. Teaching clinical reasoning and critical thinking. Chest 2020;158:1617–28. in Google Scholar PubMed

Received: 2021-07-13
Accepted: 2021-10-29
Published Online: 2021-12-10

© 2021 Walter de Gruyter GmbH, Berlin/Boston