Abstract
Objectives
Electronic alert systems to identify potential sepsis in children presenting to the emergency department (ED) often either alert too frequently or fail to detect earlier stages of decompensation where timely treatment might prevent serious outcomes.
Methods
We created a predictive tool that continuously monitors our hospital’s electronic health record during ED visits. The tool incorporates new standards for normal/abnormal vital signs based on data from ∼1.2 million children at 169 hospitals. Eighty-two gold standard (GS) sepsis cases arising within 48 h were identified through retrospective chart review of cases sampled from 35,586 ED visits during 2012 and 2014–2015. An additional 1,027 cases with high severity of illness (SOI) based on 3 M’s All Patient Refined – Diagnosis-Related Groups (APR-DRG) were identified from these and 26,026 additional visits during 2017. An iterative process assigned weights to main factors and interactions significantly associated with GS cases, creating an overall “score” that maximized the sensitivity for GS cases and positive predictive value for high SOI outcomes.
Results
Tool implementation began August 2017; subsequent improvements resulted in 77% sensitivity for identifying GS sepsis within 48 h, 22.5% positive predictive value for major/extreme SOI outcomes, and 2% overall firing rate of ED patients. The incidence of high-severity outcomes increased rapidly with tool score. Admitted alert positive patients were hospitalized nearly twice as long as alert negative patients.
Conclusions
Our ED-based electronic tool combines high sensitivity in predicting GS sepsis, high predictive value for physiologic decompensation, and a low firing rate. The tool can help optimize critical treatments for these high-risk children.
Introduction
Disseminated infections in children, including sepsis, may lead to organ dysfunction, decompensation, and death [1]. Unfortunately, recognizing children at risk for deterioration may be difficult at emergency department (ED) presentation. Delayed recognition is associated with more severe outcomes, including permanent disability and death [2], [3], [4], [5]. A number of alert tools utilize the electronic health record (EHR) to aid in identifying severe sepsis or septic shock at a child’s initial ED presentation [6], [7], [8], [9]. These electronic alerts often incorporate vital sign thresholds that are not evidence-based, leading to low specificity and positive predictive value [10], [11]. Furthermore, some alerts use indicators that tend to occur later during the illness, such as hypotension, altered mental status, need for supplemental oxygen, and lactic acidosis. Given their typical occurrence later in the progression towards physiologic decompensation, these indicators may be less useful for inclusion in an ED-based early recognition alert tool [12].
Our objective was to evaluate the performance of our empirically developed ED-based EHR tool to predict infection-related decompensation in children within 48 h of initial presentation. The tool was developed using two target standards: (1) sepsis cases having organ dysfunction that developed within 48 h of ED arrival; and (2) early sepsis cases (defined as disseminated infection with systemic inflammatory response syndrome [SIRS]) that developed within 24 h of ED arrival and resulted in “major” or “extreme” severity of illness (SOI, a diagnosis-based index of maximum physiologic decompensation during the hospital stay) as assigned by the 3M™ Corporation’s All Patient Refined – Diagnosis-Related Groups (APR-DRG®, v32) [13] algorithm.
Materials and methods
Selection of metrics for the electronic assessment tool
Development of the Children at High Risk Alert Tool (CAHR-AT) began in 2013 at the Children’s Hospital of The King’s Daughters (CHKD), a 206-bed freestanding children’s hospital in Norfolk, VA with ∼50,000 ED visits per year. This project was approved by the Institutional Review Board at Eastern Virginia Medical School under a waiver of informed consent as a review of existing medical records conducted as a quality improvement initiative.
The tool builds on our earlier hospital-wide pediatric severe sepsis screening tool described elsewhere [12]. Input data for the CAHR-AT is the hospital’s Cerner Millennium™ EHR, which includes historical information documented during previous health system visits and real-time data obtained from triage and throughout the ED stay. As referenced in Table 1, elements selected for analysis of their association with infection-related decompensation occurring within 48 h of a patient’s ED arrival were based on their availability in our Cerner ED triage system, their known association with sepsis or use in other sepsis alert systems [12], [14], [15], or for hypothesis testing of the shock index and diastolic shock index based on their potential association with shock detection [16], [17].
Data elements collected from the electronic health record and tested for association with gold standard cases of infection-related decompensation.
| Abnormal heart rate (HR): >95th percentile [22] (age-specific values shown in Table 2) or <5th percentile [14] (neonates and infants only) |
| Elevated respiratory rate (RR): >95th percentile [22] (age-specific values shown in Table 2) |
| Abnormal temperature (TMP): <36.0 °C or >38.5 °C [14] (with correction if axillary) |
| Hypotension: Systolic blood pressure (SBP) <5th percentile [12] (age-specific) |
| Elevated shock index (SI, defined as HR ÷ SBP): >95th percentile (age-specific values shown in Table 2) |
| Elevated diastolic shock index (defined as HR ÷ Diastolic blood pressure): >95th percentile (age-specific) |
| Low peripheral blood oxygen saturation (SpO2): ≤90% [14], determined by pulse oximetry |
| Abnormal white blood count (WBC) [14] within past 48 h (age-specific) |
| Emergency severity index [30] (ESI, an acuity assessment assigned to all ED patients during initial triage)=“1” |
| Neonate (age≤28 days) |
| Skin description contains “Mottled,” “Cyanotic,” or “Diaphoretic” from triage assessment |
| Breathing description contains “Labored” or “Grunting” from triage assessment |
| Level of consciousness (LOC) not “Alert” by AVPU scale, from triage assessment |
| Prolonged capillary refill time: “2–4 s” or “>4 s” from triage assessment |
| Pulse strength “weak” from triage assessment |
| Hypoxia, from chief complaint or recent coded diagnosis |
| Acute organ system dysfunction (cardiac, CNS, hepatic, renal, or respiratory), from chief complaint or recent coded diagnosis |
| Infection (known, suspected, or at increased risk), based on following list of individual metrics: |
|
|
|
|
|
| Immunocompromise (known, suspected, or at increased risk), based on the list of individual metrics below: |
|
|
Identification of “gold standard” cases
“Gold standard” (GS) sepsis cases were identified using retrospective chart reviews of possible sepsis cases presenting to CHKD’s ED during one of two time periods: November–December 2012 and August 2014–January 2015. Possible sepsis cases were independently identified without using the CAHR-AT to avoid an “incorporation bias” that might falsely raise the tool’s apparent sensitivity and specificity [18].
Reviewed cases consisted of admitted ED patients meeting at least one of the following criteria: (1) discharge diagnosis of sepsis, severe sepsis, or septic shock; disseminated bacterial, fungal, or viral infection; or a local infection having the potential of progression to sepsis, with specific ICD-9 diagnosis codes as detailed elsewhere [12]; or, (2) a positive culture from blood or fluid from a sterile cavity (e.g., peritoneal or cerebrospinal fluid (CSF)); or (3) a case meeting the definition of sepsis, severe sepsis, or septic shock by either the criteria of Goldstein et al. [14], or requiring coded ICD diagnoses of infection and organ dysfunction as used by Angus et al. [19] or Balamuth et al. [20], [21]. These criteria identified 109 and 807 cases from the two periods, respectively, representing 2.6% of ED patients during those periods.
Ten reviewers (four physicians, five nurses, and one pharmacist) were trained on the criteria to define sepsis; each received a patient summary comprising ∼50 parameters extracted from the EHR, including maximum and minimum vital sign values, laboratory results, medications, and coded diagnoses. Reviewers were asked to review any other components of the patient’s EHR to arrive at a diagnosis of “sepsis” (verified or suspected disseminated infection, accompanied by SIRS), “severe sepsis” (sepsis with organ dysfunction), or “patient not septic.” Reviewers were instructed to use age-specific vital sign ranges (as determined by our previous work [12], [22]) and currently unpublished age-specific ranges for shock index, shown in Table 2, to identify SIRS thresholds. Unlike age-specific vital sign thresholds from the International Pediatric Sepsis Consensus (IPSC) conference [14], which were not evidence-based, our vital sign thresholds were derived from a multi-centered database of ∼1.2 million ED visits [22]. Reviewers were asked to weigh trends in metrics more heavily than isolated values, to consider alternative explanations for abnormal clinical parameters, and to consider sepsis in culture-negative cases.
CAHR-AT age-specific vital sign thresholdsa.
| Age category | HR lower limit (LL)b of normal [14] | HR upper limit (UL)c of normal [22] | RR UL [22] | SI UL (SI = HR ÷ SBP) |
|---|---|---|---|---|
| <1 mo (neonate) | 100 | 184 | 59 | ≥2.43 |
| 1 to <2 mo | 90 | 187 | 58 | ≥2.22 |
| 2 to <3 mo | 90 | 184 | 58 | ≥2.10 |
| 3 to <4 mo | 90 | 182 | 57 | ≥2.02 |
| 4 to <5 mo | 90 | 181 | 56 | ≥1.96 |
| 5 to <6 mo | 90 | 182 | 54 | ≥1.92 |
| 6 to <7 mo | 90 | 182 | 53 | ≥1.89 |
| 7 to <8 mo | 90 | 182 | 51 | ≥1.86 |
| 8 to <9 mo | 90 | 182 | 50 | ≥1.84 |
| 9 to <10 mo | 90 | 182 | 49 | ≥1.82 |
| 10 to <11 mo | 90 | 182 | 48 | ≥1.80 |
| 11 to <12 mo | 90 | 183 | 47 | ≥1.78 |
| 12 to <13 mo | 90 | 183 | 46 | ≥1.77 |
| 13 to <14 mo | 90 | 184 | 46 | ≥1.76 |
| 14 to <15 mo | 90 | 184 | 45 | ≥1.75 |
| 15 to <16 mo | 90 | 184 | 44 | ≥1.74 |
| 16 to <17 mo | 90 | 184 | 44 | ≥1.73 |
| 17 to <18 mo | 90 | 183 | 43 | ≥1.72 |
| 18 to <19 mo | 90 | 182 | 43 | ≥1.71 |
| 19 to <20 mo | 90 | 181 | 42 | ≥1.70 |
| 20 to <21 mo | 90 | 180 | 41 | ≥1.69 |
| 21 to <22 mo | 90 | 179 | 41 | ≥1.68 |
| 22 to <23 mo | 90 | 178 | 40 | ≥1.68 |
| 23 to <2 yrs | 90 | 176 | 40 | ≥1.67 |
| 2 to <3 yrs | N/A | 169 | 37 | ≥1.62 |
| 3 to <4 yrs | N/A | 158 | 34 | ≥1.53 |
| 4 to <5 yrs | N/A | 152 | 31 | ≥1.45 |
| 5 to <6 yrs | N/A | 146 | 30 | ≥1.38 |
| 6 to <7 yrs | N/A | 141 | 28 | ≥1.32 |
| 7 to <8 yrs | N/A | 137 | 27 | ≥1.26 |
| 8 to <9 yrs | N/A | 133 | 27 | ≥1.20 |
| 9 to <10 yrs | N/A | 130 | 26 | ≥1.16 |
| 10 to <11 yrs | N/A | 127 | 25 | ≥1.12 |
| 11 to <12 yrs | N/A | 125 | 25 | ≥1.09 |
| 12 to <13 yrs | N/A | 122 | 24 | ≥1.05 |
| 13 to <14 yrs | N/A | 119 | 24 | ≥1.02 |
| 14 to <15 yrs | N/A | 117 | 23 | ≥1.00 |
| 15 to <16 yrs | N/A | 116 | 23 | ≥0.98 |
| 16 to <17 yrs | N/A | 116 | 23 | ≥0.98 |
| 17 to <18 yrs | N/A | 117 | 23 | ≥0.98 |
| ≥18 yrs | N/A | 117 | 23 | ≥0.98 |
-
aHR, heart rate; RR, respiratory rate; SI, shock index; SBP, systolic blood pressure. Lower and upper limit thresholds represent the 5th and 95th percentiles, respectively, among ED patients for each age range.
-
bAbnormally low HR’s are ignored for patients receiving clonidine or beta blockers recorded as recent meds or on Medication List.
-
cAbnormally high HR’s are ignored for patients receiving asthma meds such as albuterol or stimulant meds such as Ritalin™ recorded as recent meds or on Medication List.
GS cases were defined as: (1) sepsis with organ dysfunction (“severe sepsis” as defined above) that developed within 48 h of ED arrival; this time period was chosen based on its use to distinguish between community-acquired vs. hospital-acquired infection [23]; or (2) children meeting SIRS criteria while in the ED, as redefined from our earlier work (Table 3) [12], who developed sepsis (without organ dysfunction) within 24 h of ED arrival and who had “major” or “extreme” SOI using the APR-DRG v32 criteria.
SIRS criteria: original IPSCa and revised versions used to identify gold standard sepsis cases.
| Systemic inflammatory response syndrome (SIRS) components from IPSCa |
|
|
| IPSCa Criteria for SIRS: Requires a minimum of two abnormal measures, at least one of which must be in Group 2 above |
| Revised Criteria for SIRS [12]: Requires a minimum of two abnormal measures, one of which must be in Group 1 and the other must be in Group 2 above |
-
aInternational Pediatric Sepsis Conference [14].
In an initial assessment of inter-rater reliability, 22 cases (2.8% of the total) were reviewed jointly by all reviewers and showed a somewhat higher (Conger’s Kappa=0.62) agreement among raters for “severe sepsis” cases than for “sepsis” cases (Conger’s Kappa=0.55) [24]. Thus, all sepsis cases were confirmed by an independent chart re-review by one of three experienced physician reviewers, and the initial low degree of inter-rater agreement for identifying sepsis as corroborated elsewhere [25], [26] prompted our use of the APR-DRG SOI criteria as an independent indicator of physiologic decompensation [13].
The final GS selection comprised 74 cases identified during the 2012 and 2014–2015 periods, representing 0.21% of 35,586 ED patient arrivals. Eight additional GS cases (representing more recent instances with extreme SOI outcomes, including deaths, that might have been prevented by earlier recognition of sepsis) were added to “train” the alert tool as described below; however, these additional cases were not included in calculations of the tool’s performance for the study group.
Selecting metrics associated with infection-related decompensation
From the metrics in Table 1, those with a sufficient association (p≤0.2) with GS cases by univariate analyses were included in a stepwise multiple logistic regression modeling process to identify those factors having a significant independent association (p<0.05) with these cases while adjusting for the effects of each other factor in the model. For this analysis, each factor was assigned a binary score of either “0” or “1” (“1” indicating an “abnormal” value as defined in Table 1). Additionally, our model considered the added effects of all possible two-factor interactions (e.g., “abnormal temperature among neonates,” “low oxygen saturation combined with suspicion of infection,” etc.)
Assigning weights to the selected metrics
For each individual factor and two-factor interactions showing a significant independent association with the GS cases, a weighting factor was assigned reflecting the factor’s relative importance. This process was conducted empirically by first arbitrarily assigning each factor a weight (e.g., from “0” to “3”) and defining tool “firing” to be any combined score greater than or equal to a certain cut-off (e.g., ≥5), then running an iterative program to test all possible combinations of four weights (“0”, “1”, “2”, or “3”) on all selected factors for the patients in the study sample. Testing all possible combinations of four weights was carried out separately for weight ranges “0” to “3” and “1” to “4”, and for CAHR-AT firing cut-offs from “3” to “7”. An exception to this method of assigning and testing possible weights was made for significant two-factor interactions that had a negative association with the GS cases (i.e., “abnormal” values for both factors were associated with a reduced likelihood of infection-related decompensation after adjusting for their individual effects). In these cases, the interaction elements were assigned possible weights of either “0”, “−1”, or “−2”. The weights ultimately chosen for the CAHR-AT were selected from the combination(s) that resulted in the optimal receiver operating characteristics (ROC). As an early screening tool, our selection process prioritized maximizing sensitivity; secondary priorities were to minimize overall firing rate and maximize positive predictive value (PPV) for a final outcome of high SOI (“major” or “extreme”) assigned by the APR-DRG grouper. An additional criterion for our selection of an optimal weight configuration was the consistency of performance between the two patient sample periods (2012 vs. 2014–2015) that comprised our study population.
Improvement process
The initial EHR tool version was implemented in August 2017. Changes to the tool configuration and component weights were subsequently made based on analysis of cases from the second calibration time period (2014–2015) plus newly identified GS cases. Subsequent tool modifications incorporated the two-factor interactions and a series of incremental changes based on project and end-user feedback.
Validation and testing
To test CAHR-AT performance between the 2012 and 2014–2015 GS periods, we performed a split-sample validation of each implemented configuration by comparing tool performance between the two periods. The ROC curves were compared using the area under the curve (AUC) summary metric; an unpaired t-test [27] was used to test the difference in AUC for the two time periods.
Following CAHR-AT implementation, periodic tool performance evaluations included ROC metrics (sensitivity, specificity, and PPV in relation to the GS cases), firing rate, and characterizations of firings by the patients’ SOI outcome. To include more recent cases for tool performance testing, we analyzed an additional retrospective patient sample consisting of all children (totaling 26,026) presenting to the CHKD ED from January through June 2017, just preceding the implementation date. This sample contributed cases to evaluate the tool’s firing rate and PPV for high SOI outcomes.
Results
Characteristics of “gold standard” cases and tool creation
Characteristics of the ED patient population and the 82 children with GS sepsis used during the two study time periods are shown in Table 4. Notably, although 10% of all ED arrivals were assigned a discharge primary service line of “Infectious Disease” (ID) by the APR-DRG® V32 grouper, just 0.2% of arrivals were identified by chart review as GS sepsis or severe sepsis cases. This is consistent with a report from another pediatric ED that 0.3% of patients were treated for severe sepsis within 24 h of arrival [6]. A smaller percentage (0.1%) of our ED arrivals were final diagnosis coded as septicemia, sepsis, severe sepsis, or septic shock, which is consistent with reports that coding routinely under-identifies septic children by clinical criteria [28], [29]. Of our GS cases, only ∼31% were assigned “ID” as their primary service line, indicating that this service line grouper is not a useful identifier of infection-related cases. Overall, 1.2% of all ED arrivals and 10.6% of those subsequently hospitalized were classified as high SOI cases.
Characteristics of the study cohorts (n=35,586).
| Time period of ED arrival | Nov 2012–Dec 2012 | Aug 2014–Jan 2015 | Overall |
|---|---|---|---|
| Characteristics mean (SD) or % (n) | |||
| Age category | 5.9 years (5.5) | 5.3 years (5.1) | 5.7 years (5.4) |
| Neonate (0–4 weeks) | 1.8% (159) | 1.9% (507) | 1.9% (666) |
| Infant (>4 weeks to <2 years) | 33.6% (3,005) | 30.1% (8,029) | 31.0% (11,034) |
| Toddler and preschool (2 to <6 years) | 31.2% (2,787) | 29.9% (7,967) | 30.2% (10,754) |
| School age child (6 to <13 years) | 22.5% (2,007) | 23.8% (6,344) | 23.5% (8,351) |
| Adolescent and young adult (≥13 years) | 10.9% (976) | 14.3% (3,805) | 13.4% (4,781) |
| Total % of overall (n) | 25.1% (8,934) | 74.9% (26,652) | 100% (35,586) |
| Hospitalized as inpatient or observation | 9.6% (860) | 10.1% (2,688) | 10.0% (3,548) |
| Length of stay if hospitalized | 3.2 days (7.7) | 2.9 days (7.0) | 2.9 days (7.1) |
| Infectious disease (ID) primary service linea | 8.4% (751) | 10.5% (2,804) | 10.0% (3,555) |
| Sepsis-related final diagnosisb | 0.1% (9) | 0.1% (21) | 0.1% (30) |
| Gold standard (GS) casesc of severe sepsis or of early sepsis with high-severity outcomes | 0.3% (28) | 0.2% (46) | 0.2% (82)d |
| “ID” primary service linea as % of GS cases identified above | 32.1% (9) | 30.4% (14) | 31.1% (23) |
| High SOI outcomese (all arrivals) | 1.1% (96) | 1.3% (346) | 1.2% (442) |
| High SOI outcomese (if hospitalized) | 9.4% (81) | 11.0% (297) | 10.6% (378) |
| Deaths (all causes, admitted or ED-only) | 0% (0) | 0.02% (4) | 0.01% (4) |
-
aAssigned following discharge based on APR-DRG v32 classification.
-
bAny coded discharge diagnosis of septicemia, sepsis, severe sepsis, or septic shock.
-
cIdentified by chart review process.
-
dIncludes 8 GS cases identified from 01/2017 through 06/2017 added to help calibrate tool.
-
eSeverity of illness (SOI) “major” or “extreme” based on APR-DRG v32 classification.
Characteristics of and improvements to the CAHR-AT algorithm
The evolution of the CAHR-AT algorithm is summarized in Table 5. The original August 2017 algorithm was configured to identify GS cases among the November–December 2012 ED cohort. Incremental improvements, combined with a second calibration period (August 2014–January 2015), led to a new CAHR-AT algorithm (December 2018). Further incremental improvements, most importantly adding two-factor interactions to the elements considered for algorithm inclusion, led to the latest version implemented in October 2019. Each implemented CAHR-AT algorithm followed the same methodology of selecting and optimizing components through the combined use of statistical analysis and an iterative weighting process, with one exception: the factor in Table 5 (“recent [past 48 h] positive blood, CSF, or urine culture result [Cul]”) was included because of its clinical association with sepsis despite a frequency too low to establish a statistical association; this factor was arbitrarily assigned a weight of “2.”
Reassessments/reconfigurations of the CAHR-ATa.
| August 2017 (original alert tool) from 8,934 ED visits |
|
|
|
|
|
| December 2018 from 35,586 ED visits |
|
|
|
|
|
|
|
|
| October 2019 |
|
|
|
|
-
aThe listed changes do not include: minor changes; error corrections; run-time or ease of use improvements; procedures for correcting clinician data entry errors; improvements in evaluating free-text fields; additions of new ICD-10 diagnosis codes or “problem” list and chief complaint variants that trigger the “infection” and “immunocompromise” components of the alert.
-
bAbnormal values for temperature (TMP), heart rate (HR), respiratory rate (RR), or white blood count (WBC); elevated shock index (SI); Emergency Severity Index assigned a value of “1” (ESI); neonate, age ≤28 days (Neo); low oxygen saturation (SpO2); known, suspected, or at increased risk for infection (Inf) or immunocompromise (Imm); recent (past 48 h) positive blood, CSF, or urine culture result (Cul); simultaneous occurrence (interaction) of two listed factors (TMP + Neo, TMP + ESI, SpO2 + Inf).
Performance characteristics and overall CAHR-AT tool improvement over its three iterations are shown in Table 6: sensitivity increased from 70.3 to 77.0%, specificity from 87.8 to 98.1%, PPV for GS cases (PPV-GS) from 6.3 to 7.7%, PPV for high SOI cases (PPV-SOI) increased from 19.4 to 22.5%, and firing rate decreased from 2.4 to 2.0% of ED visits. Sensitivity and specificity refer specifically to the identification of GS sepsis or severe sepsis cases from the two calibration periods. The initial (August 2017) tool configuration showed higher apparent sensitivity, specificity, and PPV-GS, as determined from information derived only from the 2012 period; however, the addition of cases from the 2014–2015 period was instrumental in deriving improved versions of the algorithm. Values for alert firing rate and the predictive value for physiologic decompensation, as measured by PPV-SOI, were improved by combining cases from the January–June 2017 patient sample with the original cohort periods (yielding a total sample size of 61,612), as these latter two metrics are not dependent on the identification of GS cases by chart review.
CAHR-AT characteristics and improvementsa.
| Implementation datea | Sensitivityb (95% CI) | Specificityb (95% CI) | Positive predictive value (PPV-GS)b (95% CI) | Firing ratec (% of ED arrivals)d | Severity of illness “major” or “extreme” (PPV-SOI)d (95% CI) |
|---|---|---|---|---|---|
| August 2017 | 70.3%e (59.9, 80.7) | 97.8% (97.7, 98.0) | 6.3%e (4.7, 8.0) | 2.3% | 19.4% (17.1, 21.6) |
| December 2018 | 77.0% (67.4, 86.6) | 97.7% (97.6, 97.9) | 6.6% (4.9, 8.2) | 2.4% | 17.9% (15.9, 19.8) |
| October 2019 | 77.0% (67.4, 86.6) | 98.1% (97.9, 98.2) | 7.7% (5.8, 9.6) | 2.0% | 22.5% (20.1, 24.8) |
-
aSee Table 5 for explanation of configuration, characteristics, and improvements associated with each date.
-
bRefers to the identification of “gold standard” cases of severe sepsis that developed within 48 h of arrival or in whom early sepsis developed within 24 h with a final SOI of “major” or “extreme” during the two time periods: Nov 2012–Dec 2012 and Aug 2014–Jan 2015 (n=35,586 encounters).
-
cFrequency of a CAHR-AT score ≥5 (firing).
-
dRefers to characteristics of the alert tool during the retrospective time periods Nov 2012–Dec 2012, Aug 2014–Jan 2015, and Jan 2017–Jun 2017 (n=61,612 encounters).
-
eThe sensitivity and PPV for this configuration were originally reported as 78.6 and 9.8%, respectively, based on cases only from the Nov 2012–Dec 2012 time period.
Table 7 shows a more detailed characterization of patients in the 2012, 2014–2015, and 2017 periods identified as CAHR-AT+ using the most recent (October 2019) tool configuration. While 24.3% of these CAHR-AT+ patients were assigned an Emergency Severity Index (ESI) [30] level of “1” (i.e., recognized as high acuity cases at triage), more (33.4%) were assigned relatively low acuity levels (ESI 3–5) suggesting that the potential for decompensation may have been unrecognized. Of admitted patients, alert positive children were hospitalized nearly twice as long as alert negative children (mean LOS 5.2 vs. 2.8 days, p<0.001), which is consistent with the tool’s ability to predict more severe outcomes as measured by the APR-DRG/SOI index.
Characteristics of CAHR-AT + patients in the expanded study groupa for latest (October 2019) tool configuration.
| Characteristics | Mean (SD) or % (N) |
|---|---|
| Total firings % of all cases | 2.0% (1,255) |
| Age category | |
| Neonate (0–4 weeks) | 5.3% (66) |
| Infant (>4 weeks to <2 years) | 22.9% (288) |
| Toddler and preschool (2 to <6 years) | 24.4% (306) |
| School age child (6 to <13 years) | 27.6% (347) |
| Adolescent and young adult (≥13 years) | 19.8% (248) |
| Emergency severity index, ESIb (acuity) | |
| Level 1 | 24.3% (305) |
| Level 2 | 42.3% (531) |
| Level 3 | 20.4% (256) |
| Level 4 | 12.6% (158) |
| Level 5 | 0.4% (5) |
| Diagnostic group (APR-DRG® v32) | |
| Pneumonia, unspecified | 9.1% (114) |
| Asthma | 8.6% (108) |
| Upper respiratory tract infection | 8.3% (104) |
| Fever | 7.1% (89) |
| Bronchiolitis and RSV Pneumonia | 6.3% (79) |
| Kidney and urinary tract infection | 5.2% (65) |
| Major hematological/immunological diagnosis | 4.2% (53) |
| Respiratory signs & symptoms, minor diagnoses | 4.0% (50) |
| Sickle cell crisis | 4.0% (50) |
| Seizure | 4.0% (50) |
| All other (112 diagnostic groups) | 39.2% (493) |
| Hospitalized as inpatient or observation | 62.9% (789) |
| Length of stay (LOS) if hospitalized | 5.2 days (10.7) |
| Admitted to Intensive Care Unit (ICU) | 13.9% (174) |
-
aRepresents 1,255 CAHR-AT firings from a patient sample of 61,612 encounters during all three retrospective time period cohorts.
-
bLevel 1 is highest acuity, Level 5 is lowest.
Validation
The split-sample validation analysis tested the null hypothesis of no difference in AUC between the 2012 and 2014–2015 calibration cohorts. For the latest (October 2019) CAHR-AT configuration, the ROC outcomes were: 2012 time period n=8,934, sensitivity=85.7%, specificity=98.0%, AUC=0.918, standard error (SE)=0.034; 2014–2015 time period n=26,652, sensitivity=71.7%, specificity=98.1%, AUC=0.849, and SE=0.034. The resultant AUC difference (0.069) is not statistically significant (p=0.15). The corresponding analyses for the August 2017 and December 2018 configurations similarly showed no significant difference in AUC between the two periods (p=0.21 and p=0.12, respectively).
Prediction of physiologic decompensation
For the latest (October 2019) tool configuration, the positive and negative predictive values for CAHR-AT firing (maximum score ≥5) for high SOI outcomes were 22.5 and 98.7%, respectively. This contrasts with the low prevalence (1.3%) of high SOI outcomes among our ED arrivals, illustrating the CAHR-AT’s usefulness in predicting these high SOI cases. Among 60,355 patient encounters with SOI data available in the three time period cohorts, patients having maximum CAHR-AT scores below the firing threshold of “5” had a low probability of a high SOI outcome (Figure 1). The probability of a high SOI outcome increased rapidly for scores ≥5, reaching 100% for the few patients scoring “11” or above. Given the SOI metric’s design as an indicator of the ‘‘extent of physiologic decompensation or organ system loss of function’’ [13] reached during the hospital encounter, this finding illustrates the face validity of the CAHR-AT score as a predictor of such physiologic or organ system dysfunction.
Association of maximum ED CAHR-AT Score with outcomes of ‘‘Major’’ or ‘‘Extreme’’ severity of Illness for latest (October 2019) tool configuration.
Discussion
Our goal in developing the CAHR-AT tool was to facilitate early recognition and treatment of infection-related cases having the potential for physiologic decompensation, preferably before such decompensation occurs. Improvements in treatment compliance and timeliness since the implementation of the CAHR-AT at our institution will be the subject of a future report.
A systematic review of the diagnostic accuracy of adult automated electronic sepsis alert systems noted that evaluating these tools is complicated by the lack of a GS sepsis definition and the use of varying thresholds for SIRS criteria. [31]. Similarly, current definitions of pediatric sepsis are of limited value to bedside clinicians to identify cases of sepsis [32], and many of these definitions have poor predictive values and have not been validated, leading to discrepancies in the identification of cases [25]. Most EHR tools to identify pediatric sepsis use vital sign thresholds based on the IPSC criteria [14], which were not evidence based. We observed that the 95th percentile for heart rate and respiratory rate derived from over 1.2 million pediatric ED visits [22] were higher than thresholds used in the Pediatric Advanced Life Support course [33]. There are similar discrepancies in heart rate and respiratory rate thresholds, especially for certain age groups, observed when using the IPSC thresholds, leading to a high rate of SIRS identified in febrile children seen in the ED [10]. Despite variation in GS case definition in different studies, the proportion of GS cases identified in our study (0.21% of ED population) is similar to that observed in other trials (0.2–0.45%) [6], [10], [34].
Aspects of the CAHR-AT development that represent advancements in the early recognition of cases at risk for physiologic decompensation include:
A clinician-based chart review process that was independent of the empirically developed CAHR-AT algorithm to identify GS cases of severe sepsis and cases of early sepsis with high SOI. This resulted in an alert tool that is predictive and that minimizes the introduction of an ‘‘incorporation bias,’’ as described earlier.
We chose a retrospective, rather than prospective, time period for tool calibration to avoid the possibility that tool results affected clinician decision making and treatment that might alter patient outcomes and reduce the tool’s apparent predictive value. Also, using a standard set of patients to evaluate the tool’s performance eliminated the additional variability introduced by a constantly changing patient population represented in a prospective method of evaluation.
Our use of an empirical methodology to identify metrics included in the final algorithm, our inclusion of interactive effects between potential factors, and our iterative method for selecting component weights optimized the algorithm for identifying cases based on information generally available in the EHR for patients arriving at our hospital’s ED.
Using the APR-DRG/SOI index as a measure and predictor of eventual physiologic decompensation resulted in a CAHR-AT scoring system that is quantitatively predictive of this outcome. This expands the utility of the CAHR-AT towards identifying other infection-related cases likely to have more severe outcomes rather than strictly identifying only the rarer ED patients who present with or shortly develop ‘‘gold standard’’ sepsis or severe sepsis, the identification of which is highly variable among clinicians [25], [26].
Future development
A logical extension of the CAHR-AT project is developing a quantitative, predictive EHR-based alert tool for pediatric inpatient units. Such an alert tool would utilize additional inputs (e.g., laboratory results, treatment data [including surgical and respiratory therapy inputs], etc.). As the specific criteria for predicting the likelihood of infection-related decompensation may vary for different service lines (such as hematology/oncology), an ideal alert tool should use a flexible algorithm adapted to these variations within the patient population, including incorporating age-specific definitions of normal and abnormal vital signs that are appropriate for an inpatient, rather than an ED, population. Finally, expanding the CAHR-AT methodology to inpatient units could include criteria for organ dysfunction, which would be consistent with the recommendation to update the definition for sepsis in children [32] to be consistent with the Sepsis-3 criteria used in adult sepsis [35].
Limitations
The CAHR-AT algorithm was based only on metrics readily available at the patients’ arrival or resulted during their ED stay, which limits the predictive value of the tool. Only one rarely available metric — a recent positive culture result —was included in the implemented algorithms due to its known clinical association with GS sepsis. While the inclusion of other metrics, such as metabolic panel, lactate, or procalcitonin results, may have resulted in a higher predictive value, we had no way to assess this as they are rarely ordered in our ED. Similarly, certain indicators incorporated into other sepsis or shock assessment protocols [6], [9], [32], such as prolonged capillary refill time and diminished level of consciousness, did not show a significant association with our GS sepsis cases due to their low frequency of occurrence or to their implementation in our EHR as categorical values that are not well suited for the CAHR-AT.
Although using a sample of over 35,000 patients to identify GS sepsis and severe sepsis was ambitious due to its reliance on manual chart review of possible cases, a larger calibration time period with more GS cases would be preferred. The addition of a second calibration period from August 2014–January 2015 was helpful in improving the tool following its initial implementation. Ultimately, a study population drawn from multiple children’s hospitals should further improve the generalizability of the CAHR-AT.
Conclusions
We created and implemented an ED pediatric assessment tool that identifies and quantitatively predicts the occurrence of physiologic decompensation in cases of infection-related illness. The tool was initially designed to identify GS sepsis cases identified up to 48 h following ED arrival using a novel, empirical iterative method of algorithm creation. Subsequent tool refinement included identifying admitted ED patients with high APR-DRG® SOI to produce a tool with a favorable combination of high sensitivity in predicting GS sepsis cases, high PPV for physiologic decompensation, and low overall firing rate. Our results suggest that this tool can play an important role in a program designed to optimize the delivery of critical treatments for children at risk for decompensation associated with infection.
Funding source: U.S. Department of Education
Award Identifier / Grant number: Unassigned
Acknowledgments
The authors would like to specially thank Elizabeth Martinez for her lead role in clinical process improvement for the CAHR project. We also wish to thank the many other members of our team at Children’s Hospital of The King’s Daughters, including Christine Bovat, Adam Campbell, DeAnn Capanna, Thomas Cholis, Yvette Conyers, Regina Dawson, Sanaz Devlin, Dee Deyarmond, Meaghan Dominy, Lindsay Eilers, Dawn Jennings, Louellen Klein, Chris Mangum, Anne McEvoy, Coleman McGehee, Scott Padgett, Laura Sass, James Schmidt, Cecilia Sotomayor, Catherina Stewart, and Philip Thomas.
-
Research funding: This project was partially funded by an unrestricted grant from the Children’s Health Foundation of the Children’s Health System, Norfolk, VA.
-
Author contributions: RJS, SAG, and ALZ each made substantial contributions to the conception and design of the study. RJS conducted the data acquisition and analyses and drafted the manuscript. All three authors contributed to the interpretation of the data, critically revised the manuscript for important intellectual content, and were involved in IRB and grant applications. All authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.
-
Competing interests: Authors state no conflict of interest.
-
Informed consent: Informed consent was waived for all individuals included in this study.
-
Ethical approval: This project was approved by the Institutional Review Board at Eastern Virginia Medical School under a waiver of informed consent as a review of existing medical records conducted as a quality improvement initiative.
-
Presentations: Parts of this research were presented at the 2018 and 2019 American Academy of Pediatrics (AAP) National Meetings in Orlando, FL (11/02/2018) and New Orleans, LA (10/26/2019), and at the 2019 Institute for Healthcare Improvement (IHI) National Forum in Orlando, FL (12/02/2019).
References
1. Weiss, SL, Fitzgerald, JC, Pappachan, J, Wheeler, D, Jaramillo-Bustamante, JC, Salloo, A, et al. Global epidemiology of pediatric severe sepsis: the sepsis prevalence, outcomes, and therapies study. Am J Respir Crit Care Med 2015;191:1147–57. https://doi.org/10.1164/rccm.201412-2323oc.10.1164/rccm.201412-2323OCSearch in Google Scholar PubMed PubMed Central
2. Rivers, E, Nguyen, B, Havstad, S, Ressler, J, Muzzin, A, Knoblich, B, et al. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med 2001;345:1368–77. https://doi.org/10.1056/nejmoa010307.10.1056/NEJMoa010307Search in Google Scholar PubMed
3. Rivers, EP, Katranji, M, Jaehne, KA, Brown, S, Abou Dagher, G, Cannon, C, et al. Early interventions in severe sepsis and septic shock: a review of the evidence one decade later. Minerva Anestesiol 2012;78:712–24.Search in Google Scholar
4. Sepsis and children. Internet video. San Diego: Sepsis Alliance; 2017. Available from: https://www.youtube.com/watch?v=RsxTvzF6OG8 [Accessed 12 May 2020].Search in Google Scholar
5. Sepsis: a hidden crisis exposed. Internet video. New York: The Rory Staunton Foundation; 2015. Available from: https://www.youtube.com/watch?v=t4FQrRRTUnY [Accessed 12 May 2020].Search in Google Scholar
6. Balamuth, F, Alpern, ER, Abbadessa, MK, Hayes, K, Schast, A, Lavelle, J, et al. Improving recognition of pediatric severe sepsis in the emergency department: contributions of a vital sign-based electronic alert and bedside clinician identification. Ann Emerg Med 2017;70:759–68 e2. https://doi.org/10.1016/j.annemergmed.2017.03.019.10.1016/j.annemergmed.2017.03.019Search in Google Scholar PubMed PubMed Central
7. Cruz, AT, Williams, EA, Graf, JM, Perry, AM, Harbin, DE, Wuestner, ER, et al. Test characteristics of an automated age- and temperature-adjusted tachycardia alert in pediatric septic shock. Pediatr Emerg Care 2012;28:889–94. https://doi.org/10.1097/pec.0b013e318267a78a.10.1097/PEC.0b013e318267a78aSearch in Google Scholar PubMed
8. Cruz, AT. Between the devil and the deep blue sea: use of real-time tools to identify children with severe sepsis in the pediatric emergency department. Ann Emerg Med 2017;70:769–70. https://doi.org/10.1016/j.annemergmed.2017.04.006.10.1016/j.annemergmed.2017.04.006Search in Google Scholar PubMed
9. Lloyd, JK, Ahrens, EA, Clark, D, Dachenhaus, T, Nuss, KE. Automating a manual sepsis screening tool in a pediatric emergency department. Appl Clin Inform 2018;9:803–8. https://doi.org/10.1055/s-0038-1675211.10.1055/s-0038-1675211Search in Google Scholar PubMed PubMed Central
10. Scott, HF, Deakyne, SJ, Woods, JM, Bajaj, L. The prevalence and diagnostic utility of systemic inflammatory response syndrome vital signs in a pediatric emergency department. Acad Emerg Med 2015;22:381–9. https://doi.org/10.1111/acem.12610.10.1111/acem.12610Search in Google Scholar PubMed
11. Horeczko, T, Green, JP. Emergency department presentation of the pediatric systemic inflammatory response syndrome. Pediatr Emerg Care 2013;29:1153–8. https://doi.org/10.1097/pec.0b013e3182a9e629.10.1097/PEC.0b013e3182a9e629Search in Google Scholar PubMed
12. Sepanski, RJ, Godambe, SA, Mangum, CD, Bovat, CS, Zaritsky, AL, Shah, SH. Designing a pediatric severe sepsis screening tool. Front Pediatr 2014;2:56. https://doi.org/10.3389/fped.2014.00056.10.3389/fped.2014.00056Search in Google Scholar PubMed PubMed Central
13. Averill, RF, McCullough, EC, Goldfield, N, Hughes, JS, Bonazelli, J, Bentley, L, et al. 3M APR DRG classification system, version 31.0: methodology overview. Wallingford, CT: 3M Health Information Systems; 2013. Available from: https://www.hcup-us.ahrq.gov/db/nation/nis/grp031_aprdrg_meth_ovrview.pdf [Accessed 12 May 2020].Search in Google Scholar
14. Goldstein, B, Giroir, B, Randolph, A, International Consensus Conference on Pediatric Sepsis. International Pediatric Sepsis Consensus Conference: definitions for sepsis and organ dysfunction in pediatrics. Pediatr Crit Care Med 2005;6:2–8. https://doi.org/10.1097/01.pcc.0000149131.72248.e6.10.1097/01.PCC.0000149131.72248.E6Search in Google Scholar PubMed
15. Angus, DC, van der Poll, T. Severe sepsis and septic shock. N Engl J Med 2013;369:840–51. https://doi.org/10.1056/nejmra1208623.10.1056/NEJMra1208623Search in Google Scholar
16. Ray, S, Cvetkovic, M, Brierley, J, Lutman, DH, Pathan, N, Ramnarayan, P, et al. Shock index values and trends in pediatric sepsis: predictors or therapeutic targets? A retrospective observational study. Shock 2016;46:279–86. https://doi.org/10.1097/shk.0000000000000634.10.1097/SHK.0000000000000634Search in Google Scholar
17. Rousseaux, J, Grandbastien, B, Dorkenoo, A, Lampin, ME, Leteurtre, S, Leclerc, F. Prognostic value of shock index in children with septic shock. Pediatr Emerg Care 2013;29:1055–9. https://doi.org/10.1097/pec.0b013e3182a5c99c.10.1097/PEC.0b013e3182a5c99cSearch in Google Scholar
18. Kohn, MA, Carpenter, CR, Newman, TB. Understanding the direction of bias in studies of diagnostic test accuracy. Acad Emerg Med 2013;20:1194–206. https://doi.org/10.1111/acem.12255.10.1111/acem.12255Search in Google Scholar
19. Angus, DC, Linde-Zwirble, WT, Lidicker, J, Clermont, G, Carcillo, J, Pinsky, MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med 2001;29:1303–10. https://doi.org/10.1097/00003246-200107000-00002.10.1097/00003246-200107000-00002Search in Google Scholar
20. Balamuth, F, Weiss, SL, Neuman, MI, Scott, H, Brady, PW, Paul, R, et al. Pediatric severe sepsis in U.S. children’s hospitals. Pediatr Crit Care Med 2014;15:798–805. https://doi.org/10.1097/pcc.0000000000000225.10.1097/PCC.0000000000000225Search in Google Scholar
21. Balamuth, F, Weiss, SL, Hall, M, Neuman, MI, Scott, H, Brady, PW, et al. Identifying pediatric severe sepsis and septic shock: accuracy of diagnosis codes. J Pediatr 2015;167:1295–1300 e4. https://doi.org/10.1016/j.jpeds.2015.09.027.10.1016/j.jpeds.2015.09.027Search in Google Scholar
22. Sepanski, RJ, Godambe, SA, Zaritsky, AL. Pediatric vital sign distribution derived from a multi-centered emergency department database. Front Pediatr 2018;6:66. https://doi.org/10.3389/fped.2018.00066.10.3389/fped.2018.00066Search in Google Scholar
23. Garner, JS, Jarvis, WR, Emori, TG, Horan, TC, Hughes, JM. CDC definitions for nosocomial infections, 1988. Am J Infect Contr 1988;16:128–40. https://doi.org/10.1016/0196-6553(88)90053-3.10.1016/0196-6553(88)90053-3Search in Google Scholar
24. Gwet, KL. Handbook of inter-rater reliability: the definitive guide to measuring the extent of agreement among raters, 4th ed. Gaithersburg, MD: Advanced Analytics, LLC; 2014.Search in Google Scholar
25. Weiss, SL, Fitzgerald, JC, Maffei, FA, Kane, JM, Rodriguez-Nunez, A, Hsing, DD, et al. Discordant identification of pediatric severe sepsis by research and clinical definitions in the SPROUT international point prevalence study. Crit Care 2015;19:325. https://doi.org/10.1186/s13054-015-1055-x.10.1186/s13054-015-1055-xSearch in Google Scholar PubMed PubMed Central
26. Rhee, C, Kadri, SS, Danner, RL, Suffredini, AF, Massaro, AF, Kitch, BT, et al. Diagnosing sepsis is subjective and highly variable: a survey of intensivists using case vignettes. Crit Care 2016;20:89. https://doi.org/10.1186/s13054-016-1266-9.10.1186/s13054-016-1266-9Search in Google Scholar PubMed PubMed Central
27. Gonen, M. Analyzing receiver operating characteristic curves with SAS. Cary, NC: SAS Institute Inc; 2007.Search in Google Scholar
28. Rhee, C, Dantes, R, Epstein, L, Murphy, DJ, Seymour, CW, Iwashyna, TJ, et al. Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014. J Am Med Assoc 2017;318:1241–9. https://doi.org/10.1001/jama.2017.13836.10.1001/jama.2017.13836Search in Google Scholar PubMed PubMed Central
29. Rhee, C, Jentzsch, MS, Kadri, SS, Seymour, CW, Angus, DC, Murphy, DJ, et al. Variation in identifying sepsis and organ dysfunction using administrative vs electronic clinical data and impact on hospital outcome comparisons. Crit Care Med 2019;47:493–500. https://doi.org/10.1097/ccm.0000000000003554.10.1097/CCM.0000000000003554Search in Google Scholar PubMed PubMed Central
30. Green, NA, Durani, Y, Brecher, D, DePiero, A, Loiselle, J, Attia, M, et al. Emergency severity index version 4: a valid and reliable tool in pediatric emergency department triage. Pediatr Emerg Care 2012;28:753–7. https://doi.org/10.1097/pec.0b013e3182621813.10.1097/PEC.0b013e3182621813Search in Google Scholar PubMed
31. Makam, AN, Nguyen, OK, Auerbach, AD. Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: a systematic review. J Hosp Med 2015;10:396–402. https://doi.org/10.1002/jhm.2347.10.1002/jhm.2347Search in Google Scholar PubMed PubMed Central
32. Schlapbach, LJ, Kissoon, N. Defining pediatric sepsis. JAMA Pediatrics 2018;172:313–4. https://doi.org/10.1001/jamapediatrics.2017.5208.10.1001/jamapediatrics.2017.5208Search in Google Scholar PubMed
33. Pediatric Advanced Life Support: provider manual. Dallas, TX: American Heart Association; 2016.Search in Google Scholar
34. Balamuth, F, Alpern, ER, Grundmeier, RW, Chilutti, M, Weiss, SL, Fitzgerald, JC, et al. Comparison of two sepsis recognition methods in a pediatric emergency department. Acad Emerg Med 2015;22:1298–306. https://doi.org/10.1111/acem.1281410.10.1111/acem.12814Search in Google Scholar PubMed PubMed Central
35. Singer, M, Deutschman, CS, Seymour, CW, Shankar-Hari, M, Annane, D, Bauer, M, et al. The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 2016;315:801–10. https://doi.org/10.1001/jama.2016.0287.10.1001/jama.2016.0287Search in Google Scholar PubMed PubMed Central
© 2020 Robert J. Sepanski et al., published by De Gruyter, Berlin/Boston
This work is licensed under the Creative Commons Attribution 4.0 International License.
