Methods

This cross sectional study included all children less than 16 years of age admitted to all designated trauma centers in Pennsylvania from 1 January 1994 to 30 October 1996. Data were obtained from the Pennsylvania Trauma Systems Foundation, an independent agency that designates trauma centers in Pennsylvania and maintains the statewide trauma registry. During the time period of study there were 26 designated trauma centers, consisting of two level I pediatric trauma centers, three level I trauma centers with additional qualifications in pediatric trauma, nine level I trauma centers, and 12 level II trauma centers.

Subjects in the trauma registry included all emergency department and in-hospital deaths, all admissions >72 hours, all admissions to an intensive care unit, and all patients transferred either into or out of a trauma center. Up to 27 diagnoses for each patient were included in the trauma registry. Comprehensive text descriptions of each injury were generated by trauma nurse coordinators at each institution. These individuals then converted the text descriptions into both ICD-9-CM and AIS codes using TRICODE (Tri-Analytics, Inc) for entry into the trauma registry. This computer assisted coding system has been shown to reduce coding variability.¹⁰ Extensive human checks (including on-site audits of randomly selected patients) and computer checks for data completeness, reliability, and correctness are performed by personnel at the Pennsylvania Trauma Systems Foundation to enhance the data integrity. TRICODE independently assigns ICD and AIS codes to the injuries based on detailed text descriptions. AIS scores range from 0 (no injury) to 6 (virtually unsurvivable injury) for each of eight body regions (head/neck, face, abdomen, thorax, spine, upper and lower extremities, and external/burns). The ISS is then determined by first squaring and then summing the highest AIS scores for each of the three most seriously injured body regions. ISS scores range from 1–75. Any AIS 6 (fatal) injury is automatically coded as an ISS score of 75. Scores determined by TRICODE are referred to as “Registry AIS” and “Registry ISS” scores.

The ICD/AIS MAP-90 (an updated version of the MAP using AIS-90 scores) was used to convert all injury related ICD-9-CM diagnosis codes into AIS and ISS scores, referred to herein as ICD/AIS and ICD/ISS scores. Measures of agreement were computed using the weighted κ (κ_w) coefficient¹¹ (quadratic weights) for ordered categories (low, moderate, high severity) of ISS and ICD/ISS and for the six level AIS scores, both overall, and separately for each body region. In addition, agreement was determined separately for different age groups to explore variation in performance of the MAP across the pediatric age range. For ISS and ICD/ISS measured on a scale from 1–75, we used the intraclass correlation coefficient.¹¹ For analyses involving grouped ISS scores, the following severity groups were defined: “low” <13, “moderate” 13–19, and “high” ≥20 as has been done in previous research.⁴

Confidence intervals (CI) for these measures were computed using a clustered bootstrap with 1000 samples, selected without replacement, of the 26 hospitals.¹² In this resampling scheme, selection of a hospital means sampling all of its patients. The bootstrap was used to account for the possibility of clustering of observations within hospitals, such as might occur when individual institutions employ ICD-9-CM coding conventions that are idiosyncratic to those hospitals. The variance assuming complete independence of observations was less than the bootstrap variance, suggesting that the bootstrap methods are not only appropriate but also necessary for these analyses. All computations of agreement and accuracy were done in STATA v 6.0 (College Station TX: Stata Corp, 1999).

Results

STUDY POPULATION

A total of 8739 children were included in the study. Basic descriptive information about the study sample is provided in table 1. The majority of children suffered a blunt injury (86.3%) and were discharged alive (95.8%). Pedal cyclists, falls, motor vehicle occupants, and pedestrians accounted for the majority (83.7%) of mechanisms of injury. Nearly half of the sample (n = 4286) had more than one injury diagnosis.

AGREEMENT IN ISS SCORES

The distribution of ISS scores for each method of calculation is presented in fig 1. For both systems, the majority of patients (approximately 70% in each system) had scores <11. The intraclass correlation coefficient for agreement between registry ISS and ICD/ISS scores was 0.86, 95% CI 0.84 to 0.89, indicating excellent agreement.¹³ The κ_w coefficient for overall agreement in grouped scores (that is, low, moderate, high severity) was 0.86, 95% CI 0.85 to 0.87 also indicating excellent agreement. The mean difference between the two scores was 1.3 (ICD/ISS lower) and 88% of subjects had scores within ±5 of each other. Agreement between the two scores varied somewhat with severity. Given a level of severity as determined by the registry, the probability that the MAP produced the same level of severity was 0.97, 95% CI 0.96 to 0.98 for low severity, 0.71, 95% CI 0.66 to 0.76 for moderate severity, and 0.68, 95% CI 0.66 to 0.71 for high severity injuries. Agreement did not vary substantially by whether the patient had one injury diagnosis (κ_w= 0.79, 95% CI 0.76 to 0.82) or more than one injury (κ_w= 0.86, 95% CI 0.85 to 0.87).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Distribution of ICD/ISS and registry ISS scores. ICD/ISS: scores determined by ICD/AIS MAP; registry ISS scores: scores contained in the trauma registry determined through TRICODE.

There were 38 subjects in whom one of the two systems assigned a score of 75 (maximum) while the other system did not. In 29 of these cases, the registry assigned the score of 75 while the MAP assigned scores ranging from 4–50. In only 12 cases, the registry assigned a score of 75 while the MAP assigned a score <20, suggesting significant disagreement. All of these patients had suffered a head injury that was ICD coded as moderate (n = 3), brief (n = 2), or no (n = 2) loss of consciousness, or with loss of consciousness not otherwise specified (n = 5). According to the AIS-90 manual, these injuries should not be assigned AIS scores of 6 as they were in the registry.

An interesting pattern was noted among the nine patients assigned ISS scores of 75 by the MAP but not by the registry. Six of these patients suffered a lacerated heart or aorta which was assigned an ICD/AIS score of 6 and a registry AIS score of 5. The other three patients suffered spinal cord injuries and were also assigned ICD/AIS scores of 6 and registry AIS scores of 5.

AGREEMENT IN AIS SCORES

The distribution of maximum AIS (MAIS) scores for each method of calculation is presented in fig 2. Overall agreement in MAIS scores across all body regions and ages of children was excellent¹³, with a κ_w= 0.86 (95% CI 0.83 to 0.87). Agreement was similar for both blunt (κ_w= 0.87, 95% CI 0.85 to 0.88) and penetrating (κ_w= 0.90, 95% CI 0.88 to 0.91) injuries.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Distribution of ICD/MAIS and registry MAIS scores. ICD/MAIS: scores determined by ICD/AIS MAP; registry MAIS scores: scores contained in the trauma registry determined through TRICODE.

As noted in table 2, agreement in MAIS scores varied both by body region and with the age of the patient. For the following analyses, age was categorized as <2 years, 2–4 years, 5–9 years, and 10–15 years. In general, agreement was somewhat lower for children <2 years of age (κ_w= 0.78) than for older children (κ_w= 0.86). While agreement varied somewhat by body region, it was generally excellent (κ_w ≥0.75) for nearly every region among the youngest children and increased with age.

Discussion

Our results suggest that there is overall excellent agreement in AIS and ISS scores as determined by the ICD/AIS MAP with scores based on text descriptions of the injury determined using the TRICODE method. As with previous studies evaluating the MAP, ICD/ISS scores were, on average, slightly lower than registry ISS scores, owing to conservative assumptions made by the MAP when there is a lack of precision in the ICD coding. Of note, agreement was similarly excellent for both penetrating and blunt injuries.

While overall agreement in ISS scores grouped into three levels of severity was excellent, it declined somewhat as severity increased. To explore this further, we examined in detail those cases in which one of the systems assigned a score of 75 while the other did not. In the majority of these cases, the registry assigned the score of 75. In the few cases where the MAP assigned a significantly lower score, the disagreement was due to imprecision in the ICD coding and apparent over-scoring of injury severity in the registry. It is possible that, at the time of chart abstraction by the trauma registrar, knowledge of the patient's outcome may have influenced the AIS coding but was not reflected in the ICD coding. AIS scoring based on the TRICODE software removes some of the inconsistencies in traditional coding but it still relies on the abstractor providing the correct description of the injury. The levels of agreement reported likely reflect a “best case scenario” but also reflect what is possible with more accurate and reliable ICD coding.

Agreement in MAIS scores was generally excellent across the pediatric age range and across all body regions. Of note, the performance of the MAP in assessing severity of pediatric injuries was equal to or better than previous assessments of its performance on primarily adult patients. In a recent evaluation of the MAP, MacKenzie noted κ_w scores for agreement in body region specific MAIS scores ranging from 0.64 for face injuries to 0.90 for thoracic injuries (personal communication). Examples of corresponding levels of agreement between MacKenzie's work and results of this study included κ_w= 0.79 for head injuries (compared with 0.87 in our study) and κ_w= 0.82 for abdominal injuries (0.89 in our study). These results indicate that the MAP performs at least as well as, and in many particular situations, better in children than in adults.

The improved performance of the MAP on children compared with adults may be due to the potential for less diagnostic uncertainty, and somewhat lower severity injuries in children. Adults more frequently have multiple and more severe injuries. Because the performance of the MAP varies with injury severity, one would expect its accuracy to be somewhat less in a more severely injured population.

As hospital discharge data have become increasingly available, use of the ICD/AIS MAP for the study of large populations of injured patients has increased.^4,5,9 Previous studies have typically included mixed populations with adults far outnumbering children in the samples. Results of our study suggest that the ICD/AIS MAP can be used in similar studies on strictly pediatric populations.