Validation of the ICD/AIS MAP for pediatric use
- 1Department of Pediatrics, The Children's Hospital of Philadelphia and Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania School of Medicine
- 2Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania School of Medicine
- 3Center for Injury Research and Policy, Johns Hopkins University School of Hygiene and Public Health
- Correspondence to: Dr Dennis R Durbin, Center for Clinical Epidemiology and Biostatistics, Room 711 Blockley Hall, 423 Guardian Dr, Philadelphia, PA 19104, USA
Objective—To determine the performance of the ICD/AIS MAP (© E J MacKenzie et al) as a method of classifying injury severity for children.
Methods—Data on all children less than 16 years of age admitted to all designated trauma centers in Pennsylvania from January 1994 through October 1996 were obtained from the state trauma registry. The ICD/AIS MAP was used to convert all injury related ICD-9-CM diagnosis codes into abbreviated injury scale (AIS) score and injury severity score (ISS). Agreement between trauma registry AIS and ISS scores and MAP generated scores was assessed using the weighted κ (κw) coefficient for ordered data and the intraclass correlation coefficient for continuous data.
Results—Agreement in ISS scores was excellent, both overall (intraclass correlation coefficient = 0.86, 95% confidence interval (CI) 0.84 to 0.89)), and when grouped into three levels of severity (κw= 0.86, 95% CI 0.85 to 0.87). Agreement in AIS scores across all body regions and ages was also excellent, (κw= 0.86 (95% CI 0.83 to 0.87). Agreement increased with age (κw= 0.78 for children <2 years; κw= 0.86 for older children) and varied by body region, though was excellent across all regions.
Conclusions—The performance of the ICD/AIS MAP in assessing severity of pediatric injuries was equal to or better than previous assessments of its performance on primarily adult patients. Its performance was excellent across the pediatric age range and across nearly all body regions of injury.
Classification of injuries and individual trauma patients by severity is central to conducting research evaluating outcomes of injured patients. The abbreviated injury scale (AIS) and its derivative, the injury severity score (ISS), are the most widely recognized severity scoring systems based on anatomic descriptors.1,2 The AIS and ISS have been used extensively to compare outcomes of injured patients treated in trauma center versus non-trauma center hospitals,3 to assess the adequacy of triage in organized systems of trauma care,4 and to monitor changing trends in injury hospitalizations.5,6
A major drawback to the widespread use of the AIS is the need to review the entire medical record for scoring. In order for the AIS and ISS to be used in large, population based epidemiological studies, MacKenzie and others developed a computerized mapping system for converting injury related International Classification of Diseases, ninth revision (ICD-9-CM ) rubrics into AIS scores (© E J MacKenzie et al).7 The ICD/AIS assignments were validated by comparing results obtained by the computerized map to AIS and ISS scores obtained after a full medical record review.8 The ICD/AIS MAP (MAP) has subsequently been used in trauma outcome studies that utilize existing sources of hospital discharge data.5,9 However, these studies have focused primarily on evaluating adult trauma patients.
It is widely acknowledged that children differ from adults with respect to mechanisms, severity, and outcomes of injury, necessitating an evaluation of the performance of the MAP on a strictly pediatric population of injured patients. Therefore, the objective of this study was to assess the performance of the ICD/AIS MAP in classifying injury severity for children by determining the agreement between AIS and ISS scores assigned by the MAP to scores that are determined in a more traditional fashion.
This cross sectional study included all children less than 16 years of age admitted to all designated trauma centers in Pennsylvania from 1 January 1994 to 30 October 1996. Data were obtained from the Pennsylvania Trauma Systems Foundation, an independent agency that designates trauma centers in Pennsylvania and maintains the statewide trauma registry. During the time period of study there were 26 designated trauma centers, consisting of two level I pediatric trauma centers, three level I trauma centers with additional qualifications in pediatric trauma, nine level I trauma centers, and 12 level II trauma centers.
Subjects in the trauma registry included all emergency department and in-hospital deaths, all admissions >72 hours, all admissions to an intensive care unit, and all patients transferred either into or out of a trauma center. Up to 27 diagnoses for each patient were included in the trauma registry. Comprehensive text descriptions of each injury were generated by trauma nurse coordinators at each institution. These individuals then converted the text descriptions into both ICD-9-CM and AIS codes using TRICODE (Tri-Analytics, Inc) for entry into the trauma registry. This computer assisted coding system has been shown to reduce coding variability.10 Extensive human checks (including on-site audits of randomly selected patients) and computer checks for data completeness, reliability, and correctness are performed by personnel at the Pennsylvania Trauma Systems Foundation to enhance the data integrity. TRICODE independently assigns ICD and AIS codes to the injuries based on detailed text descriptions. AIS scores range from 0 (no injury) to 6 (virtually unsurvivable injury) for each of eight body regions (head/neck, face, abdomen, thorax, spine, upper and lower extremities, and external/burns). The ISS is then determined by first squaring and then summing the highest AIS scores for each of the three most seriously injured body regions. ISS scores range from 1–75. Any AIS 6 (fatal) injury is automatically coded as an ISS score of 75. Scores determined by TRICODE are referred to as “Registry AIS” and “Registry ISS” scores.
The ICD/AIS MAP-90 (an updated version of the MAP using AIS-90 scores) was used to convert all injury related ICD-9-CM diagnosis codes into AIS and ISS scores, referred to herein as ICD/AIS and ICD/ISS scores. Measures of agreement were computed using the weighted κ (κw) coefficient11 (quadratic weights) for ordered categories (low, moderate, high severity) of ISS and ICD/ISS and for the six level AIS scores, both overall, and separately for each body region. In addition, agreement was determined separately for different age groups to explore variation in performance of the MAP across the pediatric age range. For ISS and ICD/ISS measured on a scale from 1–75, we used the intraclass correlation coefficient.11 For analyses involving grouped ISS scores, the following severity groups were defined: “low” <13, “moderate” 13–19, and “high” ≥20 as has been done in previous research.4
Confidence intervals (CI) for these measures were computed using a clustered bootstrap with 1000 samples, selected without replacement, of the 26 hospitals.12 In this resampling scheme, selection of a hospital means sampling all of its patients. The bootstrap was used to account for the possibility of clustering of observations within hospitals, such as might occur when individual institutions employ ICD-9-CM coding conventions that are idiosyncratic to those hospitals. The variance assuming complete independence of observations was less than the bootstrap variance, suggesting that the bootstrap methods are not only appropriate but also necessary for these analyses. All computations of agreement and accuracy were done in STATA v 6.0 (College Station TX: Stata Corp, 1999).
A total of 8739 children were included in the study. Basic descriptive information about the study sample is provided in table 1. The majority of children suffered a blunt injury (86.3%) and were discharged alive (95.8%). Pedal cyclists, falls, motor vehicle occupants, and pedestrians accounted for the majority (83.7%) of mechanisms of injury. Nearly half of the sample (n = 4286) had more than one injury diagnosis.
AGREEMENT IN ISS SCORES
The distribution of ISS scores for each method of calculation is presented in fig 1. For both systems, the majority of patients (approximately 70% in each system) had scores <11. The intraclass correlation coefficient for agreement between registry ISS and ICD/ISS scores was 0.86, 95% CI 0.84 to 0.89, indicating excellent agreement.13 The κw coefficient for overall agreement in grouped scores (that is, low, moderate, high severity) was 0.86, 95% CI 0.85 to 0.87 also indicating excellent agreement. The mean difference between the two scores was 1.3 (ICD/ISS lower) and 88% of subjects had scores within ±5 of each other. Agreement between the two scores varied somewhat with severity. Given a level of severity as determined by the registry, the probability that the MAP produced the same level of severity was 0.97, 95% CI 0.96 to 0.98 for low severity, 0.71, 95% CI 0.66 to 0.76 for moderate severity, and 0.68, 95% CI 0.66 to 0.71 for high severity injuries. Agreement did not vary substantially by whether the patient had one injury diagnosis (κw= 0.79, 95% CI 0.76 to 0.82) or more than one injury (κw= 0.86, 95% CI 0.85 to 0.87).
There were 38 subjects in whom one of the two systems assigned a score of 75 (maximum) while the other system did not. In 29 of these cases, the registry assigned the score of 75 while the MAP assigned scores ranging from 4–50. In only 12 cases, the registry assigned a score of 75 while the MAP assigned a score <20, suggesting significant disagreement. All of these patients had suffered a head injury that was ICD coded as moderate (n = 3), brief (n = 2), or no (n = 2) loss of consciousness, or with loss of consciousness not otherwise specified (n = 5). According to the AIS-90 manual, these injuries should not be assigned AIS scores of 6 as they were in the registry.
An interesting pattern was noted among the nine patients assigned ISS scores of 75 by the MAP but not by the registry. Six of these patients suffered a lacerated heart or aorta which was assigned an ICD/AIS score of 6 and a registry AIS score of 5. The other three patients suffered spinal cord injuries and were also assigned ICD/AIS scores of 6 and registry AIS scores of 5.
AGREEMENT IN AIS SCORES
The distribution of maximum AIS (MAIS) scores for each method of calculation is presented in fig 2. Overall agreement in MAIS scores across all body regions and ages of children was excellent13, with a κw= 0.86 (95% CI 0.83 to 0.87). Agreement was similar for both blunt (κw= 0.87, 95% CI 0.85 to 0.88) and penetrating (κw= 0.90, 95% CI 0.88 to 0.91) injuries.
ICD/AIS MAP accurately characterizes injury severity in children.
Injury severity characterization of MAP equal to or better than that in adults.
Injury severity characterization excellent across pediatric age range and across all body regions.
ICD/AIS MAP can be utilized in studies focusing only on injured children.
As noted in table 2, agreement in MAIS scores varied both by body region and with the age of the patient. For the following analyses, age was categorized as <2 years, 2–4 years, 5–9 years, and 10–15 years. In general, agreement was somewhat lower for children <2 years of age (κw= 0.78) than for older children (κw= 0.86). While agreement varied somewhat by body region, it was generally excellent (κw ≥0.75) for nearly every region among the youngest children and increased with age.
Our results suggest that there is overall excellent agreement in AIS and ISS scores as determined by the ICD/AIS MAP with scores based on text descriptions of the injury determined using the TRICODE method. As with previous studies evaluating the MAP, ICD/ISS scores were, on average, slightly lower than registry ISS scores, owing to conservative assumptions made by the MAP when there is a lack of precision in the ICD coding. Of note, agreement was similarly excellent for both penetrating and blunt injuries.
While overall agreement in ISS scores grouped into three levels of severity was excellent, it declined somewhat as severity increased. To explore this further, we examined in detail those cases in which one of the systems assigned a score of 75 while the other did not. In the majority of these cases, the registry assigned the score of 75. In the few cases where the MAP assigned a significantly lower score, the disagreement was due to imprecision in the ICD coding and apparent over-scoring of injury severity in the registry. It is possible that, at the time of chart abstraction by the trauma registrar, knowledge of the patient's outcome may have influenced the AIS coding but was not reflected in the ICD coding. AIS scoring based on the TRICODE software removes some of the inconsistencies in traditional coding but it still relies on the abstractor providing the correct description of the injury. The levels of agreement reported likely reflect a “best case scenario” but also reflect what is possible with more accurate and reliable ICD coding.
Agreement in MAIS scores was generally excellent across the pediatric age range and across all body regions. Of note, the performance of the MAP in assessing severity of pediatric injuries was equal to or better than previous assessments of its performance on primarily adult patients. In a recent evaluation of the MAP, MacKenzie noted κw scores for agreement in body region specific MAIS scores ranging from 0.64 for face injuries to 0.90 for thoracic injuries (personal communication). Examples of corresponding levels of agreement between MacKenzie's work and results of this study included κw= 0.79 for head injuries (compared with 0.87 in our study) and κw= 0.82 for abdominal injuries (0.89 in our study). These results indicate that the MAP performs at least as well as, and in many particular situations, better in children than in adults.
The improved performance of the MAP on children compared with adults may be due to the potential for less diagnostic uncertainty, and somewhat lower severity injuries in children. Adults more frequently have multiple and more severe injuries. Because the performance of the MAP varies with injury severity, one would expect its accuracy to be somewhat less in a more severely injured population.
As hospital discharge data have become increasingly available, use of the ICD/AIS MAP for the study of large populations of injured patients has increased.4,5,9 Previous studies have typically included mixed populations with adults far outnumbering children in the samples. Results of our study suggest that the ICD/AIS MAP can be used in similar studies on strictly pediatric populations.
This work was supported by grant number R49CCR312693 from the National Center for Injury Prevention and Control, Centers for Disease Control and Prevention. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Center for Injury Prevention and Control, Centers for Disease Control and Prevention.