Statistics from Altmetric.com
Injury prevention researchers and practitioners in several countries have argued for diagnostic and cause coding of all injuries resulting in inpatient hospital treatment.1 The international standard that has been promoted in this respect is the International Classification of Diseases (ICD), the latest revision ICD-10.2
There have been a limited number of published studies that have examined the accuracy of coding of the full range of injury and external cause codes, under ICD-9, using large general population samples.13–5 Only one of these examined the accuracy of codes for diagnosis groups (eg, lower limb fracture, poisoning) and external cause code groups (eg, falls, homicide/assault).5 To date, there are no comparable published studies using ICD-10. This is important as ICD-10 represented a major change to the ICD coding structure, especially for external causes.6
While country-specific studies of the reliability of coding are obviously important for the country in question, the results from such studies may also be of significant benefit to the international community, as they provide insight into potential problems of coding and thus may provide guidance on where the focus of training should be placed and where changes to ICD may be warranted.
New Zealand (NZ) is one of a small number of countries that have been coding the diagnoses and external causes of injury for hospital inpatients for over 30 years. Since July 1999, diagnoses and external causes of injury in NZ have been coded using ICD-10-AM, the Australian Modification of ICD-10.7
To provide an initial picture of the reliability of ICD-10 coding, the aims of this study were to determine the accuracy of coding for principal injury diagnosis (PDx), external cause (E-code), place of occurrence, and activity under ICD-10-AM for public hospital discharges in NZ.
NZ Health Information Service (NZHIS) are the custodians of the National Minimum Dataset (NMDS), which includes records of all publicly funded discharges from NZ hospitals.8 Individuals who required inpatient treatment after injury form a subset of the NMDS. For 2002, it was estimated that 99% of all hospital injury discharges in NZ were publicly funded, thus the NMDS offers an accurate representation of injuries resulting in hospital attendance.910
A simple random sample of 1800 injury discharges was obtained from the NMDS for the period in which the second edition of ICD-10-AM was in use (July 2001 to June 2004).7 A sample of this size allowed the 95% CI around the overall estimate of coding error to be no wider than ± 5%. This interval represented a balance between precision and the increased cost associated with setting a narrower interval.
The sample was limited to those discharges that met all the following criteria:
the PDx was injury (S00–T98);
the discharge was the first for the injury (ie, not a readmission);
the discharge was publicly funded.
For each discharge event in the sample, the Ministry of Health’s Senior Advisor in Clinical Coding (SACC) independently coded the PDx, E-code, place of occurrence, and activity of the injuries from the hospital’s medical records. Rather than using coding software, the SACC coded these events using the ICD-10-AM manuals.7 Provision was made for the SACC to note which events had unclear documentation. The SACC also completed a “hospital survey” for each hospital that had discharge events in the sample to determine the numbers of coders and their use of coding software.
For diagnosis codes, up to five characters can be used (eg, S32.81 is fracture of ischium). Most E-codes (V01–Y89) are denoted using only three or four characters. Use of a fifth character is possible for a very limited range of codes.
The first character of a diagnosis code indicates whether the injury related to a single-body region (S) or to multiple or unspecified body regions (T). Three characters are required to obtain a diagnosis code or an E-code with the minimum level of specificity. The fourth and fifth characters of diagnosis codes are used to denote further details about the injury, eg, body site. The fourth character of an E-code is used to denote, for example, the type of firearm involved or whether the transport accident was traffic or non-traffic.
In chapter XX (External causes of morbidity and mortality) are supplementary factors related to causes of morbidity and mortality classified elsewhere (Y90–Y98). “Place of occurrence of external cause of injury” codes begin with Y92. Ten general locations including Y92.9 (“unspecified place of occurrence”) are denoted using a fourth character. Provision exists for a fifth character for Y92.2 (“school, other institution and public administrative area”).
The first three characters of all “Activity related to the external cause of injury” codes are Y93. Seven general activity codes including Y93.9 (“unspecified activity”) are denoted using a fourth character. Provision exists for a fifth character (0–9), for Y93.0 (“while engaged in sports”), to denote the particular sporting activity. Activity codes are not assigned for Y35–Y36 (“legal intervention and operations of war”), Y40–Y84 (“complications of medical and surgical care”), and Y85–Y89 (“sequelae of external causes of morbidity and mortality”).
The PDx, E-code, place of occurrence, and activity codes selected by the SACC were compared with those recorded in the NMDS. Any difference observed between the NMDS code and that selected by the SACC was considered to be due to an inaccuracy in the NMDS code. The denominator used in the calculation of the percentage of incorrect discharges was the number of NMDS records in the sample. PDxs were categorized in two ways: diagnoses groups (eg, S10–S19 (“injuries to the neck”)) and broad nature of injury groups based on the Barell matrix (eg, “fracture”).11
Results include percentages, 95% binomial exact confidence intervals, and two-sample tests of proportion. Stata V9.2 was used for the analysis.
Of the 1800 discharge events from 48 hospitals in the sample, 50 could not be coded by the SACC for logistic reasons (eg, medical records not available or not accessible at the time of the study). The remaining 1750 events related to discharges for 1747 people from 45 hospitals. Against the study protocol that all discharges should be coded in ICD-10-AM second edition, the SACC inadvertently coded the PDx, E-code, and place of occurrence code in ICD-10-AM third edition, for three, 15, and nine discharge events, respectively. One E-code was missing from the NMDS. These events were removed from the analysis, giving denominators for the PDx, E-code, and place of occurrence code comparisons of 1747, 1734, and 1741, respectively. As 202 discharge events had E-codes that were ineligible for activity coding in the NMDS, the denominator for the activity code was 1548.
Just over a quarter of the PDxs recorded in the NMDS differed in the first, second, third, fourth, or fifth character from that proposed by the SACC (table 1). The error rate in the first character (eg, S vs T; S vs M) was small (4%). This increased to 14% when the first three characters were considered (eg, S00 (“superficial injury of the head”) vs S01 (“open wound of the head”)). Of the 248 PDxs that contained an error in the first three characters, 52 (21%) had the PDx as selected by the SACC listed as the first additional diagnosis on the NMDS. Nine percent of the diagnoses were correct in the first three characters but contained an error in the fourth character (eg, S04.1 (“injury of oculomotor nerve”) vs S04.2 (“injury of trochlear nerve”)).
A comparison of diagnostic groups indicates variation in the accuracy of the PDx (table 2). Of the injuries related to single body regions, “injuries to the neck” were most often inaccurately coded. When the first three characters were considered, the observed greater inaccuracy for neck injuries (32%) was significantly different from “injuries to the hip and thigh” (10%), “injuries to the shoulder and upper arm” (8%), “injuries to the elbow and forearm” (8%), and “injuries to the knee and lower leg” (5%).
Of injuries related to multiple or unspecified body region, “burns” were least likely to contain an error in the first three characters of the PDx (6%), although when all five characters were considered, 21% contained an error. In comparison, only 13% of PDxs categorized as “poisoning by drugs, medicaments, and biological substances” were incorrect in any character.
Accuracy of the PDx also varied significantly by nature of injury (table 3). When all characters were considered, “dislocation”, “toxic effects”, and “unspecified injuries” had the highest rates of error of the nature of injury groupings with more than 10 discharges.
“Fracture” was least likely to contain an error in the first three characters of the PDx (4%), although, when all characters were considered, 18% contained an error.
The SACC coded 37 (2%) of the PDxs outside ICD-10 chapter XIX (ie, outside S00–T98). For 12 of these, the PDx was coded to ICD-10 chapter XIII (Diseases of musculoskeletal system & connective tissue; M00–M99).
The error in the first character for E-codes (ie, V vs W vs X vs Y) was 7% (table 1). Twenty percent of E-codes had the first character correct but the second or third character incorrect (eg, X72 (“intentional self-harm by handgun discharge”) vs X74 (“intentional self-harm by other and unspecified firearm discharge”)). When all characters were considered, 29% of the E-codes were incorrect.
Of the E-code groupings in table 4 with more than 10 discharges, “intentional self-harm” was the most accurately coded, with only 14% containing an error in the first three characters. This estimate was significantly lower than that observed for “assault” (25%), “falls” (30%), and “other non-transport accidents” (30%). A substantial proportion of the inaccuracies in any character occurred in the fourth character for “medical and surgical complications” and “other land transport accidents”.
Error rates in the place of occurrence and activity codes were 22% and 29%, respectively (table 5). For both place of occurrence and activity codes, almost all the inaccuracies occurred in the first character in which the code is actually specified (ie, the fourth character).
The SACC indicated that the documentation provided in the medical records was unclear for 172 of the 1750 discharges. Records with unclear documentation were significantly more likely to contain an error somewhere in the PDx (difference 16%; 95% CI 8% to 24%) compared with the remaining records. Similarly, the E-codes from medical records with unclear documentation were more likely to be inaccurate (difference 10%; 95% CI 2% to 18%). There was no evidence to suggest that accuracy was related to clarity of documentation for place of occurrence and activity codes.
Of the 45 hospitals from which the sample was obtained, four were not eligible for the “hospital survey”, as all their coding occurred at other hospitals. Of the remaining hospitals, 83% used coding software. The median length of time for which software had been used was 6 years (minimum 1 year, maximum 12 years). The number of coders at each hospital varied from 1 to 28 (median 3.5). Of hospitals with one coder, 67% used coding software. In hospitals with more than one coder, all coders used coding software apart from one hospital where one of six coders did not use coding software.
These findings do not compare favorably with the assessment of ICD-9-CM-A coding in NZ, which reported that 4% of cases contained an error in the first three characters of their PDx, with only 5% containing an error in any character.5 The present study reports error rates more than 3–5 times higher (14% and 26%, respectively). Interestingly, the quality of PDx coding of all-cause (rather than injury-specific) hospital discharge events does not suggest higher error rates for ICD-10 than for ICD-9.1213 The error rate for E-coding in the present study was also in excess of that reported in the ICD-9 study (26% error in the first three characters vs 15%).
It is difficult to compare changes in accuracy of the place of occurrence code from ICD-9-CM-A to ICD-10-AM. In the ICD-9 study, the coding error for place of occurrence, given that the first four characters of the E-code were correct, was 8%, whereas in this study the inaccuracy of the place of occurrence (22%) was assessed independently of the E-code. Activity coding under ICD-9-CM-A was very limited, making it impossible to compare the accuracy with ICD-10-AM.
Although the change in the classification from ICD-9-CM-A to ICD-10-AM may have contributed to the higher error rates reported in this study, it is unlikely that this would explain the dramatic increase in errors. PDx groupings for ICD-10 S00–S99 are based on body regions, whereas, in ICD-9, groupings for codes in the range 800–899 are based on types of injuries. The groupings are more similar for ICD-10-AM T00–T98 and ICD-9-CM-A 900–999. Despite this, table 2 suggests that the rate of coding error for S and T PDxs is very similar, with errors of 26% and 28%, respectively. The level of coding specificity available does not explain the errors either, as ICD-9-CM-A second edition has 2394 injury diagnosis codes compared with only 1783 in ICD-10-AM second edition (C Lewis, personal communication: Information Analyst, New Zealand Health Information Service, 2007).
One explanation for the higher rate of observed inaccuracies in the present study compared with the ICD-9 study may be that the percentage of medical records containing unclear documentation has increased. As the ICD-9 study did not record information on the quality of documentation, this cannot be explored further.
As the SACC used manuals rather than coding software to code the discharge events, the question arises as to whether coding software is contributing to coding inaccuracies. The opinion of the SACC is that coding software is not infallible. In this study, use of coding software among the hospitals was high, with the software having been used for, on average, 6 years. Coding software usage during the period of the ICD-9 study (1996–1998) was less than it was during the years covered by this study. The greater inaccuracy observed in this study could be partly due to the initiation of, and associated learning difficulties with, coding software. Further research should investigate whether the use of coding software is more likely to produce inaccuracies than coding using the ICD manuals.
One of the limitations of this study was that there may be some degree of error in the SACC’s coding. It was our intention to have mismatches plus a sample of other discharges reviewed by a second coder, but unfortunately there was no similarly experienced clinical coder available in NZ at this time.
The variation in accuracy of the PDx by diagnostic group indicates that greater attention is required when coding neck injuries, in particular. Of injuries related to multiple or unspecified body region, “effects of foreign bodies”, “toxic effects”, and “complications” were notably poorly coded. Of the E-codes, “falls” and “other non-transport accidents” had high error rates. Greater care should also be made when coding the fourth character of “medical and surgical complications” and “other land transport accidents”.
Given that the rate of inaccuracy in the PDxs and E-codes was greater for discharges with medical records containing unclear documentation, care should be taken by the attending doctors to ensure that note taking is as thorough as possible. Ensuring doctors have sufficient time for note-taking and specific doctor education programs may improve coding.
Users of the ICD-10-AM codes from the NMDS can have a moderate degree of confidence in the first three characters of injury diagnoses. A similar degree of confidence is not warranted for E-coding at this level. A low degree of confidence is also warranted in place of occurrence and activity coding. As considerable research and policy advice relies on injury and external-cause incidence estimates from the NMDS, it is imperative that urgent consideration is given to how coding errors can be reduced.
For countries that are considering implementing ICD-10 or one of its variants, these findings provide insight into possible limitations of the classification and offer guidance on where the focus of training should be placed. For countries that have historical data coded according to ICD-10-AM, these results suggest that some specific estimates of injury and external-cause incidence may need to be treated with caution.
This research was funded by the Health Research Council of NZ. We thank NZHIS, in particular Chris Lewis, for their assistance with this project, Daniel Russell for writing the data collection program, and Colin Cryer and Pam Smartt for their helpful reviews of an earlier version of this paper.
Competing interests: None.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.