Article Text
PDF
Abstract
Background In 2016, the CDC in the USA proposed codes from the International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) for identifying traumatic brain injury (TBI). This study estimated positive predictive value (PPV) of TBI for some of these codes.
Methods Four study sites used emergency department or trauma records from 2015 to 2018 to identify two random samples within each site selected by ICD-10-CM TBI codes for (1) intracranial injury (S06) or (2) skull fracture only (S02.0, S02.1-, S02.8-, S02.91) with no other TBI codes. Using common protocols, reviewers abstracted TBI signs and symptoms and head imaging results that were then used to assign certainty of TBI (none, low, medium, high) to each sampled record. PPVs were estimated as a percentage of records with medium-certainty or high-certainty for TBI and reported with 95% confidence interval (CI).
Results PPVs for intracranial injury codes ranged from 82% to 92% across the four samples. PPVs for skull fracture codes were 57% and 61% in the two university/trauma hospitals in each of two states with clinical reviewers, and 82% and 85% in the two states with professional coders reviewing statewide or nearly statewide samples. Margins of error for the 95% CI for all PPVs were under 5%.
Discussion ICD-10-CM codes for traumatic intracranial injury demonstrated high PPVs for capturing true TBI in different healthcare settings. The algorithm for TBI certainty may need refinement, because it yielded moderate-to-high PPVs for records with skull fracture codes that lacked intracranial injury codes.
- traumatic brain injury
- Surveillance
- epidemiology
- coding systems
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
Traumatic brain injury (TBI) is a temporary or permanent disruption of normal brain function due to an external force.1 2 Public health agencies in the USA have conducted TBI surveillance using healthcare billing datasets coded in International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM), at least since the 1990s.3–5 On 1 October 2015, healthcare systems in the USA implemented International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM). For TBI surveillance, the CDC proposed use of ICD-10-CM diagnosis codes for intracranial injury; skull fracture without crushing skull injury; injury to optic chiasm or pathways, injury to visual cortex; crushing injury of skull; and shaken infant syndrome in preparation for this change and before data coded in ICD-10-CM was available to validate their proposed code set.6
This study is the first to estimate the PPV of CDC-proposed TBI codes, one component of a comprehensive validation. The objective was to estimate PPV of TBI for the ICD-10-CM diagnosis codes specific to intracranial injury and skull fracture in emergency department (ED) billing records. This study focused on less severe TBI, such as concussion, that does not require inpatient hospitalisation.
Methods
The study design was a retrospective, cross-sectional review of medical records at ED sites in four states: Colorado, Kentucky, Maryland and Massachusetts. To assess PPV for TBI, each study site team selected two random samples of ED billing records based on any ICD-10-CM diagnosis code for: (1) only skull fracture without other proposed TBI codes or (2) intracranial injury (S06) with or without other proposed TBI codes.6 (In Maryland, one site of the two sites was a neurotrauma referral centre, not an ED, and its trauma registry was used for sampling.) To replicate methods of US public health surveillance of TBI-related ED visits, the sampling frame was limited to records that documented initial medical encounters for patients treated in EDs and discharged home during 2015–2018 (Faul et al 1, p. 49). Reviewers recorded signs and symptoms related to TBI and head imaging findings from the medical records for each sample.
Case definitions
The study focused on ICD-10-CM codes for skull fracture and intracranial injury as the potential TBI cases of interest in ICD-10-CM-coded ED discharge records, grouped as follows:
Fracture of skull bones: ICD-10-CM codes S02.0, S02.1-, S02.8- or S02.91 for initial encounter (seventh character in the ICD-10-CM code of ‘A’ or ‘B’ or missing) in any discharge diagnostic field (first-listed or secondary diagnosis), without concurrent ICD-10-CM diagnosis code for intracranial injury (S06); and
Intracranial injury: ICD-10-CM codes S06 for initial encounter (seventh character of ‘A’ or missing) in any diagnostic field, with or without a code for skull fracture or other proposed TBI codes (S04.02 for injury to optic chiasm, S04.03- for injury to optic pathways, S04.04- for injury to visual cortex, S07.1 for crushing injury of skull or T74.4 for shaken infant syndrome).6
TBI, especially mild TBI treated in the ED, can manifest in transient signs and symptoms. Therefore, the study team developed a case definition that represented levels of certainty regarding TBI based on the presence of TBI-related clinical documentation in the medical record. With input from TBI experts at CDC, the study team adapted an unpublished, self-reported definition for concussion surveillance and a clinical TBI definition to create rank-ordered levels of certainty of TBI based on the number and type of symptoms present in the medical record and imaging findings (table 1).7 This study definition used the same symptoms as the CDC concussion definition except time of symptom onset after the injury event, because the records lacked precise time of onset.
Certainty of TBI based on signs, symptoms and imaging result: multisite review of ICD-10-CM codes for intracranial injury and skull fracture treated in EDS
Supplemental material
Shown in table 2, the study team implemented the study in various types and numbers of hospitals in the four states, reflecting differences in public health authority in the states and the large geographic size and distribution of hospitals in one state (Colorado). The study site in Kentucky was a two-hospital system with a Level 1 trauma centre that cares for a large majority of patients with brain injury in the eastern half of Kentucky. The study sites in Maryland were the R. Adams Cowley Shock Trauma Center, part of the University of Maryland Medical System, and the Johns Hopkins Hospital. Due to triage and transfer protocols in Maryland, emergency medical services take the majority of adults with TBI directly to the Shock Trauma Center, the primary adult neurotrauma referral centre for Maryland, and most paediatric patients with TBI to Johns Hopkins Hospital.
Setting, locations, relevant dates and other study characteristics: multisite review of ICD-10-CM codes for intracranial injury and skull fracture treated in EDs
Study population and data source
The study population was persons treated in the ED or shock trauma centre for an intracranial injury or skull fracture and discharged home. The study used two data sources: (1) administrative billing records for ED visits to select eligible records and (2) the corresponding original medical record, to collect TBI signs, symptoms and imaging findings. Staff limited the ED billing records to records for patients who were state residents. Selecting state residents replicated selection criteria used in TBI surveillance (Faul et al 1, p49). The sites had different lag time in receiving ED billing data or needed a longer time to accumulate at least 385 eligible ED records for the review, resulting in different study periods for each site.
Variables of interest from medical records
The reviewers at the study sites collected TBI-related signs and symptoms as well as findings from any imaging of the head by X-ray, CT scan or MRI. Reviewers confirmed that the medical record indicated a recent injury event preceding the signs and symptoms. Signs and symptoms variables were those typical of TBI presentation: loss of consciousness; being dazed, foggy or confused; other memory problems; nausea/vomiting; headache or pressure in head; dizziness/poor balance; change in vision; poor concentration; sensitivity to noise or light; irritability or change in mood or personality; drowsiness or change in sleep (particularly in infants and very young children); speech problems; hearing problems; or weakness/numbness. Due to the typically short time between an injury event and patient arrival at an ED, some signs and symptoms might have resolved or could have occurred after discharge from the ED. The Kentucky study team added a variable that asked their clinical nurse reviewers, based on the complete record review, to give their clinical opinion by selecting one of these choices: No TBI, Possible TBI, Probable TBI or TBI.
To reduce systematic differences (observer bias) among the reviewers and over time, the study team developed a common abstraction form to record the variables of interest from the original medical record and a detailed reviewer manual with definitions and examples, including clinical terms. Based on input from subject matter experts, the study team developed a list of clinical terms and phrases to assist medical record reviewers in determining whether findings from a CT, MRI or X-ray image of the head were positive, suspect or negative for TBI (see online supplemental file for list.)
Study size
To determine the desired sample size, the study team used 90% as the expected PPV of TBI for both intracranial injury and skull fracture, based on unpublished data from states’ CDC-funded TBI surveillance during the ICD-9-CM era. Each study site chose a random sample of records to provide 385 reviews and achieve a margin of error no larger than 3% for the 95% CI of the estimated PPV.
Statistical methods
Staff from the four states independently calculated the counts and percentage of reviewed medical records that met the criteria for each level of TBI certainty, using uniform categorisation of TBI certainty (table 1). Staff calculated certainty of TBI for the sample with an ICD-10-CM code for intracranial injury and repeated this analysis with the sample for skull fracture. Although they had not been validated, the levels of TBI certainty provided a method of sensitivity analysis in this study to examine the extent to which the constellation of signs, symptoms and diagnostic imaging affects the PPV and to determine whether there is a level of certainty at which the PPV was similar across the four study sites, given that two sites used clinical researchers or nurses and had access to the full electronic health record in those hospitals. Data analysis was performed using SAS V.9.3 or V.9.4, and IBM SPSS V.23.0.
Results
Table 3 presents the results of the signs and symptoms inventories for the traumatic intracranial injury samples. The majority of ED medical records at all four study sites had documentation of at least one of the following signs or symptoms after an injury event: dazed/foggy/confused, memory problems or loss of consciousness. However, the proportion of each sign or symptom varied by study site. Loss of consciousness was the most common of these three signs and symptoms at the study sites in Maryland, Kentucky and Colorado. The study sites in Maryland (referral centres) had the largest proportion with loss of consciousness. Headache/pressure in the head and nausea or vomiting were the most common signs and symptoms among the ones given less weight in the level of TBI certainty at all four study sites. In the Maryland study site, 88.6% of the intracranial injury records documented imaging of the head, and 33.6% of the findings were positive for TBI. In the Kentucky study site, 71.2% of the medical records documented imaging of the head, and 19.5% had positive or suspected TBI findings. In Colorado, 61.4% had imaging and only 4.3% (of all 376 records) had a positive finding for TBI. In Massachusetts, 61.1% had imaging and only 5.5% (of all 388 records) had a positive finding for TBI.
TBI-related signs and symptoms and imaging findings among a random sample of medical records assigned traumatic intracranial injury ICD-10-CM codes*, by study site
When defining PPV based on the highest certainty of TBI (table 4), the TBI PPV of the sample selected using ICD-10-CM intracranial injury codes was 87.7% (95% CI: 84.8% to 90.9%) in Maryland and 74.0% (95% CI: 69.3% to 78.3%) in Kentucky, 74.5% (95% CI: 69.8% to 78.8%) in Colorado and 70.4% (95% CI: 65.5% to 74.9%) in Massachusetts. Based on high and medium certainty levels, PPV in three sites (82.1%–86.1%) approached but did not reach the 91.7% PPV for TBI in Maryland. The Kentucky clinical reviewers, based on complete record review assessment, determined that 218 (76.5%) patients whose record had a traumatic intracranial injury code had a TBI or probable TBI out of the 285 records with high certainty of TBI based on the study algorithm (data not shown). Similarly, in the clinical reviewer’s opinion, 26 (83.9%) patients had a TBI or probable TBI out of the 31 records that had a medium certainty of TBI based on the algorithm.
PPV for TBI based on a review of ED records with traumatic intracranial injury ICD-10-CM codes*, by study site
Table 5 presents signs, symptoms and imaging results for the skull fracture sample. The proportion of the skull fracture sample with documented loss of consciousness or any Group one or any Group two signs or symptoms was lower compared with the intracranial injury sample. The most common Group two symptom was headache/pressure in the head. Almost all skull fracture records in Maryland, Kentucky and Colorado study sites indicated imaging and the proportion with imaging findings positive for TBI varied: 23.1%, 34.5%, and 64.8% respectively. In Massachusetts, 83.8% had imaging, and 72.5% of the 388 records had positive findings.
TBI-related signs and symptoms and imaging findings among a random sample of medical records assigned skull fracture ICD-10-CM codes*, by study site
Based on the highest certainty of TBI (table 6), PPVs for the sample selected using ICD-10-CM codes for skull fracture ranged from 54.2% (95% CI: 49.4% to 59.0%) for Maryland to 77.7% (95% CI: 73.1% to 81.9%) in Colorado. Variability in PPV across study sites remained when PPV was expanded to include medium certainty of TBI, with a PPV of 57.0% in Maryland, 61.0% in Kentucky, 84.5% in Colorado and 82.0% in Massachusetts. Based on a complete record review assessment, the Kentucky clinical reviewers determined that 106 (48.4%) patients had a TBI or probable TBI out of the 219 records with a high certainty of TBI based on the study algorithm (data not shown), and 5 (31.3%) patients had a TBI or probable TBI out of the 16 records with a medium certainty of TBI.
PPV for TBI based on a review of ED records with skull fracture ICD-10-CM codes*, by study site
Discussion
Surveillance strategies that yield accurate findings support the development of sound public health policy and effective allocation of resources. In this multisite study, the PPVs of intracranial injury ICD-10-CM codes to identify TBI demonstrate an acceptable level of accuracy to justify using these codes for public health surveillance. The TBI PPV of intracranial injury codes stabilised at 82%–92% across the four sites when PPV included both high and medium evidence of TBI. TBIs treated only in the ED are likely mild TBIs, which justify calculating PPV based on signs, symptoms and imaging findings representing high and medium certainty of TBI.
A traumatic skull fracture without an intracranial injury is not a clinical TBI. However, public health surveillance of TBI included skull fracture diagnoses only, due to the proportion that represent missed TBI and to limit the risk of under-reporting TBI.1 7 This multisite study found more than half of records with a skull fracture code but no code for traumatic intracranial injury had evidence of a TBI with high certainty. The skull fracture records from the two study sites that used clinical reviewers had lower PPVs for TBI and had a larger proportion without any evidence of TBI, compared with results from the two sites that used professional coders. These multisite findings suggest that the untested algorithm used retrospectively to assign certainty of TBI may yield an excessive number of false-positive cases. The additional information from one site on clinical impression further supports that the algorithm or its application identified a large proportion of false-positive TBIs. In developing the algorithm, the study team sought feedback from fewer clinicians than public health practitioners on the level of certainty method and the criteria for positive findings from imaging the head (details are in the online supplemental file 1). As a result, the analytical approach towards a broad public health definition prevailed based on previous public health surveillance, as indicated by the algorithm categorising positive findings of skull fractures from head imaging as a TBI.1
This study was limited by the absence of a validated method for retrospective categorisation of transient signs, symptoms and imaging into levels of certainty that a TBI has occurred, reflecting the clinically challenging nature of TBI’s presentation and functional deficits.2 7 Our method assigned more weight (‘high certainty’ of TBI) to any of three signs and symptoms, two of which (loss of consciousness and memory problems) had a higher prevalence in the intracranial injury samples in the two study sites with university hospitals as referral centres and clinical reviewers. Without an objective diagnostic aid for TBI and a validated consensus clinical definition, the nature and extent of bias (false positive or false negative TBIs) are unknown. Our study highlights the need for further validation and testing of the algorithm to improve the methodology for medical record case confirmation studies on TBI.
This study had limited scope. The study assessed only PPV and not NPV, sensitivity and specificity to fully validate the proposed ICD-10-CM codes for identifying TBI. The findings in a related review of medical records for ED visits with ICD-10-CM diagnosis codes for unspecified head injury suggest that there are missed cases of TBI.8
This study did not assess all potential threats to validity of the results, for example, clinical documentation practices and the influence of medical or electronic health record software on documentation across the four study areas. The two-hospital study sites with more uniform electronic health records might have more consistent clinical documentation. The impact of record type inconsistency on the validity of documentation or diagnosis coding is unknown. The reviewer type and access to medical information varied across areas: the clinical reviewers in two study areas accessed electronic health records, while a professional coder in another study area had access to electronic health records or scanned copies of medical records, and professional coders in the fourth area accessed copies of medical records. The coders in the latter two areas did not always receive all the medical documents requested, notably the emergency medical services pre-hospital report. This study did not assess ICD-10-CM coding training or hospital coding practices.
Differences among study settings justified not pooling study results. Results across the four study areas must be compared with caution, considering the known differences in the study areas (table 2). However, knowing the range of estimated TBI PPV of ICD-10-CM codes in a variety of ED settings can be a strength. For example, regardless of whether the study reviewers were clinical or professional coders, the reviewers found documentation consistent with a TBI in the records with ICD-10-CM codes for intracranial injury.
This study has implications for public health practice. This study found that the ICD-10-CM codes for intracranial injury identified TBIs in the four samples at an acceptable level of accuracy (PPV 82%–92%), despite differences in study sites and study limitations. The PPV for the ICD-10-CM skull fracture codes with no concurrent intracranial injury code was lower than that for the sample with intracranial injury codes. One might expect the TBI PPV of the skull fracture codes to be even lower, if the original medical coders assigned accurately all the ICD-10-CM diagnosis codes to reflect all the physician documentation in the medical record. This finding suggests that before using the ICD-10-CM codes for skull fractures, applied epidemiologists will want to consider the number and type of hospital EDs represented in their ED billing dataset and select a similar study site, if comparable. The PPV finding for skull fracture or the lack of a comparable study site can inform the epidemiologist’s decision to use skull fracture codes to identify TBI in their ED billing dataset and to better interpret results of any skull fracture analysis. Further research can address the limitations of this study, especially validating a method for retrospective confirmation of a true TBI and assessing the sensitivity and specificity of the CDC’s proposed TBI ICD-10-CM codes. Public health practice would benefit from such research.
What is already known on the subject
On 1 October 2015, the USA implemented the International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM).
Before ICD-10-CM coded data were widely available in the USA, the CDC proposed in 2016 a set of ICD-10-CM codes for population surveillance of traumatic brain injury that included codes for traumatic intracranial injury and skull fracture.
What this study adds
In study sites in four states, more than two-thirds of randomly selected emergency department records with ICD-10-CM diagnosis code(s) for traumatic intracranial injury included documentation of (1) loss of consciousness, confusion or memory problems after an injury event; (2) a positive finding from imaging of the head; or (3) three or more signs or symptoms suggestive of a traumatic brain injury. This definition of ‘high’ certainty of TBI resulted in PPVs of TBI for intracranial injury codes ranging from 70% to 89%.
The PPVs of the ICD-10-CM diagnosis code(s) for traumatic intracranial injury ranged from 82% to 92% in samples of emergency department records from study sites in four states, based on high certainty and medium certainty (fewer signs or symptoms or imaging results indicated suspected TBI).
In study sites in four states, about a third of randomly selected emergency department records with ICD-10-CM diagnosis code(s) for skull fracture (without a code for traumatic intracranial injury) documented loss of consciousness, confusion or memory problems after an injury event.
The PPVs of the ICD-10-CM codes for skull fracture (without a code for traumatic intracranial injury), based on documented evidence for medium-to-high certainty of TBI, was 57% and 61% in two sites with clinical reviewers in university/trauma hospitals and 82% and 85% for two study sites with professional medical coders reviewing statewide or nearly statewide samples. These findings suggest the untested algorithm used retrospectively to assign certainty of TBI may yield an excessive number of false-positive cases.
Acknowledgments
The authors thank the following contributors: Matthew J. Breiding, PhD and Alexis B. Peterson, PhD for methodological guidance; Alcy R. Torres, MD for answering select clinical questions; Jeffrey Yu, MPH for comments on a draft manuscript; Sharon Emmerling for data collection at Colorado sites; Huong Luu, MD, PhD, MPH and GYeon Oh, PhD for analysis of Kentucky data; and reviewers and project leads at the University of Maryland School of Medicine National Study Center for Trauma and Emergency Medical Systems and Johns Hopkins Center for Injury Research and Policy for data collection and analysis at the Maryland sites.
Footnotes
Contributors All authors contributed substantially to the study design, interpretation of the findings, drafting the manuscript and critically revising it. Authors BAG, JJ, JEH and BH analysed the data. All authors read and approved the final manuscript.
Funding This work was supported in part by the Centers for Disease Control and Prevention through the following cooperative agreements: 6NU17CE924841-03 to Colorado; 6NU17CE924846-03 to Kentucky; 6NU17CE924835-03 to Massachusetts; 5NU17CE924831-03 to Maryland.
Disclaimer The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of the CDC in the United States Department of Health and Human Services, the State of Colorado, the State of Kentucky, the State of Maryland or the State of Massachusetts.
Competing interests None declared.
Patient and public involvement statement Patients or the public were not involved in the design, or conduct, or reporting of our research.
Patient consent for publication Not required.
Ethics approval The Medical Institutional Review Board of the University of Kentucky approved a waiver of informed consent process for protocol IRB# 43762 to conduct record-review research on the basis that the review poses minimal risk, does not involve direct interaction with subjects, will be conducted in a secure environment that minimises risk of breach, will not adversely affect the rights and welfare of subjects, and could not practicably be carried out without the requested waiver due to the volume of records. The following institutional review boards (IRB) approved the study in Maryland with a waiver of informed consent. The Maryland Department of Health IRB approved protocol #17-30, the University of Maryland, Baltimore IRB approved protocol HCR-HP-00073895-2, the Johns Hopkins School of Public Health IRB approved protocol #00009503. Colorado and Massachusetts have state laws and Board of Health reporting rules that allow the state public health agency to access medical records without patient consent for specific, limited conditions that include trauma (Colorado) and all injuries (Massachusetts).
Provenance and peer review Commissioned; externally peer reviewed.
Data availability statement Data are not available. The data use agreements governing this work do not allow sharing individual record-level data. The instruction manual for the medical records review is available upon request.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.