Article Text
Abstract
Background: Injury researchers are increasingly using the US National Trauma Data Bank (NTDB). However, there are some methodological issues that might threaten the validity of studies that use this database for injury research.
Methods: Two methodological issues were evaluated: clustering of patients within trauma centers and missing data. To illustrate how these issues might affect the results of a study, the following four analytical approaches that evaluated the association between patients’ blood alcohol concentration (BAC) in the emergency department (ED), patients’ resource utilization, and ED or hospital disposition were compared: (A) deleting subjects with missing BAC and ignoring clustering of patients within trauma centers; (B) deleting subjects with missing BAC while taking into account clustering; (C) using imputed values for patients’ BAC and ignoring the clustering issue; (D) using the imputed data while taking into account clustering.
Results: Adjustment for clustering of patients within trauma centers increased the CIs in models B and D. The results of the analyses based on imputed data showed that estimates based on complete case analysis were biased. For example, the odds ratio for the use of a head CT scan fell from 1.84 (95% CI 1.49 to 2.28) in approach B to 1.26 (95% CI 0.98 to 1.64) in approach D.
Conclusions: Excluding patients with missing values for BAC in studies that evaluate the association between this variable and patients’ resource utilization and ED or hospital disposition, using the NTDB, led to biased estimates. Furthermore, ignoring the clustering design led to artificially narrow CIs.
Statistics from Altmetric.com
The American College of Surgeons’ National Trauma Data Bank (NTDB) is the largest trauma registry in the USA. By 2006, 650 trauma centers, including 124 level I and 139 level II trauma centers from 43 states and the District of Columbia, had voluntarily contributed more than 2 million records to the registry.1 Owing to the large number of subjects and trauma centers, researchers are increasingly using this databank to explore different aspects of injury epidemiology and trauma care in the USA. We used the National Center for Biotechnology Information online search engine (http://www.ncbi.nlm.nih.gov/entrez/query.fcgi) to estimate the number of published peer-reviewed articles that have referred to this database. The phrase “National Trauma Data Bank” was reflected in the title, abstract, or key words of 54 published articles. Twenty of these papers were published in 2006, 16 in 2005, 10 in 2004, five in 2003, and one in 2002, suggesting increased utilization of this invaluable data source.
In spite of the importance of the NTDB, there are a few methodological issues that might affect study design, data analysis, and, in turn, results and interpretation of findings. Firstly, the NTDB is a compilation of data from different trauma care facilities. Because of measured and unmeasured characteristics such as age, sex, type and mechanism of injury, and the quality of pre-hospital and hospital care, patient outcomes are almost certain to be more similar within each trauma center than to patient outcomes across different trauma centers.23 This correlation of patient outcomes within centers violates the main assumption behind standard statistical analysis methods that assume that patients’ outcomes are independent of each other.4–7 In the analysis phase, ignoring clustering of the subjects within trauma centers can lead to artificially narrow CIs and small p values.4–7
The second challenge is the high proportion of missing values for some variables such as diagnostic and therapeutic procedures and blood alcohol concentration (BAC) in emergency departments (EDs). The large number of subjects in the NTDB gives researchers a large sample size, even after excluding the subjects with missing values—that is, to carry out a “complete case analysis”.89 Complete case analysis might not affect the results of a study that is based on a variable with less than 5% random missing values. However, it might threaten the validity of a study that uses a variable such as BAC in ED patients, with more than 60% missing in most trauma registries in the NTDB. Complete case analysis is based on the assumption that the data are missing completely randomly.8–11 In other words, the missing pattern does not correlate with any other variable, measured or unmeasured, in the study. Our preliminary analysis of the NTDB showed that missing values for reported BAC in EDs depends on variables such as race/ethnicity, sex, type of injury, etc (data not presented). Therefore, the missing pattern does not seem to be missing completely at random. As a result, removal from analysis of subjects with missing BAC in the ED might introduce bias.
We illustrate the potential influences of the above methodological issues using a study that evaluated the association between BAC in the ED and patients’ resource utilization and ED and hospital disposition, using the NTDB data.
METHODS
We used the fifth version of the NTDB which was released in November 2005. It includes data on trauma patients hospitalized between 1 January 2000 and 31 December 2004 in 567 trauma care facilities from 43 states and US territories.1 Level I (123) and level II (123) trauma centers comprised 43% of these facilities. All hospitalized trauma patients with an International Classification of Disease, 9th revision, Clinical Modification (ICD-9-CM) codes of 800–959 were eligible to be included, excluding those with late effects of trauma (codes of 905–909) such as complications of surgery.1 From this database, we focused on 15–65-year-old patients who were hospitalized in a level I or II trauma center. Therefore, children, older adults, and patients hospitalized in level III or IV trauma centers were excluded from this analysis.
For illustration purposes, we considered two groups of outcomes. The first comprised some of the most commonly used diagnostic and therapeutic resources for evaluating and treating trauma patients.1213 These included x-ray examination of the cervical spine, CT scan of the head, thorax, and abdomen, abdominal ultrasonography, and endotracheal intubation.1213 The second included disposition status at the time of discharge.
Of 246 level I and II trauma centers in the NTDB, 109 did not report any CT scan data, 42 had reported at least one CT scan of the head, thorax, or abdomen, and only 72 had reported all three diagnostic procedures for at least one patient each. We used only the data from these 72 trauma centers.
On average, 70% of the patients in these 72 trauma centers had missing values for BAC. There are several methods that can be adopted to deal with missing data. We used multiple imputation (MI),8–11 which replaces missing data with m>1 simulated values,9 assuming that missing pattern means “missing at random”. MI is not the only way to deal with missing data and it is not necessarily the best approach either.9 The algorithm of managing missing data in MI is very similar to the expectation-maximization algorithm and other computational methods for calculating maximum-likelihood estimates on the basis of observed data alone.14 However, the major difference is that MI calculates the maximum likelihood on the basis of Monte-Carlo simulation.89 Because of the wide availability of the procedure in different statistical packages, the use of MI to deal with missing data has become more and more common.15 We used Stata V9.2 to impute the missing values for patients’ BAC in the ED.
Because of potential differences in factors such as patient and injury-related characteristics, urban/rural, per capita income, and strictness of alcohol laws and their enforcement in different trauma centers, we imputed the missing values for BAC for each trauma center separately.16 In order to perform center-specific MI, we had to exclude trauma centers that had not reported any values for BAC. As a result, 17 centers with 100% missing values for BAC were excluded from the rest of the analysis.
We considered the following criteria for inclusion of variables in the MI model17–19: (a) a priori selected variables that were planned to be included in the final statistical models; (b) variables that influenced the missing status of BAC; (c) variables that correlated with the presence or absence of BAC. van Burren et al18 have defined a correlation of 0.15 as a threshold for including variables in the model. As a result, our final MI model included the following variables: age, sex, race/ethnicity (white, black, Hispanic), type of insurance (self-pay, Medicaid, other types of insurance), type (blunt versus penetrating) and mechanism of injury, injury severity score,20 Glasgow Coma Scale 21 and its components, ED and hospital disposition, length of intensive care unit and hospital stay, resource utilization, therapeutic interventions provided in pre-hospital and hospital settings, and patients’ transfer status to other centers.
We conducted 10 imputations for each of the remaining 55 trauma centers. Each imputed dataset was analyzed separately, and the results were averaged. We used the following formula (also known as Rubin’s formula) to calculate the standard error for the calculated mean odds ratios (ORs)10:
in which M is the total number of imputations (10 in our study), bk is the measured outcome for the kth imputed database, Sk is the standard error of the measured value in the kth imputed database, and b̅ is the mean of the k calculated b.
To demonstrate the potential influence of ignoring missing data and clustering of patients within trauma centers on the results of a study, we adopted four different approaches. In approach A, we conducted a complete case analysis and excluded subjects with missing data for BAC from the analysis. Method B also excluded subjects with missing BAC, but took into consideration the clustering of observations within trauma centers. Methods C and D used the imputed data for BAC. However, the issue of clustering was also taken into consideration in model D.
As mentioned, approaches A and C ignored the clustering design of the NTDB and assumed that all the patients belong to the same general population. Therefore, standard logistic regression models were used to evaluate the association between subjects’ BAC in the ED and patients’ resource utilization and ED and hospital disposition. However, in approaches B and D, a statistical method provided adjusted estimates by taking into account clustering.
Localio et al5 have nicely summarized different approaches that could be adopted for clustered data analysis. Conditional or stratified models are commonly used in studies in which patients are randomized to a treatment or control group within each site. Researchers calculate the effect of intervention for each stratum or site and then summarize the effects across the centers. These models can be further divided into fixed-effect and random-effect models. In fixed-effect models, each site is assumed to be representative only of itself, and therefore results are not generalizable to other sites. In random-effect models, each site in the study is representative of other similar sites. Mantel–Haenszel analysis, conditional logistic regression, and generalized linear and latent mixed models22 are among the methods in this category.
Unconditional (or marginal) methods, on the other hand, first estimate the association between exposure and the outcome of interest, without taking into consideration site effects, and then adjust the CIs for the effects of clustering of patients within sites.5 This is the method of choice, when researchers are not interested in any single site and they want to evaluate the overall effect of the exposure on the outcome. Generalized estimating equations (GEEs)2324 can be categorized in this group.
We used GEEs in approaches B and D to capture the marginal effect of patients’ BAC on resource utilization and ED and hospital disposition. The details of the pros and cons of this method are beyond the scope of this paper but can be found elsewhere.24–28 Briefly, GEE estimate regression coefficients and standard errors with sampling distributions that are asymptotically normal can be applied to test main effects and interactions, and can be used to evaluate categorical or continuous independent variables.29 GEE estimates for binary outcomes are the same as those produced by logistic regression when the dependent variable is binomial and no correlation between patients’ outcomes is assumed.
We would like to remind our readers that, although GEEs allowed us to compare the odds of utilization of a service (or patient’s disposition) between patients with and without high BAC in the ED, an OR might not be a good approximation of the risk ratio for outcomes that are common (a prevalence of 10% or more).3031 However, as the OR is probably the most commonly used measure of excess risk in studies that have used the NTDB data, it is used in the current analyses.
We used χ2 analysis to compare the distribution of the categorical variables among different trauma centers.32 Analysis of variance was used to compare the continuous outcomes among different centers.32
RESULTS
A total of 128 674 patients were hospitalized in the 55 level I and II trauma centers included in this study. The number of patients per center varied from 531 to 6844. Table 1 summarizes relevant characteristics of the patients, reflecting the substantial heterogeneity among different trauma centers. A test of homogeneity by center produced p<0.001.
Table 2 summarizes the mean reported utilization rate of the healthcare resources. As presented, there was a considerable difference in the reporting of utilized services among different centers (p<0.001 for all comparisons).
Table 3 shows the observed reported range of ED and hospital dispositions; it also compares the average length of hospital stay reported from different trauma facilities. A test of homogeneity by center produced p<0.001.
Each line of table 4 represents the results of a separate model that evaluated the association between BAC and one of the outcomes of interest, adjusted for age, sex, race/ethnicity (white, black, Hispanic), type of insurance (self-pay, Medicaid, other types of insurance), type (blunt versus penetrating) and mechanism of injury, and injury severity score. Columns A and B in table 4 are the results of a complete case analysis approach—that is, subjects with missing BAC were excluded from the analysis. Although the point estimates are the same in these two columns, adjustment for the clustering effect increased the width of all CIs (column B). As a result, some of the statistically significant associations became statistically non-significant as observed in the ORs that include 1.
The same format is followed in columns C and D, but the imputed data are used for analysis. In column D, the CIs were also adjusted for clustering of the subjects within trauma centers. Whereas adjustment for the clustering effect only increased the CIs, the use of the MI method changed the point estimates (table 4), substantially in some instances. For example, the OR for head CT scan and abdominal CT scan fell from 1.84 (95% CI 1.49 to 2.28) and 1.33 (95% CI 1.28 to 1.39) in approach B to 1.26 (95% CI 0.98 to 1.64) and 1.00 (95% CI 0.78 to 1.27) in approach D. Interestingly, all ORs in approach D moved toward 1, except the OR for diagnostic peritoneal lavage (OR 1.4, 95% CI 1.34 to 1.46), which moved away from 1 (OR 1.46, 95% CI 0.98 to 2.19), although the CI still included unity.
We adopted the same analytical approach to evaluating the association between BAC and final disposition (table 5). Similar to the analysis of resource utilization, adjustment for CIs in approach B (compared with approach A) increased the width of the CI (table 5). As a result, the CIs for hospital admission and transfer to other centers included unity. Therefore, we were not able to detect any statistically significant association between patients’ BAC in the ED and these outcomes, if in reality such associations existed.
The use of imputed data (approaches C and D) changed the point estimates, substantially for some of the outcomes. For example, the OR for ICU admission fell from 1.40 (95% CI 1.20 to 1.63) in approach B to 1.17 (95% CI 0.97 to 1.41) in approach D (table 5). At the same time, the OR for ED death increased from 0.33 (95% CI 0.27 to 0.40) in approach B to 0.65 (95% CI 0.43 to 0.92) in approach D.
What is already known on this topic
-
Ignoring the clustering issue in data analysis may lead to artificially narrow CIs.
-
Excluding subjects with missing values for a particular exposure of interest may introduce bias if missing status is not completely at random.
What this study adds
-
Ignoring clustering of patients within trauma centers in studies that use the US National Trauma Data Bank (NTDB) for alcohol research led to imprecise estimates and inappropriate conclusions.
-
Owing to the high proportion of non-randomly missing values for patients’ blood alcohol concentration in the NTDB, complete case analysis led to biased estimates.
DISCUSSION
Although this study addresses some of the unique features and challenges of the NTDB pertaining to alcohol, our findings are also applicable to other studies that use this databank for evaluation of other aspects of trauma care. As revealed here, the significant statistical association between an exposure and an outcome might disappear after proper adjustment for the clustering design. For example, we were not able to detect any statistically significant association between BAC and thorax CT scan, diagnostic peritoneal lavage, hospital admission from ED, and transfer to other centers after adjustment for the clustering design. Therefore, had the results of the analyses based on the first model (complete case analysis, ignoring the clustering issue) been presented, inappropriate conclusions about statistical significance between the exposure and the outcomes of interest would have been reached. On the basis of the results of this study, these significant associations either do not exist or cannot be demonstrated using the NTDB data.
As mentioned, whereas ignoring the clustering effect mainly influences the CIs, selection bias due to differential misclassification of exposure or outcome could bias point estimates toward or away from unity.3031 The prominent change in the OR of the use of head CT scan comparing approaches B (1.84) and D (1.26) is a good example of the level of bias that can be introduced into a study when the missing pattern is not completely at random.
Studies that have used NTDB have often assumed that the missing status of a particular variable is completely at random. In other words, the missing status does not depend on other measured or unmeasured variables.8–10 Because of the lack of access to unmeasured data, it is virtually impossible to prove this assumption. However, the assumption can be rejected by showing that there is at least one variable in the database that correlates with missing status. Our preliminary analysis of the NTDB data showed that missing status for BAC was significantly associated with several variables, including sex and race/ethnicity (data are not presented here). Thus, missing pattern was not completely at random, which can justify the observed biases in point estimates resulting from complete case analyses.
In addition, a comparison of columns B and D shows that, for example, whereas ORs for x-ray examination of the cervical spine and thoracic CT scan moved away from 1 (strengthening the observed association), the ORs for other utilized services moved towards 1 (weakening the observed association). This observation is another indication that complete case analysis in our study led to the differential misclassification of exposure and/or outcome—that is, the direction of the bias cannot necessarily be known or taken into account when interpreting findings.
CONCLUSIONS
Researchers that use the NTDB for evaluation of different aspects of injury epidemiology and trauma care should be aware of the special features and methodological challenges that may affect the robustness of their studies. In spite of the large number of subjects in the NTDB, the substantial heterogeneity in reporting of data may restrict researchers’ ability to detect small but significant associations between exposure(s) and outcome(s) of interest.
Acknowledgments
We acknowledge Dr Thomas Koepsell and the reviewers for their invaluable comments on an earlier draft of this paper.
REFERENCES
Footnotes
Competing interests: None.