Information about pediatric injury in Canada can be gleaned from a number of sources. These include population surveys (for example, the National Longitudinal Survey of Children and Youth and the National Population Health Survey), medical record reviews, hospital and trauma registries (including administrative databases), and injury surveillance systems.¹ With regards to the latter, the Canadian Hospitals Injury Research and Prevention Program (CHIRPP) is a primary source of information on patterns of childhood injury in Canada.

CHIRPP is a computerized emergency room based injury surveillance system that operates in 10 pediatric and five general hospitals across the country. It gathers important data (such as mechanism of injury, nature of injury, and body part) relating to children’s visits to hospital emergency departments for injury. Pickett and collaborators have provided an overview of the system’s strengths and weaknesses.² Its strengths are: (1) information on the circumstances in which injuries occur that cannot be obtained elsewhere, (2) its high rate of parental compliance, and (3) its data are invaluable for the development of appropriate preventive interventions. Weaknesses of CHIRPP include its need for active cooperation of doctors and emergency staff as well as technical support (adding to the expense of running such a system), and its representativeness, as the rate of children’s use of emergency departments after sustaining an injury could vary among communities.^3–5 Finally, it is somewhat limited in that it does not provide data relating to injuries for which care was sought outside the emergency department (such as pediatrician’s or physician’s office).

Physician billing claims represent another data source for injury research. Taylor et al, used the billing records (fee-for-service billing data) of a tertiary care pediatric emergency department in Halifax, Nova Scotia, Canada to predict the number of future emergency department visits for trauma compared with those for respiratory infection.⁶ They concluded that billing data, that includes diagnostic codes based on the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM),⁷ provide a good measure of future trauma occurrence risk among children. In Quebec, the fee-for-service database of the Quebec Health Insurance Board, or the Régie de l’assurance maladie du Québec (RAMQ), was shown to provide a valid source of information for outcome assessment of injuries in the elderly.⁸ The sensitivity of the claims data for the measurement of any injury was found to be 81.3%, but the sensitivities for specific injuries varied widely from 14.3% to 97.2%. The accuracy of this type of data was also found to be high in cases of cardiovascular prognosis,⁹ hip fracture,^10–12 and stroke.¹³ Although these data are primarily used for reimbursement and accounting purposes, physician billings claims have been suggested as an appropriate data source for quality of care assessment,^14–16 pharmacoepidemiologic research,¹⁷ technology assessment,¹⁸ evaluation of medical care appropriateness,¹⁹ and cost utilization studies.²⁰ The advantages of using claims data are: (1) large samples of geographically dispersed patients, (2) longitudinal records, (3) convenience and low cost (the data are already collected and available), and (4) defined sampling frames.^14,21–25 Limitations of these types of administrative data include inaccurate coding that is thought to be due to patient complexity, vague context of ICD-9-CM code definitions, institution-specific variations in coding practices, and financial incentives to record certain diagnoses.^9,26,27 To our knowledge, the use of physician claims data has never been formally validated for pediatric injury, and in particular for head injury (HI). Moreover, physician claims data have never been validated against an injury surveillance system.

The purpose of this study was to compare data gathered using the CHIRPP surveillance system with that from the physician claims files from the RAMQ (Quebec Provincial Health Insurance Board) for the same group of children who visited an emergency department for the same index injury. Specifically, we sought to determine (1) the agreement between childhood injury diagnoses (HI and orthopedic injury), and (2) the sensitivity and specificity of diagnostic and procedure codes in billing claims for identifying HI and orthopedic injury among children.

METHODS

Data from the CHIRPP database were compared with those from the physician claims database of the RAMQ. CHIRPP data were collected at the emergency departments of the two pediatric hospitals in Montreal from December 2000 to October 2001 (Montreal Children’s Hospital and Hôpital Saint-Justine). The CHIRPP database contains information about the circumstances of the injury that is completed by the patient or an accompanying adult on a one page self administered CHIRPP questionnaire. Also on the back of the same form, the physicians record the following clinical information: diagnostic codes, injured body part, and treatment provided. The child’s personal health insurance number (a 10 digit number that provides a unique identification number for each beneficiary of the Quebec health insurance plan), date of birth, sex, postal code, and date of visit are also recorded on the CHIRPP sheet.

The Quebec Health Insurance Board or RAMQ—the government agency responsible for administering the provincial health insurance plan—uses a computerized billing service to record service use and to reimburse service providers. This database contains the physician’s identification number, the patient’s provincial health insurance number, the date and location of visit, type of consultation, as well as the ICD-9-CM diagnostic codes for the visit, procedure codes for services provided, and the cost of these services.

Procedure

The records of 3145 children aged 1–18 years who sought care for an injury during the study period were identified and extracted from the CHIRPP database. Children under one year of age were excluded because they have not received their permanent health insurance number and we would be unable to link them with the RAMQ database.

The physician recorded diagnostic codes in the CHIRPP database separated the children into three groups: (HI, probable HI, and orthopedic). This classification was chosen for the purposes of a larger study examining the risk of second HI among children. The HI group was defined as children with a diagnosis of HI (for example, skull fracture, minor HI, concussion, intracranial injury, and multiple injuries with associated HI). When a child sustained multiple injuries and one of the injuries included an HI, the child was classified into the HI group. The probable HI group included children with injuries to the eyes, face, and teeth or isolated facial lacerations only when accompanied with one of the following mechanisms of injury: struck forcefully against a hard surface, a fall from a height, or both. The orthopedic group consisted of children diagnosed with an orthopedic injury of the upper or lower extremity (for example, fracture, sprain, and dislocation).

A file containing this CHIRPP based information (diagnostic group, age, sex, postal code) was sent by registered mail to the statistical department of the RAMQ (Service des statistiques of the RAMQ). It was returned to us containing the complete registry of services paid to fee-for-service physicians who provided care for 3049 children during 12 months after the index visit for an injury. Data were unattainable for 96 children due to missing RAMQ numbers. The confidentiality of subjects was maintained using a scrambled personal insurance number. This however prevented any additional linking of the two data sources.

To enable a comparison between the two data sources with respect to injury diagnosis, the RAMQ data were grouped into the same three diagnostic categories. This was done using an algorithm (formed of ICD-9-CM diagnostic and procedure codes) developed and validated by a team of researchers (including an emergency department pediatrician) with a sample of 500 children. The HI group was defined as all children who received health services for an HI (ICD-9-CM codes 800, 801, 803, 804, 850–8549, 9083 or procedure codes specific to HI 7500–7507, 7595–7598) or either one of these codes. The probable HI group included all children who had a combination of the following diagnostic (ICD-9-CM 802, 830, 873, 910, 920, 959) and procedure codes (1320, 2113, 2505–2527, 8010–8030 8258–59, 8570). For example, a child having a visit billed with a diagnostic code for concussion (ICD-9-CM 8540) was assumed to have had an HI. A child with a diagnostic code of imprecise trauma (8290) and a procedure code indicating magnetic resonance imaging of the head (8570) was assumed to have had a probable HI. The orthopedic group consisted of all children using health services for an injury to the upper or lower extremities (ICD-9-CM 810–817, 820–829, and 831–839). A list of the pertinent diagnostic and procedure codes associated with injury are presented in table 1.

Analysis

Two analyses were performed. Firstly, we created a 3×3 table (HI, probable HI, and orthopedic) and calculated the percent overall agreement and a weighted kappa statistic between childhood injury diagnoses obtained from the two data sources. Secondly, using CHIRPP as a gold standard, we calculated sensitivity and specificity for HI and orthopedic injury by creating a 2×2 table where the HI group was combined with the probable HI group and compared to the orthopedic group. Sensitivity for HI was defined as the probability of having an HI (HI group and probable HI group) indicated in the RAMQ database given that HI was recorded as the CHIRPP diagnosis. Specificity was defined as the probability of not having an HI recorded in the RAMQ database when HI was truly absent as indicated by the CHIRPP diagnoses.

RESULTS

The descriptive characteristics of the sample are presented in table 2. There were more males (60.5%) than females with an injury, and across all groups, injuries mostly occurred at home (39%), followed by those at school inside and outside (21%) and in recreation centers (13%). Thirty percent of children with HI were sent home from the emergency department with advice only, whereas 98% with probable HI and 93% of those with orthopedic injury were treated before being sent home.

Concordance between the two data sources was found to be “substantial’ (weighted Kappa 0.66; 95% CI 0.63 to 0.69) according to the interpretation scale of Landis and Koch, and percent agreement was 81%.²⁸

There were 446 exact matches for HI, 173 for probable HI, and 1849 for orthopedic injury. Discordant cases (n = 581 children) were individually examined to determine the underlying reasons of disagreement. Reasons for disagreement varied and some were more easily accounted for than others (table 3). For example, among the discordant cases, 52% were billed with the diagnostic code “multiple unspecified wound or trauma” and 95% of these (n = 289) were classified by CHIRPP as HI or probable HI. The mean age of children among the discordant pairs (6.6 years) was significantly different (p<0.001) from that of the rest of sample (8.5 years).

The sensitivity of diagnostic and procedure codes in the RAMQ database for identifying HI and for orthopedic injury were 0.61 (95% CI 0.57 to 0.64) and 0.97 (95% CI 0.96 to 0.98), respectively. The specificity for identifying HI and for orthopedic injury were 0.97 (95% CI 0.96 to 0.98) and 0.58 (95% CI 0.56 to 0.63), respectively.

DISCUSSION

We compared childhood injury diagnoses using CHIRPP data with that from a physician claims database for the same group of children who visited an emergency department for the same index injury. Our results indicate that the concordance between the two data sources is “substantial” and that the sensitivity of claims data for identifying orthopedic injury was higher than that for identifying HI. There are several possible reasons for the less than optimal level of concordance. Firstly, we observed a high frequency of non-specific diagnostic codes in physician billings. It is interesting to note that of the 304 cases that were billed for “multiple unspecified wound or trauma”, 95% (n = 289) of these were classified by CHIRPP as HI or probable HI. Another 18% of cases (n = 107) were billed with a diagnosis unrelated to injury (for example, sinusitis, chicken pox) yet 96% of these had HI or probable HI CHIRPP based diagnoses. It would thus appear that the physician claims database underestimates the frequency of HI (and probable HI) by approximately 12% (that is, (107+289)/3049). Our results also suggest that the concordance may be associated with the child’s age because the children in the discordant pairs were significantly younger than the concordant pairs. This is probably related to a less precise diagnosis among younger children.

Other possible reasons for our results include coding errors in both databases. Clearly, there are errors when completing RAMQ reimbursement forms because 107 children among the discordant pairs, who completed a CHIRPP form for an injury, were billed with a RAMQ diagnostic code other than trauma. This phenomenon is however unlikely to be related to fee structure (that is, financial incentives). Reimbursement fees for an HI are not necessarily higher than those for an orthopedic injury but appear to vary in both cases according to the time of day of the visit, day of the week of the visit, and so on. Errors could also have occurred in the transcribing of physician notes onto RAMQ reimbursement forms. Certain recording errors of injuries using the CHIRPP system may also exist because 99 children among the discordant pairs were billed with diagnostic codes indicating HI or probable HI but classified by CHIRPP as an orthopedic injury. Perhaps these children suffered multiple injuries and for some reason the physician coded the visit as one for an HI. Physicians who write the initial diagnoses do not necessarily complete the CHIRPP form—sometimes this information is recorded by a clerk using the child’s medical file as reference. Finally, a small percentage of missing diagnostic codes in the RAMQ data may also have contributed to the less than perfect concordance.

Indeed, each database was developed for a specific purpose and has its own strengths and weaknesses. Although CHIRPP provides important detailed information about injuries treated in an emergency department, one must keep in mind that the injury surveillance system is not population based. This is in contrast to the physician claims database used in this study, which covers the full continuum from ambulatory to hospital based care, and provides information on almost all contacts with physicians in the healthcare system. Physician claims data are however limited to healthcare systems where fee-for-services payment is the predominant means of reimbursement. This is the case in Quebec and other parts of Canada, but it may differ in other countries where physicians are paid by capitation or salary. Clearly, there is no single best source for monitoring injury in a population. It would be ideal to link these datasets (injury data with administrative data) to complement the unique strengths of each type of data and provide a more complete picture of childhood injury.

To our knowledge this is the first study in Quebec that measures the accuracy of physician claims data for pediatric injury including HI. We were interested in knowing if physician claims data could be used for another purpose other than physician reimbursement. We believe this study demonstrates a new application and the potential capabilities of using diagnostic and procedures codes from physician billing claims data to study injury (including HI) among children. In particular, one could determine with relative confidence the number of children who receive medical care for an HI, where they receive this care, and the costs associated with emergency department visits for HI.

A number of limitations should be considered in the interpretation of the results. This study involved data for children in Montreal, Quebec and the results may not be generalizable to older populations or to those in other regions. Ideally, sensitivity and specificity estimates require that subjects be classified into diagnostic groups using an error-free gold standard. As discussed above, misclassification within the gold standard could have occurred.

In Quebec, where fee-for-service billing is the predominant method of remuneration, the combination of diagnostic and procedures codes in a physician claims database may be a valid method of estimating injury occurrence among children. Its use may however lead to an underestimation of the frequency of visits for HI, particularly in younger children.