Article Text
PDF
Abstract
Objective Population-based health surveys are increasingly popular sources of data on injury occurrence. Self-reported surveys can yield estimates of the total incidence of non-fatal injuries while simultaneously capturing a rich repository of contextual data that may be informative for exploring determinants of injury risk. Although survey data are rarely recognised as complete, several researchers have expressed concerns about the sensitivity and validity of self-reported injury data, questioning whether captured cases are representative of the population experience of injury, particularly among children and youth. The present study sought to compare the population incidence of paediatric injury estimated from self-reported survey responses to those documented by a complete-capture health service utilisation database among Ontario children.
Methods Injury incidence rates documented from the National Longitudinal Survey of Children and Youth and the National Population Health Survey were compared with those reported in Canada's National Ambulatory Care Reporting System for Ontario youth aged 0–19 years for fiscal year 2002/3, stratified by the child's age and geographical location of residence.
Results The two self-reported health surveys underestimated the population incidence of injury among Ontario children by at least 49% and 53%, respectively. Systematic errors exist in survey data capture such that injuries in infants and preschoolers (<4 years of age) and urban residents were most likely to be missed by the population health surveys.
Conclusion Injury incidence estimated through self-report is not representative of the population burden and experience of paediatric injury for Ontario children, and may produce biased estimates of risk when analysed as independent sources of data.
- Behaviour change
- bicycle
- burden of disease
- Canada
- child
- concussion
- descriptive epidemiology
- epidemiologic
- epidemiology
- health disparities
- helmet
- implementation/translation
- injury epidemiology
- methodology
- pedestrian
- public health
- reliability and validity
- risk/determinants
- risk factor research
- sensitivity and specificity
- standards
- surveillance
- traffic
Statistics from Altmetric.com
- Behaviour change
- bicycle
- burden of disease
- Canada
- child
- concussion
- descriptive epidemiology
- epidemiologic
- epidemiology
- health disparities
- helmet
- implementation/translation
- injury epidemiology
- methodology
- pedestrian
- public health
- reliability and validity
- risk/determinants
- risk factor research
- sensitivity and specificity
- standards
- surveillance
- traffic
Since the mid-1940s, injuries have been among the leading causes of death, disability and medical costs for infants and children across the globe.1 While the pattern and aetiology of injuries and their outcomes vary substantially across populations, there is growing awareness of the importance of epidemiological research investigating the burden of childhood injury, and the identification of determinants within local environments that lead to increases in injury risk and those most likely to be amenable to prevention and intervention.
In most countries, two sources of information are typically used to capture and monitor the incidence of paediatric injury: health service utilisation data retrieved from administrative databases such as provincial/state insurance claims or hospital discharge records, and self-reported injury occurrence obtained during population health surveys. Several sources of administrative data are routinely used to monitor and explore the epidemiology of paediatric injury in Canada, including the National Trauma Registry,2 the Canadian Discharge Abstract Database,3 the National Ambulatory Care Reporting System (NACRS)4 and the Canadian Hospital's Injury Reporting and Prevention Program.5 While trauma registries and surveillance systems such as the Canadian Hospital's Injury Reporting and Prevention Program contain a rich repository of contextual data, these sources are rarely population based and cannot be used to estimate the total incidence of injury occurring within defined populations. Furthermore, although health service utilisation data are widely available and usually collected on an ongoing basis, these sources only capture injuries in which an individual received formal medical care. Consequently, minor injuries are generally not included in the estimation of injury incidence, and many desired indicators of the child's overall health, wellbeing, family functioning and socioeconomic vulnerability are not typically captured.
Survey data have the added benefit of capturing these individual and family-level characteristics that may be more informative when understanding injury risk and prevention.6 As a result, population-based health surveys are increasingly used as routine sources of data for describing the characteristics of injured children and exploring potential causes and contributors to injury morbidity.6–11 Population health surveys can also yield estimates of the total burden of non-fatal injuries across the entire spectrum of injury severity,11 although sample sizes can be insufficient for investigating population subgroups and rare injury events.6 10–12 Two population health surveys capture data pertaining to paediatric injury occurrence in Canada: the National Longitudinal Survey of Children and Youth (NLSCY)13 and the National Population Health Survey (NPHS).14 These databases fill a longstanding gap in Canadian injury epidemiology by capturing a broader spectrum of severity (ie, injuries in which medical attention was required, but did not necessarily result in death or hospital admission); however, several researchers have expressed concerns about the sensitivity and validity of self-reported injury data regarding their ability to capture paediatric injury incidence accurately6 10–12 15 16 and whether captured cases are representative of the actual experience of injury among young Canadians.11 12 15
An in-depth understanding of the mechanisms and determinants associated with injury risk for Canada's paediatric population necessitates investigating the sensitivity, validity and biases of self-reported survey data for capturing the population burden of childhood injury. Therefore, the purpose of this study was to compare the incidence of paediatric injury estimated from two population-based health surveys (NLSCY and NPHS) with those documented by a complete-capture health service utilisation database (NACRS) among Ontario children aged 0–19 years for fiscal year 2002/3. In addition to examining the agreement of injury incidence between the two types of data, we also examined bias in the estimation of incidence and RR among population subgroups with small sample sizes.
Methods
Data sources and population of interest
The National Ambulatory Care Reporting System
The NACRS is a health service utilisation database that collects information on patient visits to emergency departments, day surgeries and ambulatory clinics in participating facilities across Canada.4 Abstract submission has been mandatory for all visits to Ontario emergency departments since fiscal year 2002/3, making this a complete capture utilisation database for Ontario residents.4 Demographic, clinical and administrative data are collected at the time of service delivery and clinical data are coded according to the ICD-10-CA/CCI classification system of disease. For this study, data on children and youth aged 0–19 years who presented to an Ontario emergency department for treatment of an injury between 1 April 2002 and 31 March 2003 were extracted from the NACRS database. Children were considered to have sustained an injury if they had one or more documented visits to an Ontario emergency department during the 1-year period with a most responsible diagnosis falling within the trauma codes starting with S and T, and a corresponding external cause of injury code (E-codes V01–X59). To describe the incidence of paediatric injury accurately, multiple visits within the 1-year time frame were counted; however, children and youth who were not residents of Ontario at the time of treatment, defined by their postal code provided on triage, were excluded. The total number of injury-related emergency department visits for Ontario children and youth captured by NACRS in 2002/3 was then divided by the total paediatric population of Ontario obtained from the 2001 Canada census to get the annual incidence rate of injury per 100 000 children.
National Longitudinal Survey of Children and Youth
The NLSCY is a national, prospective longitudinal study of Canadian children designed to measure and track changes in child health, development and wellbeing over time.17 Data collection for cycle 1 began in 1994 and the survey is conducted biennially.13 Each cycle uses a purposeful sampling strategy representative of the original longitudinal population excluding first nations people living on reserves and children residing in institutional settings.17 Data on injury incidence was obtained from cycle 5 of the NLSCY, an independent cross-sectional survey completed in 2002/3 using a representative sample of 25 439 children and youth. This cycle contained three distinct cohorts of respondents: a new cross-sectional sample of children aged 0–5 years, the original longitudinal sample of children, now between the ages of 8 and 14 years and the original sample of young adolescents, now youth aged 15–19 years.17 Children between the ages of 6 and 7 years were not captured in this cycle of the survey.17
National Population Health Survey
The NPHS is a similar prospective longitudinal study, but unlike the NLSCY, the NPHS was developed to explore demographic, social, economic, occupational and environmental determinants of health among all Canadians.14 Data collection began in 1994 with a survey of 26 430 households and is administered to one random member of each household every second year.14 The NPHS uses a comparable sampling strategy that aims to be representative of the original longitudinal population, excluding first nations people living on reserves, residents of armed forces bases and individuals residing in institutional settings.14 This study utilised data from cycle 5 of the NPHS, which was completed in 2002/3 using a representative cross-sectional sample of 12 546 families.18 Due to the longitudinal nature of the survey, children under the age of 8 years were not captured in cycle 5 of the NPHS, therefore injury incidence in this study is only calculated for adolescents and youth aged 10–19 years.
To estimate injury incidence documented from the population health surveys, respondents in the NLSCY and NPHS were asked a series of questions relating to the occurrence of injuries that required formal utilisation of the healthcare system within the 12 months before being surveyed. Children and youth who reported experiencing one or more such injuries were then prompted for further information on the number of incidents requiring medical attention, the nature of the most severe injury and the activity they were engaged in at the time of that injury's occurrence. All instances of injury requiring medical care within the past year were counted. To adjust for unequal probabilities of selection, non-response and to reflect age and gender distributions of the Canadian population at the time of the survey, incidence counts were weighted by the population sampling weights provided by Statistics Canada for each database. Annual incidence rates of self-reported injury were then calculated by taking the weighted number of injuries reported in the NLSCY and NPHS, respectively, divided by population estimates from the 2001 Canada census, per 100 000 children.
Analysis
To compare the agreement of injury incidence estimated from self-reported survey responses with those documented by the complete capture utilisation database, we took the incidence rates estimated from the NLSCY and the NPHS, respectively, and divided these by the rates documented in NACRS, and report the capture sensitivity of the population health surveys along with 95% CI. Incidence rates and rate ratios were calculated for each of the following age categories: infant and preschool, 0–4 years; school aged, 5–9 years; young adolescents, 10–14 years; and youth 15–19 years. To assess further any bias in the survey data towards small sample sizes, incidence rates were stratified by whether the child resided in an urban, mixed urban, mixed rural or rural geographical location based on the population density of their county of residence. According to Statistics Canada, counties containing a population density greater than 400 persons per square kilometre are considered urban, while all other counties are defined as rural.19 To increase the sensitivity of this analysis, rural counties were subdivided into the following three categories: counties with a density between 100 and 400 residents per square kilometre were classified as mixed urban, those with a density between 20 and 99 were considered mixed rural and counties with a density of less than 20 residents per square kilometre were defined as rural. These thresholds are consistent with those used in previous research investigating geographical variations in injury incidence across Ontario.20
Finally, we examined the validity of the self-reported data for exploring determinants of risk by calculating the age-adjusted RR of injury by geographical location of residence using Poisson regression, and comparing the estimates of risk across the three databases. Ethics approval was obtained from York University, and Statistics Canada data publication guidelines were followed throughout the analyses. All data manipulation and analyses were performed using SAS 9.1.
Results
Between 1 April 2002 and 31 March 2003, 376 358 children between the ages of 0 and 19 years (excluding ages 6 and 7 years) were seen in an Ontario emergency department for the treatment of an injury according to the complete capture utilisation database (table 1). During the same year, estimates from the NLSCY suggested approximately 200 450 Ontario children sought medical treatment for an injury outside of the home (again excluding ages 6 and 7 years). As such, the utilisation database captured an 88% higher incidence of paediatric injury compared with self-reported estimates, despite covering a narrower range of the non-fatal injury spectrum. Two hundred and fifty-two thousand two hundred and forty-eight of the events documented in NACRS occurred among adolescents and youth aged 10–19 years. Within the same demographic group, 112 000 injuries requiring medical treatment were captured in the NPHS. Calculating overall incidence agreement between NACRS and the NPHS, NACRS captured over twice the number of youth injuries compared with self-report.
Injury incidence documented in the NACRS, the NLSCY and the NPHS according to select characteristics: Ontario, Canada: 2002/3
Across the province, the age-adjusted injury incidence in Ontario for 2002/3 was 13 666 injuries per 100 000 children determined from the NACRS, 6677 injuries per 100 000 children from the NLSCY and 7188 injuries per 100 000 children according to the NHPS, resulting in overall sensitivities of 49% (95% CI 47% to 50%) and 53% (95% CI 51% to 54%), respectively (table 2). Self-reported injury incidence was underestimated in all paediatric age groups and across all geographical locations, although the NLSCY did capture more injuries among youth aged 15–19 years, surpassing the complete capture utilisation database in all but urban areas. The sensitivity of the NLSCY was lowest when estimating injury incidence among infants and preschoolers aged 0–4 years (0.09 to 0.22) and for children residing in urban counties (0.09 to 0.33).
Annual incidence rates of paediatric injury per 100 000 children and capture sensitivities of the NLSCY and the NPHS according to select demographic characteristics: Ontario, Canada: 2002/3
Finally, we calculated the age-adjusted RR of injury in each of the four geographical areas by data source (table 3). According to both the NACRS and the NLSCY, the RR of injury increased with a decline in population density such that injury risk was statistically higher for children living in mixed urban, mixed rural and rural counties compared with urban residents.15 While trend estimates from the NLSCY are similar to those reported in other studies,12 15 the magnitudes of risk appear overinflated relative to previously published estimates (RR ranging from 4.37 to 9.80)12 15 and are probably driven by the undercapture of true incidence among urban children. In contrast, although estimates of geographical risk appear consistent for children residing in mixed urban (RR 1.12; CI 1.19 to 1.22) and mixed rural counties (RR 1.47; CI 1.44 to 1.49) according to the NPHS, the estimates for rural children deviate from this rising pattern of risk, falling well below that expected among rural residents (RR 1.19; CI 1.17 to 1.22).
Age-adjusted RR of injury by geographical area calculated from the NACRS, the NLSCY and the NPHS: Ontario, Canada: 2002/3
Discussion
In summary, the findings from this study indicate that self-reported injury data obtained from population-based health surveys are not sensitive or valid sources of information for describing and monitoring the incidence of paediatric injury. As such they can produce biased estimates of injury risk when analysed as independent sources of data, highlighted by the comparison of RR according to geographical location. The two health surveys examined in this investigation underestimated the total burden of injury requiring medical intervention by at least 50% among Ontario children in 2002/3, despite their ability to capture a larger number of events that required a formal physician visit but did not necessarily result in a trip to the emergency department. More importantly, differences between sensitivities suggest that systematic errors in sampling and reporting probably exist for survey data. In particular, injuries in children under the age of 4 years and among urban residents were more likely to be missed by the population health surveys. While both sources fall short in their ability to capture the population burden of paediatric injury, the sensitivity of the NPHS demonstrated greater stability and consistency across the subgroups investigated.
This study provides important information regarding the quality of data currently used to capture and monitor the population burden of paediatric injury in Canada. To our knowledge, this is the first study to compare data quality in multiple population-based data sources for use in determining incidence rates and injury risk within the same study. This approach enabled us to evaluate the sensitivity of self-reported data by comparing captured incidence rates with those reported in a well validated health service utilisation database. Furthermore, this approach allowed us to examine biases in capture among demographic subgroups, and to assess the overall validity of the datasets by comparing geographical variations in incidence rates across the three sources. Despite these contributions, several limitations should be noted. Due to sampling limitations of the NLSCY, children aged 6 and 7 years were not captured and were therefore excluded from all analyses. With this in mind, true injury rates reported for children aged 5–9 years in both the NACRS and the NLSCY would differ from those reported here and would also adjust the overall provincial average. There is no reason to suspect that the sensitivity of the NLSCY would change had these children been included. In addition, incidence rates and RR were determined based on the child's geographical county of residence as opposed to the location of injury occurrence. While there is no way to assess the number of cases in which children were injured outside of the geographical area in which they reside, misclassification is likely to be minimal given that previous research has shown that children are most likely to be injured close to home.21 This analysis aimed to assess the sensitivity and validity of injury data from population-based surveys by comparing incidence with a complete capture database under the assumption that incidence should at minimum exceed that of the NACRS given its boundary on injuries that required presentation in a hospital emergency department. Whether or not the NACRS can and should be considered as a reference standard for injury data in Canada remains open to debate and an important area for further work. Finally, the conclusions and recommendations put forth in this paper stem from an analysis in which self-reported survey responses are treated as independent sources of information on injury occurrence. While it is clear from these findings that self-reported injury data are not reliable for making inferences at the population level, linking survey responses with utilisation data might actually improve overall methodology, providing a more comprehensive source of information than either used individually.
Although survey data are rarely recognised as complete, it was thought that self-reported responses obtained from Canada's population-based health surveys provided reasonable estimates of injury incidence at the population level.11 However, results from this study show otherwise, highlighting that many injuries occurring within the paediatric population are not routinely captured in national surveys, particularly among our youngest children and for residents of rural areas. Small sample sizes can distort the numerator of both incidence rates and risk ratios, even after the application of sampling weights and bootstrapping. Retrospective recall has also been shown to result in significant underestimation of non-fatal injury rates. Recall bias is a known concern during early childhood when survey responses are obtained by proxy from individuals who may not be aware of all instances when the child received medical care.6 9 22 23 Rates of decline are also influenced by injury severity and recall period, with incidence rates demonstrating greater stability for severe injuries (those resulting in death or hospitalisation)16 22 23 and recent events (<3 months).16 22 23 These methodological issues are probably echoed in our analysis of the survey data, which uses a lengthy recall period and includes non-verbal children. These systematic errors can decrease the validity of results drawn solely from survey data by underestimating the true burden of paediatric injury, decreasing statistical power, and miss-estimating the magnitude of risk when investigating injury determinants, as demonstrated above.
In conclusion, injury incidence estimated through self-report is not representative of the population burden and experience of paediatric injury for Ontario children and may produce biased estimates of risk when analysed as independent sources of data. While survey data may be useful for exploring determinants and risk among a defined group of children, the use of these data for estimating total injury incidence and for exploring risk factors at the population level should be conducted with caution.
What is already known on this subject
Population-based health surveys provide important public health information on injury incidence and determinants of risk among children, particularly for minor injuries that do not require formal utilisation of the healthcare system.
Little is known about the accuracy and representativeness of self-reported injury data.
What this study adds
Injury incidence estimated from self-reported health surveys is not representative of the population burden and experience of paediatric injury in Canada and may produce biased estimates of risk.
Survey data should not be used as the sole source of information when analysing injury incidence or for exploring risk factors at the population level.
References
Footnotes
Data for this study were provided by the Canadian Institute for Health Information under the Graduate Student Data Access Program and by the Social Sciences and Humanities Research Council. The findings, opinions and conclusions are those of the authors and do not necessarily reflect the views of either organisation.
Funding HLW is supported by a student-mentorship grant from the Ontario Neurotrauma Foundation [no 2005-PREV-MS-354].
Competing interests None.
Patient consent Secondary analysis of administrative databases. Consent is granted under the collection of the original databases.
Ethics approval This study was conducted with the approval of the York University Research Ethics Board.
Provenance and Peer review Not commissioned; externally peer reviewed.