Designation as a World Health Organization (WHO) Safe Community (SC) is based on local capacity to meet six criteria. Criterion 4 states that communities must have: “Programmes that document the frequency and causes of injuries” (http://www.phs.ki.se/csp/index_en.htm). This is typically interpreted as information that pertains directly to their community.

The reasons for doing so have been summarised by Nilsen et al1:

“Community-based injury prevention programmes need local IS [injury surveillance] to identify and characterize unique community injury problems, to develop tailored prevention strategies and to evaluate the effectiveness of local programme interventions”. In addition: “Local data can play an important role in motivating local action by increasing the community feeling of ownership and accountability for the mitigation of the injury problem” (p36).

It is important to note the evaluation need mentioned by Nilsen, since criterion 5 for designation as an SC requires: “Evaluation measures to assess programmes, processes and effects of changes” (http://www.phs.ki.se/csp/index_en.htm).

The review of Nilsen et al1 of 25 WHO SCs in Scandinavian and 16 Canadian Safe Community Foundation programmes reported that many of these programmes experienced significant difficulties accessing local injury data and few utilise these data effectively. In our evaluation of two small SCs, we noted similar difficulties.2 Nilsen et al1 recommend that, given the limited resources of most SCs, the situation be addressed by a greatly expanded supportive role of the coordinating or affiliate support centres of the two networks. They suggest “…the local programmes or the centres could collect IS data with the centres supporting analysis and interpretation with involvement of collaborating injury prevention researchers” (p41).

In this commentary, we demonstrate that expectations of SCs in terms of local surveillance systems are unrealistic. Our commentary is structured as follows:

• Characterising problems and tailoring interventions

• Evaluation

• Local data motivate local action

• What should small SCs do in terms of surveillance?

For this demonstration, we examine the issues in the context of the six WHO-designated SCs in New Zealand (NZ), namely: Waitakere City, Waimakariri District, New Plymouth District, Wellington City, Whangarei District and North Shore City.

For reasons that will become evident later, we focus on important injuries, which we have defined as those that are fatal, high threat to life, high threat of disability or high cost. As will be illustrated, shortcomings in data sources have meant that the focus of our illustrations has had to be on fatal, hospitalised (inpatient) injury and high threat to life (HTL) injuries.

We accessed injury data for the populations resident within the SCs from NZ’s national Mortality Collection (NMC) and from the National Minimum Dataset (NMDS). The NMC records all deaths in NZ. The WHO International Classification of Disease external cause codes are assigned to all injury deaths. To determine injury mortality, we selected all deaths with an external cause code in the range S00–T78.The NMDS has records for all publicly funded hospital inpatient visits for NZ. The inpatient treatment of the vast majority of the injuries in the acute phase is funded publicly. A detailed description of this database is provided elsewhere.3 To determine the incidence of injury resulting in inpatient hospitalisations, cases had to have a primary diagnosis of injury in the ICD-10 range S00–T78 and an external cause code in the range V01–Y36. Usual residence had to be within the territorial local authority boundaries applicable to the SC of interest. Readmissions, deaths and “0 days stay” cases were excluded.

To determine the incidence of fatal and HTL injury, the methods developed for injury outcome indicators for the New Zealand Injury Prevention Strategy (NZIPS) were used. For HTL injuries, the selection criteria outlined above for hospitalised (inpatient) cases were used with an additional screening so that only cases that had an estimated 6% or more probability of death were included. For fatalities, the approach was to select cases with external cause codes in the range V01–Y36. Details of the methods are available elsewhere.45

CHARACTERISING PROBLEMS AND TAILORING INTERVENTIONS

Table 1 shows the population for the six NZ SCs together with the frequency of various injury outcomes. In communities with a population such as these, the number of injury deaths is too small to undertake any meaningful analysis. For example, in Waimakariri in 2004 there were 18 injury deaths from all causes. Obviously, deaths should not be the sole focus of SCs as there are many significant non-fatal injuries worthy of prevention efforts.

In this context, a common strategy is to consider injury that results in hospital inpatient treatment. As table 1 shows for all injury, there are significant numbers of these events in all six communities.

The major difficulty, however, is that, in the absence of an explicit theoretical definition of injuries in the community of interest, it is unclear what these data represent.6 For example, the inpatient data system includes injuries that range from those that pose a very minor threat to life to those that are virtually unsurvivable. Whereas we can be reasonably confident that most HTL injuries are captured by a hospital inpatient system, the same cannot be said of injuries that represent a minor threat to life. Thus the hospital inpatient system captures most HTL injuries but an unknown portion of minor threat to life injuries. There may be major biases with the latter in terms of sociodemographic factors. For example, the distance a person lives from the hospital and other social circumstances can determine health-service-seeking behaviour and whether a person will be admitted for treatment. Moreover, the provision of health services can vary by place and time, and this influences the likelihood of admission for minor injuries.7 One solution to this is to focus on HTL injuries because we can be far more confident that all such cases will be admitted, irrespective of these extraneous factors.

Table 1 shows the number of HTL injuries for the five NZIPS priority areas and all injury. It is obvious that there are few HTL injuries in most communities. This presents statistical power problems in terms of determining whether an observed trend reflects a true effect.

HTL injuries, however, provide a far from optimal basis from which to characterise an injury problem in a community. This is because there are many injuries that do not represent HTL but may be important because they result in significant disability—for example, an eye injury that ultimately results in blindness. In NZ, and to the best of our knowledge elsewhere, there currently is no way of routinely identifying disabling injuries from hospital inpatient databases. Moreover, attempts to do so would result in a biased distribution of disabling injury as it has been shown that there is a significant proportion of injuries that do not result in inpatient treatment in the acute phase but result in serious disability.6

NZ is, in theory, advantaged in this respect, in having a national no-fault injury compensation scheme administered by the Accident Compensation Corporation (ACC) (http://www.acc.co.nz/index.htm). Under this scheme, persons who are injured have a large portion of their acute immediate medical treatment (eg, visit to general practitioner) paid for. Those who have ongoing disability are entitled to a range of compensation benefits (eg, home help, wages for time off work). This class of claims is referred to as entitlement claims, and the class for which no such claims are made are referred to as medical-fees-only claims. Most injury claims are for medical fees only. Although entitlement claims are in the minority, the ongoing cost associated with them makes them the major driver of the ACC scheme costs.8

There are two major barriers to SCs accessing information on entitlement claims for the purpose of addressing high threat of disability injury. First and foremost, ACC does not have a database that records the nature and severity of disability arising from injury. Although cost of injury could be used as proxy for disability, there are major biases associated with this approach. For example, a major component of entitlement claims costs is the wages compensation while the injured person is off work. Consequently, selecting high-cost entitlement cases is biased against those who are not earning wages, such as children, older people and homemakers. Secondly, the coding frames used for classifying the circumstances of injury do not allow the identification of major classes of injury event. For example, motor vehicle traffic crashes (MVTCs), assault, self-harm, drownings, work-related injuries and falls have been identified as major causes of injury fatalities or serious non-fatal injury and are defined as priorities for the NZIPS.9 Only work-related injury and MVTCs can be identified reliably.5 Even in these contexts there are questions as to the reliability of the data.10

EVALUATION

The influence of minor injury, defined in terms of threat to life, is evident when trends in injury incidence rates are examined to seek insight into the effectiveness of community interventions. Using hospital inpatients as an example, the trend line will be driven by low threat to life injury and that trend may be significantly different from that for HTL injury.4 This is because there are substantially more low threat to life injuries than HTL injuries. To the degree that admission to hospital for the treatment of low threat to life injury is more likely to be influenced by access and service delivery factors,7 this could result in the conclusion, for example, that the programme had been successful when the reverse had been the case. The importance of this is illustrated by the evaluation by Nilsen et al11 of several community-based injury prevention initiatives in terms of hospital discharge rates. That evaluation concluded that only five of the 14 SC municipalities “outperformed” their respective comparison municipalities. The authors raise several methodological issues to consider in interpreting this finding, one of which was “Another potential bias is changes in admission policies, therapeutic technologies, and diagnostic coding practices over time (Ekman et al 2005). Such changes may indeed have been effected during the 1987–2002 period and it is likely that there were differences in the nationwide implementation.” p272

These biases could be regarded as working at random—that is, on the basis of a “sample” of 14 matched pairs, for some matched pairs of SC and comparison areas, the bias will favour the SC, and for others it will favour the control. If one assumes that this is the case, the authors’ conclusion, that the majority of the SCs did not perform any better than their comparison group, is not unreasonable. However, evaluation is typically occurring in the context of one SC, and as such these potential biases are sufficient to raise serious questions about the validity of conclusions on the success or otherwise of an SC.

The evaluation of Neilsen et al11 highlights another of our concerns, namely assessing outcomes in terms of trends in all (causes and intents) injury. For a variety of reasons, including limited resources and the practical realities of implementing known effective strategies, SCs typically focus on a specific range of injury issues. Some may argue that, if an SC is mobilised into preventing, say, falls from playground equipment, alcohol-related crashes and dog bites, there will be flow-on effects into other areas and thus it is appropriate to examine all injury. We have concerns about making such assumptions. For example, in developed countries, we have been very successful at lowering fatal MVTCs. This contrasts dramatically with our efforts to reduce suicide. There may well be flow-on effects, but they are unlikely to be at the same level as the reductions observed in the specific injury issues that an SC is targeting.

The inappropriateness of using the all-injury approach is illustrated by the following simplified hypothetical example. Let us assume:

• an SC has 5000 important injuries a year;

• 1000 of these injuries are from MVTCs and a further 1000 are sports injuries;

• the SC focuses its prevention in these two areas;

• its objective is a 5% reduction in both areas in the first year;

• it is 100% successful in meeting its objective, ie, a reduction of 50 injuries for each area.

Thus all things being equal, after 1 year, the 2000 injuries from MVTCs and sport would be reduced by 100 to 1900. The overall burden of 5000 injuries would, however, only be reduced to 4900, ie, a 2% reduction. Let us also assume, however, that there was a flow-on effect to injuries other than from sport and MVTCs. If the flow-on effect was 20% as effective as that in the primary areas of focus, a 1% reduction would occur (1% of the other 3000 injuries), ie, 30 injuries prevented. The nett effect of the programme would therefore be 130 prevented injuries: a 2.6% reduction in all important injuries.

It is not uncommon to hear the argument that, if we successfully reduce minor injury, we can assume a reduction in less common but more serious injury. There are two problems with this argument in the context of assessing the performance of SCs using administrative databases. Firstly, it is difficult to demonstrate because most administrative databases do not capture minor injury consistently over time, as it is these injuries that are most subject to extraneous influences over time. For example, our local hospital management has made several attempts in recent years to divert minor injuries and illness from the emergency department to local general practitioners. Secondly, even assuming that the collective of many individuals and organisations in NZ has been successful in reducing minor injury, the evidence from NZIPS injury outcome indicators suggests that there has been limited, or no, impact on HTL injury. For several NZIPS priority areas, the incidence rate of HTL injury has remained relatively stable in recent years.12

LOCAL DATA MOTIVATE LOCAL ACTION

Although it has been argued that local data are a motivator for local action, our concern remains one of the importance of the issue being identified. From a national perspective, and using NZ as an example, the national priorities are those of the NZIPS. As we have illustrated (table 1), in most SCs, these outcomes will be very rare. Consequently, it makes no sense to hold each community accountable for evaluating its success or failure in terms of these outcomes. Equally, it makes no sense for an SC to ignore these priorities simply because they rarely occur in their community. A nation can be viewed as a collection of communities. If all communities ignored the national injury priorities, progress would be considerably impaired.

WHAT SHOULD SMALL SCS DO IN TERMS OF SURVEILLANCE?

The 2007 Cochrane Review of WHO SCs13 stated that their review was hampered by:

“The extremely limited information available for these programmes about the implementation process, and the impact of the intervention on risk factors for injury, prevent any real attempt to explain differences in outcome on the basis of process and impact factors.” (p9)

It is evident that most small to medium SCs are very unlikely to be able show any statistically significant impact on incidence of important injury. We would propose that better use of their limited resources would be to focus on measuring the effect of relevant risk factors for important injury. For example, MVTCs are likely to be a priority no matter what dimension of importance one chooses to adopt. Thus the monitoring of alcohol-impaired driving, bicycle helmet wearing, appropriate seatbelt/restraint use and speeding, all strategies known to reduce injury, would be a more appropriate use of their resources.

CONCLUSIONS

We have shown that there are major challenges to SCs and their supporting centres undertaking local injury surveillance. We have used six SCs in NZ to illustrate the issues. The communities ranged in size from approximately 44 050 to 216 900. In this respect, they are typical of SCs in other countries. As at March 2008, there were 133 WHO SCs with a mean population size of 140 768 (median 52 500) (http://www.phs.ki.se/csp/who_safe_communities_network_en.htm).

SCs elsewhere will face greater challenges than we have demonstrated in this NZ example. NZ in many respects is rich in injury data sources, having comprehensive national mortality, hospital inpatient and medical treatment (ACC) databases. Few developed countries have such a suite to choose from, and most developing countries would struggle to have one reliable data source.

Beginning early in 2009, applicants to the WHO SC network must demonstrate their capacity to meet a minimal set of activities for each criterion. Although these activities are still in draft form, we note that, for criterion 4 (Programs that document the frequency and causes of injury), the requirement still exists to obtain local data where that is possible. For criterion 5 (Evaluation measures to assess programmes, processes and the effects of change), it would appear that communities will not be required to demonstrate effects of change in terms of injury outcome.14

SC umbrella organisations who require SCs to undertake local injury surveillance and evaluation in terms of injury outcomes need to revisit the appropriateness of these requirements with a view to replacing them with criteria relating to the surveillance of safety behaviours (eg, use of cycle helmets) and other impact measures (eg, changes to playground equipment), particularly those that have been shown to reduce important injury.

• Download figure
• Open in new tab
• Download powerpoint