Article Text
Abstract
Objective: To evaluate the effects of a community based, all age, all injury prevention program, the Safe Living Program, on injury risk and injury rates.
Design: A quasiexperimental population based evaluation using an intervention and comparison community design.
Setting: The intervention community (Shire of Bulla, n=37 257) is an outer metropolitan area of Melbourne, Australia. The demographically matched comparison community (Shire of Melton, n=33 592) is located nearby.
Subjects and methods: The Safe Living Program in the Shire of Bulla targeted injury reduction in all settings with a focus on high risk groups. Strategies included program publicity, education and training, injury hazard reduction, and environmental change. Baseline and follow up measures of program reach, risk factors, and injury rates in both communities were used to evaluate program process, impact, and outcome.
Results: Increase in program awareness was moderate and similar to other community based programs. The program achieved injury hazard reduction on the road, in schools, and, to a more limited extent, in the home. Other changes in injury risk factors could not necessarily be attributed to the program as similar changes were observed in the comparison community. No significant changes were found in rates of injury deaths, hospitalisations, or emergency department presentations in the Shire of Bulla after six years. Self reported household injuries, mostly minor, were reduced in the intervention community, but had been higher at program launch than in the comparison community.
Conclusions: The Safe Living Program was unable to replicate the significant reductions in injuries reported in other community based interventions. Replication of apparently successful community based injury prevention programs in different settings and populations requires evidence based interventions, sustained and effective program penetration, reliable data systems to measure change, at least one control community, and sufficient budget and time for effects to be observable.
- program evaluation
- community based
- VIMD, Victorian Inpatient Minimum Database
- VISS, Victorian Injury Surveillance System
- program evaluation
- community based
- VIMD, Victorian Inpatient Minimum Database
- VISS, Victorian Injury Surveillance System
Statistics from Altmetric.com
- program evaluation
- community based
- VIMD, Victorian Inpatient Minimum Database
- VISS, Victorian Injury Surveillance System
- program evaluation
- community based
- VIMD, Victorian Inpatient Minimum Database
- VISS, Victorian Injury Surveillance System
Evaluating community based injury prevention programs is complex in terms of both methodology and the interpretation of results.1 Unknown efficacies of many injury countermeasures, lack of measurement of program reach and impact on risk factors, lack of continuity of data collections, and frailties of program sustainability have hampered many evaluations.1–8
In 1990, the Shire of Bulla, a local government in Victoria, Australia, implemented a community based injury prevention program over six years (1991–96). The Safe Living Program, an all age, all injury prevention program had objectives (box 1) and procedures similar to the Swedish model reported to be effective in Falkoping.9–12
Box 1: Objectives of the Safe Living Program
-
Increase community awareness of injury prevention and create a “safer community” environment.
-
Develop strategies which provide an overall reduction in the number and severity of injuries within the Shire of Bulla.
-
Reduce the number of hospital bed days that result from injuries.
-
Reduce the frequency and severity of injuries requiring hospital attendance or admission.
-
Reduce the incidence of accidental death.
-
Reduce injury hazards.
-
Increase the use of safety devices and equipment.
-
Institutionalise the program (added in 1994).
The Safe Living Program set out to determine whether reductions in risk factors and injury rates reported in the Swedish community based program could be replicated in Australia.13 Evaluation objectives are shown in box 2.
Box 2: Evaluation objectives of the Safe Living Program
-
Evaluate the extent to which the intervention objectives were met.
-
Undertake a formative evaluation to feed back into the program and into wider strategic planning processes.
-
Identify potential improvements to both the intervention and evaluation methodologies.
-
Assess ongoing effectiveness of the Safe Living Program on overall injury.
-
Examine the institutionalisation and sustainability of the Safe Living Program.
METHODS
Safe Living Program
The Safe Living Program aimed to increase public awareness of prevention measures, reduce injury hazards, and reduce the incidence and severity of injuries in the Shire of Bulla. The Shire of Bulla was responsible for program development, employment of program staff, program management, and budget administration.14 Working parties, comprising individual community members and representatives from organisations within and external to the community, contributed to specific interventions. Program strategies targeted injury reduction in multiple settings, particularly home, school, sports and road, with a focus on high risk groups, such as children and older persons, in these environments.14–16 Timelines for selected components of the Safe Living Program are shown in fig 1.
Subjects and measures
The evaluation of the Safe Living Program (Shire of Bulla population 32 257 in 1990) was a quasiexperimental study employing a comparison community design.13 The selected comparison community (Shire of Melton population 33 592 in 1990) was matched to the intervention community on relevant demographic variables. Both shires were within the then relatively small catchment of the Victorian Injury Surveillance System (VISS) for emergency department childhood injury presentations.17
The Safe Living Program evaluation encompassed process (fidelity of implementation as planned), impact (effects of program on injury risk factors), and outcome (effects of program on injury occurrence). The evaluation was formative with regular program and community feedback. Measures included program reach, community involvement, working party participation, program publicity, receipt and use of the home safety package, education and training course participation, safety product outlets, bicycle helmet wearing rates, implementation of footpath cycling legislation,16 and safety audits of school playground equipment and arterial roads. Data for many of these measures were collected from shire records and by using random sample pre-intervention (1990), interim (1993) and post-intervention (1996) household telephone surveys. The range for response rates for these three telephone surveys was 50%–62%, each involving a representative sample of over 2% of both populations.
Program outcome was evaluated by comparing injury deaths, hospitalisations, emergency department presentations, and self reports.13 Sources of injury data included: the Victorian Coroner's Facilitation System (all reported injury deaths to 1994/95); the Victorian Inpatient Minimum Database (VIMD) (all public hospital admissions for Victoria to 1996)18 and the VISS database (emergency department injury presentations) for children under 15 years.17,19 The VISS collection (five major Victorian public hospitals) was replaced from 1995 by the all age Victorian Emergency Minimum Dataset (26 hospitals encompassing 80% of all Victorian public hospitals) with data available for 1996. Injury readmissions within 30 days to the same hospital, and injury caused by adverse effects of medical treatment were excluded. Interhospital transfers (5.6%) were not excluded because of technical difficulties.
Analyses
Data were analysed using SPSS for Windows and Excel.20,21 χ2 Tests and differences in proportions were used to test significance of pre-intervention and post-intervention differences. Injury rates for the two shires were age standardised by the direct method using Australian census (1986, 1990) population projections for 1992–96 assuming linear growth between 1986 and 1991. Time trend models of injury in the shires were derived using injury frequency as the dependent variable in GLIM (generalised linear models).13,22 Sensitivity analyses were conducted for a range of comparative hospital admission rates.
RESULTS
Program implementation
The Safe Living Program implemented 113 activities in the first three years (box 3).14,23 Some activities were short term while others were implemented over a longer period or repeated intermittently.
Box 3: Major intervention activities in the first three years of the Safe Living Program
Publicity for the Safe Living Program
-
Publicity of the program in local newspapers.
-
Production of newsletter, Safe Living News.
-
Safety displays and promotions at festivals/fairs.
-
Household safety package to all households.
Education and training for injury prevention
-
Traffic safety education programs for schools.
-
Safety products outlets: education.
-
Train the trainer course in children's safety.
-
Introduction to emergency first aid course.
-
National sports trainers scheme courses.
-
Sports safety equipment grants.
-
Successful small farm management course (including safety).
-
“Walk with Care” for older pedestrians.
-
Safe living: out and about for older community members.
-
Home care staff talks and referrals.
-
Home safety talks to senior citizens.
Promotion and action for injury hazard reduction and environmental change
-
Production and dissemination of activities guide for seniors.
-
“Walk with Care”: environmental changes to roads and paths.
-
Local safety products retailers: identification and promotion.
-
Safety product sales.
-
Child safety centre sales.
-
Installation of safety switches in council buildings.
-
Sales of finger-safe devices and other products from manufacturer to prevent finger jam injuries.
-
Finger-safe devices installed in council preschool centres.
-
Smoke detector subsidies for older persons.
-
Child restraint fitting station.
-
Child safety harness rebate scheme.
-
Usage of Early Childhood Injury Prevention Program (ECIPP).
-
Safety audit of main roads and council action.
-
Safe routes to schools.
-
Footpath cycling and helmet wearing promotions.
-
Schools playground equipment safety audit and resultant action.
-
Occupational safety for council employees.
-
Shire and community health centre home handyman service.
-
Medication card for older community members.
-
Personal alarm call system for older community members.
-
Safety display homes in new housing estates.
Information about the Safe Living Program appeared once a week on average in the local newspaper throughout 1991–96. Fifty issues of the Safe Living Program newsletter were produced and sent to key stakeholders. A total of 345 persons participated in volunteer working parties.
Program reach of interventions ranged from <1% to 27%. The 1996 telephone survey found higher rates in the Shire of Bulla compared with the Shire of Melton for awareness of the program (47% v 3%) and receipt of the Safe Living News (17% v 3%). The two home safety packages delivered to all Bulla households in 1991 and 1996 were reportedly received by 27.5% and 9.3% of households, respectively. In Bulla, 37.4% of residents recalled safety displays at fairs compared with 26.6% of Melton residents. Pre-existing and widely available preschool and primary school traffic safety education programs were promoted. Altogether 75% of preschools and 86% of primary schools used the program in the Shire of Bulla in 1991 and 1993, compared with 50% and 56% respectively in the Shire of Melton.
Program impact
The Safe Living Program impact evaluation showed effects on risk factors associated with road and car occupant safety for children, bicycle safety, school safety, and safety training (table 1). Attempts to measure sales of safety products other than those listed in table 1 were generally unsuccessful.
Five per cent of households reported making safety changes as a result of receiving the home safety package in 1991 compared with 0.3% in 1996. Changes in other injury risk factors (installation of smoke detectors, first aid courses) were similar to those observed in the comparison community. No effect on the Bulla community could be identified related to high profile safety display homes which incorporated many safety design features.
Program outcomes
No significant changes occurred in rates of deaths, hospitalisations (fig 2), duration of hospitalisation (fig 3), or emergency department presentations attributable to injuries. Subanalyses by age identified no significant differences between the Shires of Bulla and Melton. Regression modeling for cause of injury hospitalisations found significant reductions for the Shire of Bulla compared with the Shire of Melton for motor vehicle occupant injury only.
Household report of injury in the Shire of Bulla declined from 19% to the same level as Melton (13%) by the interim survey in 1993. There was no difference in reported rates (both 13%) in the 1996 post intervention survey. Most self reported injuries were minor, requiring no medical treatment.
DISCUSSION
For the period 1991–96, the Safe Living Program was unable to replicate the reduction in injury rates reported in a Swedish community based study.9 Although program reach was modest, successful initiatives were demonstrated in specific areas where evidence based recommendations were made directly to both the shire and schools (road and playground equipment audits, footpath cycling law). However, many results were mixed even where promotional and educational initiatives directly targeted the Bulla population.
Evaluation method
The evaluation was complicated by changes to geographic boundaries and the restructuring of both local government15 and the VISS. Interpolated census data may have added further uncertainty. Budget restrictions limited process evaluation to selected measures.
The statewide change in funding structure for Victorian hospitals in July 199324 was associated with a rapid increase in Victorian hospitalisations in the following year, possibly distorting injury rates and masking small rate changes. These effects were not uniform by age or hospital so could not readily be included in modelling analyses. Sensitivity analyses for hospitalisation rates indicated that if Melton rates remained static, Bulla would have to experience at least a 25% reduction over six years to be statistically significant. Since the numbers of deaths were small, meaningful comparisons of frequencies or manner of death between the shires were not possible. Similar data quality issues have been reported for some of the controlled studies of community based injury prevention programs elsewhere.6,9
The comparison community had similar demographics and socioeconomic status to Bulla and was also located within the catchment of the VISS, a unique advantage for identifying childhood injury presentation to emergency departments. Despite their proximity, there was little evidence of program contamination in the Shire of Melton as measured by program awareness and receipt of Safe Living News. Nevertheless, general injury prevention initiatives across the state of Victoria such as safety displays at fairs, and school traffic safety education programs, showed effects in both shires. However, the greater effects in the Shire of Bulla were associated with enhanced promotion of these initiatives through the Safe Living Program.
More appropriate designs for community based injury prevention programs using cluster randomised trials6,7,25 would provide robust evidence for effective strategies. Logistical problems and substantial costs are major barriers to this approach.3 Indeed most community based injury prevention programs do not employ even a comparison community8 and few of the current 61 World Health Organisation Designated Safe Communities26 have published evaluations in the international literature.
Despite these methodological issues, data from a range of sources indicated a limited effect of the Safe Living Program, with results lower than the range suggested for an acceptable positive effect.27 As the rate of injury presentation per person was low, the binomial model provided as good a fit to the data as the Poisson model. Any substantial findings would be unlikely to differ greatly regardless of analytical method employed.5,27
IMPLICATIONS FOR PREVENTION PROGRAMS
Our results and those of others, indicate that a high level of program reach and uptake of targeted and sustained efficacious countermeasures is essential for achieving program objectives.6,7 Community based injury prevention may work better in cohesive, homogenous, stable, and isolated communities with good organisational involvement.9 The “snowball” effect gained by training professionals and community volunteers to implement the program was a vital tool in the Swedish programs. This suggests that insufficient recruitment and training for implementation may be a factor in the low effect achieved for the Safe Living Program.
Key points
-
The Safe Living Program achieved reductions in self reported injury, mainly minor, but had minimal effect on more severe injury.
-
Further development of the theoretical base for community based injury prevention and its evaluation is warranted, particularly as these programs are increasing in numbers in many countries.
Institutionalisation involving incorporation into the routine function of a responsible organisation such as local government may be an important component in maintaining community based injury prevention programs. The long term effects of such institutionalisation should be evaluated, particularly for program sustainability and injury prevention outcomes.
Major lessons
Comprehensive standardised documentation of intervention strategies, research methods, and results is essential for program evaluation. Systematic record keeping by organisations responsible for intervention programs is necessary not only for quality assurance management but also for evaluation. In addition to data on severe injury, the availability of both emergency department and, where possible, reliable measures of lower severity injuries seen in primary care, would enhance the scope of evaluation.28,29 Improved measures of hospitalisations beyond number of bed days would help clarify issues related to injury severity in evaluation studies, though accurate coding may be costly. As reporting on every available intervention measures is expensive and inefficient, there is a need to focus on those risk factor measures which are most strongly associated with injury reduction.
In addition, telephone surveys may be decreasing in value due to the high prevalence of answering machines and the increasing proportion of households declining participation. Key informant interviews may provide a useful contribution to explanations of successful and less successful program effects.
Comparison communities may be non-equivalent in a number of ways and differences may evolve over time (for example baseline injury rates, methods of injury data collection, size, and nature of community). To the extent possible, alternative data sources should be established as a contingency for unanticipated changes to data systems over time. Importantly, intervention populations with high baseline injury rates are likely to be more susceptible to reductions in rates due to regression to the mean. In future community based injury prevention programs, consideration should be given to: fewer, well developed and more targeted, efficacious interventions; greater community reach; achieving organisational change; and enhanced evaluation design, including benefit/cost analyses.1–3,7 In particular, larger communities or a number of randomised communities are required to demonstrate significant reductions in serious (hospitalised) injury rates.6
Acknowledgments
This research was funded by the Victorian Health Promotion Foundation, VicRoads, the Shire of Bulla, and the City of Hume in the State of Victoria. The authors are indebted to the program officers and the many other members of the Shire of Bulla community who participated in evaluation activities. A large number of staff at the Monash University Accident Research Centre made valuable and dedicated contribution to the evaluation over six years.
REFERENCES
Linked Articles
- Editorial