Methods

Process evaluation data were obtained from the program officer's diaries, reports and media file, and through semistructured interviews with key individuals.

Pre-intervention and post-intervention observations were conducted through random household telephone surveys in April 1992 and 1995 to gather data for the process (program awareness), impact (knowledge, attitudes, and practices), and outcome (injury in the previous two weeks) evaluation. Survey dimensions and response rates are shown in table 1. Self reported injury rates for the two week period before the surveys were calculated using the total persons surveyed as the denominator. These rates were not age standardised as the ages of injured persons were not recorded.

Emergency department injury data for five years (1991/92 to 1995/96) were obtained through the Victorian Injury Surveillance System (VISS) at the Latrobe Regional Hospital, the only public hospital within the relatively well defined geographic area of the program. Data collection processes and details of the VISS database have been published.^{6, 7} The surveillance system captured presentations of acute injury on the first attendance. Annual capture rates were 100% for admissions and not less than 85% for presentations according to regular audits of the department register.⁵

Targeted injuries were identified by the Injury Surveillance Information System codes for injury in home, sport, and playground locations.⁸ Data on injuries not targeted by the program were extracted as a comparison. Work and road traffic injuries were excluded from all analyses because they were the subject of statewide strategies over the intervention period. Injury incidence data were adjusted for capture rates.

Rates per 100 000 population were calculated using the estimated resident population for the program postcode areas purchased from the Australian Bureau of Statistics. Rates were age standardised to the Victorian population by the direct method.⁹

Linear trends in emergency department presentation rates were estimated using Poisson regression.^{10, 11} The analyses were conducted separately for all injuries and for more severe injuries (those requiring review, referral, observation, or admission) to examine program impact on injuries of varying severity.

Results

PROCESS

The key informant interviews revealed that the presence of the program increased the rate of implementation of safety strategies. Links between the injury prevention program and other local organisations became strategic partnerships resulting in increased participation in safety related activities by organisations such as the local municipal council and the police.

There were at least 46 000 educational contacts with the community on home injury prevention through a safety display home, a safety display in the hospital emergency department, and a mobile safety product display. In addition, home injury prevention education sessions for professional groups were attended by 170 people. More than 6000 home injury prevention resource items were distributed.

A playground safety education session was undertaken in two consecutive years with the local Primary School Principals' Association (40 members). Playground safety was presented in the local Bachelor of Education (Primary) university course, reaching approximately 60 trainee teachers.

Injury prevention materials were incorporated into the local football coaching courses and sports trainers' programs, reaching approximately 365 coaches and trainers. Sports injury prevention displays reached approximately 880 people. The program used a major sports sponsorship funded by a health foundation to promote player preparation, coach accreditation and codes of conduct for players, supporters, and clubs.

IMPACT

Despite consistent coverage of the program by local media, general awareness of local safety programs increased marginally between the two household surveys (18.7% aware of safety programs compared with 20.8%, p=0.42). A larger increase in awareness and knowledge was observed for home safety and this appeared to have been translated into action by a small proportion of respondents (table 2). There was a very small increase in the average number of safety features reported per household (1.9, 2.2, p=0.83).

Coinciding with partnership development with local government bodies, nine new community playgrounds were constructed and six pieces of hazardous equipment removed. Tan bark undersurfacing was replaced regularly in municipal playgrounds, and from 1994 a local government maintenance crew conducted monthly playground audits using tick box checklists.

There was no difference in the pre-intervention and post-intervention surveys in the proportion of respondents able to list ways to prevent sports injury (72.8%, 95% confidence interval (CI) 68.0 to 77.2; 72.5%, 95% CI 67.8 to 76.8). Although not directly observed, an increase in the sports safety equipment use, particularly for Australian football, was implied by policy changes and sales data. Partnership development between the program and football clubs coincided with the requirement by four clubs for the under 10 and under 12 year old teams to wear helmets. Low cost mouth guards purchased from the Latrobe Regional Hospital Dental Clinic increased from an average of four to eight per month in the first and second program years respectively.

OUTCOME

There was a non-significant decrease in the rate of self reported injuries recorded in the telephone surveys (62.7/1000 persons, 48.2/1000 persons, p=0.19). The age standardised rate per 100 000 persons for emergency department presentations for all targeted unintentional injury fell from 6593.7 in year 1 to 4821.0 in the final evaluation year (table 3). A decreasing trend in emergency department presentation rates was observed for home, sport, and untargeted injuries (table 4). The decrease for home injuries, but not sports injuries, was significantly greater than that for untargeted injuries. The only category in which more severe injuries declined was home injury.

Discussion

Like other community based program evaluations,^{10, 12, 13} this evaluation has a number of limitations, and clearly demonstrates the difficulty of conducting robust evaluation when appropriate data are not readily available and inadequate budgets limit evaluation design and activities. Methodological difficulties included poor response rates in the second household survey resulting in demographic differences, and the unknown reliability of self reported safety practices and household features. As is often the case, the community was self selected and measurement of the outcome was at the individual level.

The most significant limitation is the lack of comparison data, constraining conclusions about association of the program with the observed changes in impact and outcome measures. At the time of the study, Latrobe Regional Hospital was the only non-metropolitan hospital in Australia conducting all age emergency department surveillance. Statewide hospital admissions data that could have provided comparison for more severe injuries were not reliable for this time period due to large fluctuations reflecting changes to health care system funding.⁷ Budgetary constraints precluded the conduct of telephone surveys in a comparison region.

Time series analysis was also not an option since pre-intervention surveillance data were not available due to the close proximity of the start dates for the program and the surveillance system.

Therefore, injury categories not targeted by the program, excluding work and road injury that were targeted by other interventions, were adopted as an internal comparison. Differences between the trends in targeted and untargeted injury were observed for only one targeted injury category: home injury. This was also the area of greatest program activity, as evidenced by the large number of educational contacts and the increased awareness and knowledge demonstrated in the household survey. This compares with the smaller scope of educational activities about playground and sports injury, and no change in knowledge about sports injury reduction.

Changes in patient presentation patterns do not appear to explain the observed injury reductions. Latrobe Regional Hospital remains the only public hospital serving the Latrobe Valley. During the evaluation period, there was only one small (45 beds) private hospital serving the region, which had no facilities for outpatient injury management. Anecdotal evidence indicates that there was a decline in the number of general practitioners serving the area since the late 1980s, which would tend to increase, rather than decrease, emergency department presentations (Henk Harberts, personal communication).

Statewide injury prevention activities could also explain the differential injury reduction observed in the program. During the last two years of the evaluation period, statewide agencies were implementing the Victorian Injury Prevention Strategy, launched in September 1994.¹⁴ Home, sport and playground equipment related injury were all included in the state strategy, and the area of greatest activity was home injury prevention. Significant strategies included promotion of safe home design, a major scalds prevention campaign, and subsidised smoke detector installation. Sport and playground safety received less attention.¹⁵

Notwithstanding the above caveats, the program developed collaborative relationships with local organisations and contributed to structural, environmental, and organisational changes that have the potential to provide a cumulative benefit in terms of injury reduction. The extent of this contribution above that made by the statewide strategy in the latter years of the program is not able to be determined.

Similar evaluations are likely to provide diminishing returns in terms of advancing the understanding of community based programs as an injury prevention strategy. The quasiexperimental design has limitations, even if a comparison community is included. Controlled trials of randomly assigned multiple communities would obviously be more robust. The financial and methodological challenges of such trials are substantial and challenging.