• program evaluation

Funders require it. Practitioners fear it. Epidemiologists narrowly focus it on outcomes. Perspectives on evaluation vary, but there is general agreement that we need more of it in the injury field. The question is how much? and what kinds of evaluation can be done in a real world situation, with limits on resources? Randomized control trials may be the “gold standard” and quasi-experimental design a close second choice, but most of us do not have the luxury of engaging in such studies. Rigorous and multilevel evaluations that encompass formative (design), process (implementation), and outcome (knowledge, behaviors, injury rates, and institutionalization) measures are still relatively rare in the injury prevention literature.

Four articles in this issue (see 125, 130, 151, and 154) illustrate the range of possibilities as well as many of the difficulties inherent in evaluating community based programs. Three of the four employ multiple strategies in a pre-post design, with varying outcome measures. Although none of these studies is perfect, there is, nevertheless, much that can be learned from them.

The study of Bhide et al (154) contains some elements of a process evaluation by examining the effectiveness of the manner of distribution of a one shot, prepackaged educational program. It presents the number exposed to the program and the components that were implemented as intended, for example the leader's guide. Given the minimal penetration (16%) and only partial implementation (46%), it is unlikely that a better designed outcome based evaluation would have shown more encouraging, statistically significant changes. In fact, if the researchers had only used an outcome evaluation, we may have concluded that the program was not effective, when, in fact, the real issues are the penetration and its incomplete implementation. This study clearly shows why an essential part of evaluation is determining whether a program has been implemented as planned. And, in this case, it illustrates the special difficulties when programs must depend on schools to distribute the prevention material.

In the study of Lee et al (151), one element of an outcome evaluation of a three year bicycle helmet promotion campaign is presented. The study uses a comparison of self reported helmet wearing among teenagers in intervention and control sites as proxy measures for the injuries of interest. Despite the somewhat greater, but modest, increase in self reported helmet wearing over the five year period, many questions are left unanswered. Process measures, describing the extent of implementation, or the inclusion of observed helmet use as an outcome measure (as noted by the authors), would provide greater assurance that the conclusions are valid. They would also assist others who wish to replicate this approach. The use of a control group is an advantage of this study, but adds to the complexity. This is evident in the inability to obtain accident and emergency reports of injuries as planned. It also raises questions about possible contamination of the control sites by television and radio coverage of the campaign.

In contrast, the study of Hanfling et al (125) uses actual observations of the proper use of seat belts and car seats to evaluate the short term effects of a six component public education campaign. The study compares two intervention sites to control sites after nine months of intervention. Such a design has a higher level of methodological rigor than the previous examples and lends credibility to the positive results. The use of control groups is clearly an asset, but does not answer all questions. As in the previous study, the use of a multilevel intervention approach with several partners probably contributed to the positive findings, but it is next to impossible to conceive of methods that would determine the contribution of each component, that is, was one element of the intervention more effective than another. This study also identified potential confounding variables that spring up in assignment of control sites. It illustrates the complications that so often arise when trying to obtain good controls. Finally, had they included process measures, such as exposure to the interventions, the number of incentives, or number of citations issued by the police, this might have explained why the rate of restraint use achieved was much less than that reported in countries with well enforced legislation.

The study of Coggan et al (130) is the most comprehensive, ambitious, and rigorous in its evaluation. It is based on a multitargeted community injury prevention program that combines both qualitative and quantitative data. The investigators include a multilevel process and outcome evaluation, with intervention and control sites matched on several variables. They conducted a formative evaluation to choose the interventions for each of the targeted audiences. The qualitative data enhance our understanding of implementation details and illuminate the successful aspects of the program. The case studies provide the information needed to adapt programs to the culture. It is clearly the most sophisticated but probably the most costly of the four studies. Knowing more about the penetration rate might influence the replicability of this program, but these details are not given.

So where does this leave us? Despite the best intentions, constraints on evaluation are inevitable for community programs. What do we need to do a better job?