EVALUATION AGAIN: THINK BIG

For many, “evaluation” is a feared word. It may be as intimidating for researchers as it is for those responsible for programs. The threat it conveys reflects the difficulty in doing scientifically respectable evaluation studies, and for program people, the ever-present possibility that the result will fail to justify their efforts. In spite of these barriers, we cannot responsibly ignore the pressure to evaluate. We cannot justify applying a different standard to the preventive interventions we advocate than those that apply to pharmaceutical manufacturers, for example. What is sauce for the goose is sauce for the gander: we are all bound by the need to make our programs evidence based. Thus, every preventive initiative should be evaluated as well as resources permit and any that are being promoted that make no attempt to do so must be viewed with caution and skepticism.

By far the most challenging tasks for evaluators is assessing the worth of community programmes. In this issue we present two examples of how difficult this can be (p 18 and p 23). As well, invited commentaries (p 6 and p 8) offer words of advice and some of consolation. Their suggestions are important for readers who intend to conduct this sort of evaluation in the future.

One of the most daunting issues is that most evaluations compare only one or, at best, two communities, usually before and after an intervention. But no matter how many subjects (or injured people) there may be in each community (that is, no matter how large its population) the main comparison uses an “n”of 2. This is so because from a statistical viewpoint, those in one community share many characteristics, that may affect how they respond to the intervention. They are not independent, but clustered. Thus, although the unit of analysis can be the individual, the effect of “treatment” is completely confounded with the effect of “cluster” and estimating these effects when there are only two units is almost impossible.

Both commentaries offer helpful suggestions on how to address this and related issues. In discussion with my colleagues, an additional idea was put forward that seems worth considering: whenever possible, we need to try to move from n=2 situations to n=4 or more. In other words, community evaluations should include (at least) two randomly assigned treatment communities and two control communities. This would provide an opportunity to examine the variation within each of the main groups and thus determine the extent to which clustering confounds the results. In short, we need to think bigger than has been customary.

A related message is about evaluating programs in general, especially, perhaps, those that are hospital based. Here lurks a technical challenge analogous to the n=2 issue. Assuming your hospital or program is conscientious enough to wish to evaluate its novel program, the first step is to find a suitable population. (Actually the first step is to find funds, but we'll overlook this for now . . .) The usual population of choice is those attending the emergency department (A&E). If these subjects are randomly selected, and can reasonably be viewed as independent, a certain number (X) of such patients randomly divided into experimental and comparison groups, should provide some convincing answers about whether your program is effective. The minimum number that X represents is determined by sample size calculations.

But my message is that if these calculations suggest that X=100 is good, it is a sad but true fact that 2 × (200) is better, and 400 better still. The more you have the smaller the differences you will be able to detect and the more informative your analyses. Funding agencies are increasingly aware of the need for adequate samples. My hunch is that you are more likely to be penalised (that is, not funded) by asking for too little money based on too small samples, than the reverse. So, think big; don't waste valuable time and energy trying to do evaluations on a shoestring. At least ask for what you need, and trust that the funders have read these commentaries and this editorial.

A NEW FEATURE: BONES OF CONTENTION

As well as our cleverly named popular Splinters & Fragments column, in this issue we introduce a cousin: Bones of Contention (see p10). We will continue to publish opinion-dissent essays, but only when we can find authors whose views are sharply divergent. “Bones” is intended to spark debate as well, and those who disagree with the authors are invited to submit rebuttals. In this instance, the authors, Schieber and Vegega, challenge the widely held view that we must choose between education and environmental safety measures to prevent pedestrian injuries. Some may see this as a straw man: few would disagree that some combination of both will always be preferable to either alone. They are too polite to put the corollary as bluntly as I see it: that although much lip service is given to multidisciplinary approaches, far too many programs, both governmental and non-governmental, continue to rely entirely on educational approaches. They do so in spite of overwhelming evidence that education alone does not prevent injuries!

INJURY PREVENTION ONLINE: WWW.INJURYPREVENTION.COM

We are about to move into the modern era. With this issue we will launch a full text web site. The internet is an extraordinary resource for almost anyone with access to a computer and modem. Editors and publishers see great potential for disseminating findings, ideas, and opinions and, not incidentally, for popularising their journals internationally. Over the next few years the numbers of journals online will increase exponentially and some predict that journals without full text web sites are headed for extinction.¹ Until now our web site was limited to a listing of contents, names of board members, and instructions for authors. The new site will offer much more.

The new site has the same address (as above) but originates from California. The elite club led by the BMJ at HighWire Press is a division of Stanford University's Green Library. HighWire's mission is to “foster research and instruction by providing a more direct link between the writers and readers of scholarly materials”.² As visitors to the BMJ's web site (www.bmj.com) will know, online journals have many added value features in contrast to the print version.

For example, all publishers on HighWire share “toll free links”. This means that references to articles in any of the more than 200 HighWire journals will link directly to the full text of that article without any subscription charges. Eventually, the BMJ Publishing Group aims to join an initiative that will permit references to link directly to articles in participating journals regardless of where they are hosted. References to non-HighWire journals already link directly to the Medline listing where available and there may be similar links to Current Contents.

Each new issue of Injury Prevention will be available in full (including figures and tables) with additional “web extra” material. These articles will be downloadable as “printer friendly” pdf files so you will not have to read them on screen. There will also be a fully searchable archive from March 1998, as well as abstracts and tables of contents from earlier issues.

Email @lerts customisable to your area of interest will automatically notify you when a new issue of a HighWire journal is available. A “Collections” feature will allow you to search quickly for material in subspecialty areas and link directly with the BMJ and other specialist journals in the BMJ Publishing Group stable.

Thus, Injury Prevention Online will be more than just an electronic mirror of the print journal. It will also permit you to express your views on anything we publish by using the “eLetters” feature. Comments submitted online can be posted within a few days (subject to editorial screening). Of equal importance, supplementary material such as more detailed tables, illustrations, and references for which there is now insufficient space in the print journal will be easily accommodated.

Finally, once all the wires are connected, we will encourage online submission and peer review and each subsequent issue will be on the web when the print version is distributed. Therefore, readers outside the UK will not have to endure the whims of various postal services before their journal arrives. Access to Injury Prevention Online will be free to all for a limited period, and will then be available at no charge to personal print subscribers. Online only subscriptions will be available to personal subscribers, and institutions can purchase a site licence allowing access to all qualifying members from both within the institution and off site (details available on the web site). Non-subscribers will be able to purchase individual articles or short term access to the whole site.

We are pleased to embrace these technological advances so that our entire diverse international injury prevention community can acquire information where and when it is needed. To fulfil the needs of authors and readers we ask for your comments on these developments, both through the web site and more traditional methods.

FIVE EXCELLENT WAYS TO ANNOY EDITORS

I teach a course on scientific presentations. The final slide is entitled “Remember: journals need you almost as much as you need them”, or words to that effect. Although it is true that most journals want the privilege of publishing excellent papers, and one way of ensuring that most are excellent is to have many to choose from, don't ignore the phrase “almost as much”. The fact is that, other than the truly superb papers editors might actually wrestle over, most papers are neither great nor awful. They inhabit the grey, twilight zone between these extremes. And, for the most part, there are plenty of such papers . . . certainly we receive far more than we could ever hope to publish. So, editors remain in the driver's seat; they heavily influence what gets accepted. It is not wise to annoy them.

Should you, for some bizarre reason or other decide to test this assertion, the following are five ways to annoy even the most tolerant of editors:

1. Do not bother reading or following the instructions for authors.

2. Do not check your manuscript for spelling and other basic errors.

3. Do not bring your references up-to-date and ignore those from countries other than your own.

4. Ignore all basic rules of grammar, syntax, and punctuation.

5. Be sure to be out-of-town when galleys arrive, or better still do not provide an address.

BOARD CHANGES

Editorial boards, like so many institutions, must change to survive. Sadly, this means saying farewell to many who have served the board so well. Alert readers will note the absence of many familiar names on the masthead of this issue and the appearance of new ones. They may also note that we have abandoned the post of regional editors. We welcome the new members, each of whom bring fresh ideas and expertise to help address the broadening scope of this journal. Personally, I want to thank all those who are departing, especially Terry Nolan, Liz Towner, Lyne Warda, Katharina Purtscher and Jim Nixon, all of whom have been with us since the inception of the journal eight years ago. I will greatly miss the old guard, and their guidance.