Article Text
Statistics from Altmetric.com
Lynda S Robson, Harry S Shannon, Linda M Goldenhar, Andrew R Hale. United States Department of Health and Human Services, National Institute for Occupational Health and Safety, Centres for Disease Control. Cincinnati, Ohio, April 2001 (free at: www.cdc.gov/niosh).
The stated aim of the publication is to “provide students, researchers and practitioners with the tools and concepts required to conduct systematic evaluations of injury prevention initiatives and safety programs”. This review contends that the authors have fulfilled their aim and commends the publication as an excellent, easily followed introduction to evaluation in the area of injury prevention.
Chapter 1 introduces effectiveness evaluation of safety interventions. Chapter 2 argues for planning right from the start, and covers models of evaluation, including qualitative and quantitative methods, choosing evaluation design, and practical tips. Chapter 3 deals with before and after design, including some of the threats to validity. Chapter 4 covers quasiexperimental and experimental design. Chapter 5 deals with study samples and who should be included in the intervention and evaluation process; chapter 6 with measuring outcomes; chapter 7 with qualitative methods; and chapter 8 with assessing whether the results are significant. Chapter 9 is a summary of recommended practices. There are three sets of appendices on models to assist in planning; examples of statistical analysis; and examples of reporting results. There is a glossary and a bibliography. At the end of the document there is an evaluation form provided as a mechanism for getting feedback and improving the document.
In thinking critically about the publication, there are some areas where it could be clearer about defining relevant research and evaluation questions and making decisions from them about the mix of methods, but this is more of a quibble on an issue of concern to the reviewers, than a major problem.
The range and scope of the literature canvassed is good. While there could perhaps have been a wider discussion of qualitative methods, this would have made the book longer, to its detriment.
The use of Patton as the base for qualitative research is a good choice because he does not get caught up in the sniping so often seen when qualitative and quantitative meet. Patton has an excellent description of the role of sampling in qualitative research.
Another small area of weakness is the lack of a discussion of the difference between statistical significance and practical significance. If the numbers are large enough (that is, if the power is high enough) even very small changes will reach the threshold of significance. Where mass data are being used this means that changes that seem to be small and meaningless in terms of output efficiency will be statistically significant and the findings publishable. On the other hand when dealing with small or minority groups even quite large differences will be seen as not reaching the threshold of statistical significance. The changes, however, may be of great practical significance even though they can’t be published. Some discussion of the level at which the threshold of significance should be set for different types of decisions would have been of value. This has been extensively canvassed in the journal Evaluation over the years.
That said, we return to the starting point of the review, that the guide is an excellent tool to improve evidenced based decision making in injury prevention.