Objective: To improve surveillance for deaths and injuries associated with potential motor vehicle safety defects.
Design: Vehicles in fatal crashes can be studied for indications of potential defects using an “early warning” surveillance statistic previously suggested for screening reports of adverse drug reactions. This statistic is illustrated with time series data for fatal, tire related and fire related crashes. Geographic analyses are used to augment the tire related statistics.
Results: A statistical criterion based on the Poisson distribution that tests the likelihood of an expected number of events, given the number of events that actually occurred, is a promising method that can be readily adapted for use in injury surveillance.
Conclusions: Use of the demonstrated techniques could have helped to avert a well known injury surveillance failure. This method is adaptable to aid in the direction of engineering and statistical reviews to prevent deaths and injuries associated with potential motor vehicle safety defects using available databases.
- consumer product safety
- epidemiologic methods
- safety defects
- FARS, Fatality Analysis Reporting System
- NHTSA, National Highway Traffic Safety Administration
- TREAD, Transportation Recall Enhancement, Accountability, and Documentation Act
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
- FARS, Fatality Analysis Reporting System
- NHTSA, National Highway Traffic Safety Administration
- TREAD, Transportation Recall Enhancement, Accountability, and Documentation Act
Crashes of utility vehicles and other light trucks precipitated by tire failures began to receive widespread attention in the American news media in February 2000. By September 2000 a congressional investigation revealed a national need for improvement in surveillance for vehicle and equipment defects that cause deaths and injuries.1 It was later determined by the National Highway Traffic Safety Administration (NHTSA) that failures of certain Firestone tires supplied primarily on Ford Explorers had killed 271 persons and injured at least 800.2 A major dimension of the controversy was that the problem was brought to light by a local television station rather than by NHTSA or by the manufacturers themselves. The first published, comparative statistical study (by Robertson) appeared on an internet website.3 The “failure” of NHTSA’s Fatality Analysis Reporting System (FARS) to trigger an early warning of the problem is remarkable given the coverage of FARS, the number of deaths, and the specificity of the vehicles and crash types involved.4
In October 2000 Congress passed the Transportation Recall Enhancement, Accountability, and Documentation (TREAD) Act under which NHTSA has issued regulations requiring manufacturers to periodically forward information including claims for deaths and serious injuries, property damage, consumer complaints, warranty claims, and field reports beginning in July 2003.5,6 The United States Department of Transportation’s Inspector General has recommended that innovative surveillance methods be adapted to these data—and to existing surveillance data—to detect motor vehicle safety defects at an early stage.7 In this study we illustrate the use of an “early warning” statistic related to several suggested for screening reports of adverse drug reactions in the United States Food and Drug Administration’s Spontaneous Reporting System that would be useful for this purpose.8
We used FARS data to analyze the occurrence of tire related and fire related fatal crashes of model year 1996 light duty passenger vehicles (that is, passenger cars and station wagons, pickup trucks, utility vehicles, minivans, large vans, and cargo vans) in single vehicle, fatal crashes over the period, 1995–2001 (4740 vehicles, 4737 with valid crash dates).9 Overall, tire failures occurred in about 2% (109/4717) of the vehicles without missing tire failure data. Fires occurred in about 3% (154/4737) of the crashes studied.
Crashes involving non-motorists were excluded from the analysis. The model year, make, series, body type, and vehicle type of all passenger motor vehicles for model years since 1981 were identified using the Highway Loss Data Institute’s VINDICATOR program, based on the reported vehicle identification number in FARS.10 We use the term, “vehicle fleet”, to denote a grouping of vehicles as identified by VINDICATOR through specific model year, make, series, body type, and vehicle type classifications. An example of a vehicle fleet would be “1996 Saturn Station Wagons” or “1996 GMC Truck S-15 Jimmy four door, 4×2 utility vehicles”. JMP statistical software, produced by the SAS Institute, was used for the purposes of data management and analysis in this study.11 Azimuth mapping software, produced by Graphsoft, Inc, was used to produce the maps.12 Mapping coordinates were obtained from the United States Census Bureau.13
A crash in FARS that involves a tire failure can be determined at the vehicle and driver levels. At the vehicle level, “tire factors” can be recorded denoting tires that are inadequate, inoperative, faulty, damaged, or defective. At the driver level, flat tires or blowouts are recorded. We have characterized a “tire failure” based on data at either level. We have set this variable to “missing” if vehicles are noted to have improper tire pressure or are coded as having encountered roadway hazards. Twenty of 4737 cases are coded as missing. We have counted fires based on the occurrence of any fire in the vehicle, whether or not it is the most harmful event in the crash. There were no missing data for this field.
Geographic analyses can aid in the identification of injuries and deaths associated with potential vehicle defects that are influenced by regional climatic or roadway conditions. For this study, crash locations were mapped at a county level.
For vehicle fleet i with Ni single vehicle, fatal crashes, the total event counts with tire failures are defined as NTiresi. This count of actual events may be compared with a baseline count that would be expected under explicitly stated conditions, ETiresi. A very simple value for the expected number of tire failures by vehicle fleet could be calculated as,
A refinement we prefer is to compute the expected value for a particular vehicle fleet i based on tire failures in all other vehicle fleets, not including fleet i. Therefore, the early warning statistic used in this study is based on,
Large values of the simple ratio, NTiresi/ETiresi presumably measure an “excessive” number of events compared to a baseline expectation. Unfortunately, even large values of this ratio may not be appropriate to directly serve as indicators of potential interest. There is considerable variability in the FARS tire data. ETiresi have relatively small values and large differences in the ratios are caused by changes to NTiresi of a single failure. For similar reasons, the use of a statistic such as [(NTiresi−ETiresi)/√ETiresi], is also problematic.
To address these issues, DuMouchel has suggested several interesting strategies.8 Applied to the FARS data for tire failures, one of the simplest techniques would be to approach the variability issue by testing as a hypothesis that the expected value of the tire failure count is actually ETiresi—given the observed count, NTiresi. That is, the probability of [X⩾NTiresi] is calculated assuming that X is distributed as a Poisson variable with mean ETiresi. This early warning statistic for vehicles with at least one tire related failure is defined as,
For vehicle fleets without any tire failures, EWTiresi is left undefined. For vehicle fleets with one or more reported tire failures for vehicles in single vehicle, fatal crashes, the calculation of EWTiresi for each vehicle fleet i follows the Poisson based formula,
The early warning statistic is readily extensible to other types of fatal crashes such as single vehicle, fatal crashes involving vehicle fires. In this study, we have defined EWFiresi for any vehicle fire in a single, vehicle fatal crash (not just most harmful event fires).
All non-missing values of EWTiresi are plotted separately for 50 makes and series of model year 1996 vehicles on fig 1 by calendar year quarter on a cumulative basis. After the first quarter of 1998, the early warning statistic for the 1996 Ford Explorer four door, 4×2 utility vehicle shows markedly extreme scores on this logarithmic scale. The first tire recall affecting this vehicle did not occur until the third quarter of 2000. In the fourth quarter of 2000, the statistic appears to level off. A second peak occurs in the third quarter of 2001 and then begins to fall with the second tire recall in the fourth quarter of 2001 affecting some of the tires sold as original equipment for this fleet. By the fourth quarter of 2001, this fleet had 34 vehicles with reported tire related crashes in 107 total fatal, single vehicle crashes (about 32%).
Figure 1 also shows two additional candidates with extreme EWTire scores. The second highest ranked vehicle in the figure at the fourth quarter of 2001 is the four wheel drive version of the 1996 Ford Explorer four door utility vehicle. The original equipment tires for this fleet were also affected by the recalls and replacement programs in 2000 and 2001. The third highest ranked vehicle at the last quarter shown in the figure is the 1996 Ford E-350 (Super Club Wagon) passenger van. The FARS data do not identify the manufacturers or the models of tires for these crash involved vehicles.
Mapping fatal crashes of potentially problematic vehicles can provide compelling, supplemental evidence of a need for further review in some instances. For example, it is well recognized that tire characteristics are affected by exposure to heat and climatic conditions.14 According to information from the vehicle manufacturer, tires from two different tire manufacturers were originally fitted to the 1996 Ford Explorer four door, 4×2 utility vehicle, depending on the vehicle’s plant of manufacture and the model year.15 Tires from one tire manufacturer were recalled (Firestone), the other tire manufacturer’s tires were not (Goodyear).
Figure 2 shows the locations of single vehicle, fatal crashes of 1996 Ford Explorer four door, 4×2 utility vehicles originally equipped with Firestone tires that occurred with and without reported tire failures through calendar year 1999. This figure can be compared with fig 3 showing single vehicle, fatal crashes through calendar year 1999 of the same model year 1996 vehicle originally equipped with Goodyear tires. These two figures show strikingly different patterns of tire failures.
Figure 4 shows non-missing values of EWFiresi plotted separately for 94 makes and series of model year 1996 vehicles by calendar year quarter on a cumulative basis. By the fourth quarter of 1997, the early warning statistic has become especially pronounced for the 1996 Dodge Stratus four door sedan. At the end of the period studied, this fleet had nine fires reported in 29 single vehicle, fatal crashes—about 10 times the average for model year 1996 vehicles in all other vehicle fleets involved single vehicle, fatal crashes.
An injury surveillance system could be devised to identify any instance of a potential problem that departs from an expected or baseline rate. At the same time, it is beneficial if a surveillance system can also prioritize potential problems with important consequences for public health. A statistical criterion for early warning screening that tests the likelihood of an expected baseline value, given the number of events that actually occurred, is a promising compromise between these two approaches. Rankings of vehicle fleets based on this criterion can be used as an alternative method to one that relies on counts of registered vehicle years or vehicle years of insurance policy coverage as a measure of “exposure”.
Wu and Meeker have developed a system also based on Poisson probability models to set decision criteria that signal quality problems in automotive warranty databases, emphasizing graphical tools to show failures over time.16 Our emphasis is more directed to the ranking of potential problems that merit additional statistical and engineering review. Such an approach may be more useful in an environment with fixed resources to commit to problem investigation.
The early warning statistics presented here depend on the actual number and the expected value of failure events. Both values are influenced by driver, usage, and other vehicle factors that may be unrelated to a potential defect. Refinements to the computation of the surveillance scores could include stratification by these risk factors. The expected values on which the surveillance scores are based might also be modelled to incorporate such factors. Even so, the rankings of the early warning statistic as applied to these examples of tire failures and fires are considerably robust. Increasing the expected value of the crash count (ETires in formula 2) by a factor of seven for the 1996 Ford Explorer four door, 4×2 utility vehicle would still leave this vehicle fleet with the highest ranked early warning statistic (EWTires in formula 4) of all makes and series considered for the 1996 model year at the end of the period studied. Increasing the expected value, EFires, by a factor of four for the 1996 Dodge Stratus four door sedan would not affect the ranking of this fleet’s early warning score for fires (EWFires) at the ending period.
When using police reported accident data for surveillance purposes, it must be recognized that some potential vehicle problems are more obvious than others. Tread separations, tire blowouts, vehicle fires, and rollovers are examples of possible problem indicators that can be readily seen by investigating officers in many instances. More subtle problems may not be so obvious. In contrast, NHTSA’s TREAD data and complaint databases are oriented to the reporting of specific component failures by affected manufacturers and consumers. The surveillance methods we demonstrate in this study are compatible with these databases. However, it should be noted that NHTSA’s consumer complaint database contains less than 10% of the number of complaints consumers make directly to manufacturers.7 For some categories of injuries from potential defects, FARS data should be recognized for their unique surveillance potential because case selection is not influenced in any known way by attention to a potential motor vehicle safety problem in the news media and because the coverage of the data is virtually complete.17,18 Unfortunately, annual FARS data are not now publicly released by NHTSA until approximately nine months after the end of the calendar year reporting period. This adversely affects the timely identification of potential vehicle safety issues from these data. The accessibility of NHTSA’s TREAD data to researchers is not yet known.
Crash deaths and injuries related to tire failures in utility vehicles and light trucks received considerable media attention in the summer of 2000. These casualties resulted in part from a remarkable failure of an important surveillance system to trigger timely remedial action by tire and auto manufacturers, as well as by government regulators.
Subsequent United States congressional action and federal regulatory activity will soon result in the generation of new, large surveillance databases. It is hoped that these data will avert a recurrence of the original problem.
Little attention has been devoted to improving the methods by which motor vehicle safety data may be analyzed to trigger appropriate early warnings of potential safety defects.
This paper extends a relatively simple surveillance method previously suggested in the statistical literature for post-marketing surveillance of pharmaceutical drugs by the United States Food and Drug Administration.
Innovative statistical techniques can help to prevent future, costly, vehicle safety surveillance failures. Such techniques must be consistently applied to data with sufficient detail, coverage, timely availability, and accessibility to researchers.
There is no certainty that any vehicle, tire, component, or safety equipment identified as a candidate for further review by the methods illustrated in this study will have a safety related defect. Nor is it likely that only those vehicles, tires, components, or equipment with a high surveillance score could have a safety related defect. The calculation of surveillance scores is not based either on legally required standards or on performance standards that are economically achievable with the best available design, manufacturing, and testing practices. Nevertheless, the application of a logically consistent statistical approach to potential vehicle defects can be reasonably expected to bring improvements in injury surveillance sensitivity and specificity. The needs for improvements in data coverage and detail, as well as the needs for timely data availability and accessibility to researchers, are well known from past experience. Innovative statistical techniques can also help to prevent future vehicle safety surveillance failures with consequent property damage, adverse commercial impacts, injuries, and loss of life.
Fatality Analysis Reporting System (FARS): a national census of fatal motor vehicle crashes on public roads in the United States, published annually.
National Highway Traffic Safety Administration (NHTSA): a United States federal regulatory agency mandated to promote highway safety.
Transportation Recall Enhancement, Accountability, and Documentation (TREAD) Act: a United States law requiring reporting of safety data to the NHTSA by vehicle manufacturers and others.