The use of cost–benefit analysis in road assessments: a methodological inquiry
- Johns Hopkins International Injury Research Unit. Department of International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA
- Correspondence to Dr Adnan A Hyder, Johns Hopkins International Injury Research Unit, Department of International Health, Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, Suite E-8132, Baltimore, MD 21205, USA; email@example.com
- Received 16 November 2012
- Revised 18 March 2013
- Accepted 17 April 2013
- Published Online First 24 May 2013
Background Cost–benefit analysis is a useful tool for priority setting in road safety. The value of statistical life (VOSL) is a metric used to estimate the benefits of road interventions in cost–benefit analyses. The International Road Assessment Program (iRAP), for example, created a rule-of-thumb to calculate VOSL benefits of road infrastructure when performing cost–benefit assessments in countries where data on VOSL are sparse.
Aim To evaluate the rapid assessment metric developed by iRAP and provide suggestions for improvement in these methods.
Methods We replicated iRAP calculations in order to make a critical assessment of the sources, results and conclusions.
Conclusions We found the iRAP metric a good example for highlighting some relevant aspects that should be considered in any VOSL estimation in order to enhance its use as a guiding principle for assessing road interventions. Specifically, we recommend the explicit disclosure of the assumptions, the use of sensitivity analysis and the avoidance of omitted variables bias.
Road safety is a growing concern in low- and middle-income countries where 1.30 million road-related deaths and more than 47 million injuries occur each year.1 In an effort to reduce this burden, governments and non-governmental organisations are working to implement interventions that maximise healthy lives saved. These efforts are however conditional on their budget constraints and can be informed by performing cost–benefit analyses.
Such efforts have been carried out by the International Road Assistance Program (iRAP), a charity whose aim is to provide technological and technical support on safer road infrastructure in collaboration with governments and non-governmental institutions. iRAP has developed a ‘rule-of-thumb’ to estimate the value of statistical life (VOSL), which is a metric used to estimate the benefits of road interventions in cost–benefit analyses. The VOSL corresponds to the estimation of the society's willingness to pay (WTP) for reducing the risk of death in an given average individual.2 By definition, the VOSL must be calculated using WTP studies which measure the income the average individual is willing to forgo to reduce the risk of death.2 An alternative method that has been used is commonly known as ‘Human capital approach’ (HC), which estimates the estimated future income forgone due to death.2 This method is not currently accepted in the literature as a VOSL calculation technique because it only takes into account working population and ignores the indirect costs associated with death other than forgone labour productivity.2
The purpose of the iRAP metric is to calculate the cost of both deaths and serious injuries that can be prevented by improvements in road infrastructure, so that policymakers can rely on an ex-ante cost–benefit assessment.3 This rule-of-thumb was developed based on a WTP approach, and it is an appealing tool that can be used to quickly estimate the return of any road safety intervention, especially in low- and middle-income countries where resources are limited and the efficient allocation of resources is fundamental to maximising deaths averted. This is the only rapid estimation technique we know for VOSL calculation used frequently in injuries and road safety field; however, other attempts have also been made to calculate VOSL in other fields.4 This-rule-of-thumb has also been used by iRAP itself for priority setting such as designing investments plans.5
Despite this is an appealing tool to produce quick VOSL estimates (without investing the time these studies normally take), in our attempts to replicate iRAP results we found some gaps in the estimations which are important to review for researchers reproducing these metrics and for policymakers who are compelled to take investment decisions under uncertainty based on this information. A more detailed methodological description and a careful review of some of the methods used in such a rule-of-thumb are needed in order to improve the accuracy of the results obtained with this approach. In this paper, we assessed the empirical strategy and assumptions used by iRAP for the calculation of VOSL in road infrastructure. We aim to provide an academic critique of the potential consequences of the iRAP rapid assessment method, and to provide recommendations for improving such calculations.
In this section, we describe the methods used by iRAP in the assessment of the VOSL and the associated rule-of-thumb.
Value of statistical life
We first attempted to evaluate the sources of the published VOSL values in iRAP methods but references to the studies supporting them were not available in the iRAP methodological documents. However, iRAP apparently did not use available data on WTP values for countries and instead iRAP used only HC data, although a comprehensive meta-analysis was published in 2003.6 A more recent meta-analysis (after the publication of the iRAP methods report) has also been published.7 Both meta-analyses and other results demonstrate that for many countries in which iRAP used HC studies, WTP estimations had been performed and it is not clear why they were not used.
The iRAP approach is described in one of their recent publications where researchers used a set of VOSL for 11 developed countries (five of them using WTP measurements), and 13 developing countries (only two of them using WTP values) for which data and the equations to replicate the calculations are provided.3 All the countries listed in the iRAP methodological report are presented in table 1, including year and method used for VOSL calculation.3
In addition to the absence of clear references to the sources of the data on VOSL used in this methodological report, the empirical approach has a series of methodological issues that require further attention. The iRAP report provides a set of equations that are used to estimate the relationship of Gross Domestic Product (GDP) per capita and method of estimation on VOSL (equation 1) and the relationship of method of estimation on VOSL/GDP per capita (equation 2). 1 2
In these equations, METHOD is 1 if the method used to calculate the VOSL was WTP, and 0 if it used the HC.
We replicated these equations using the data provided in the iRAP report. The results of using equation 1 with the all the data presented in table 1 (24 countries plus Singapore for which no data were provided by iRAP) yield an elasticity value of 0.99, whereas for iRAP these equations yield an income elasticity value of 1.125 (β1=1.125). Given that our values were different than those reported by iRAP, we repeated the analyses using only 22 out of the 25 countries listed with VOSL values in table 1 (which omits Cambodia, Philippines and Laos for unknown reasons). The income elasticity value of the last estimation yields a value of 0.93.
From equation 2, the estimated value of VOSL/GDP per capita corresponding to iRAP equals to 71 (model 1). This is the constant value that has to be multiplied by GDP per capita in order to obtain each country's VOSL value. Further, we made an additional estimation (equation 3 or model 2 in table 2) in which we included the developed/developing variable and compare both models 1 and 2 using an appropriate goodness of fit metric to compare models, such as Akaike Information Criterion. 3
Here, DEVELOPMENT LEVEL is 1 if the country is defined as a developed country according to table 1, and 0 otherwise.
In order to estimate the effectiveness of interventions preventing serious injuries, iRAP also assessed data on the Maximum-Abbreviated Injury Scale (MAIS) from the USA and Thailand, looking at the distribution of injuries with MAIS codes 3–5. As the distribution of injury codes for all road users was found similar in both countries (USA and Thailand) and data on costs were available only for the USA, iRAP weighted the US data in costs by the distribution of each MAIS category in order to calculate the estimated cost of serious injuries with respect to VOSL. The result was US$510 000, which represents 17% of the VOSL in the USA, which is $3 million.3
As the rate of injuries involving pedestrians is higher in developing countries,8 iRAP carried out the same exercise using only US pedestrian injuries data.5 The results found that an average pedestrian injury costs 28% of the US VOSL. For this reason, iRAP established a rule-of-thumb for the cost of injuries of 25% (20%–30%) of the total value of a fatality in any given country.
During our assessment of the iRAP methodology, we did not find listed any reference about the sources of the VOSL estimation. This is an important pitfall, as previous literature has discussed the differences existing between studies using stated versus revealed preference methods.2
By assessing and replicating the iRAP analyses we found that the iRAP report extrapolates values from developed to developing countries and argues that a country's VOSL/GDP per capita are better explained by methods of data collection (either WTP or HC) than by being classified as a developed or developing country. However, the criterion used to define model fit is the R-squared of the regression obtained by iRAP, a metric that is currently not considered to be appropriate to compare model's fit.9 ,10 In table 2, we replicated the iRAP results from equation 2. Model 1 corresponds to our replicated iRAP regressions and model 2 corresponds to an additional regression (equation 3) in which we included the developed/developing variable. Using an appropriate goodness of fit metric to compare models (such as Akaike Information Criterion), we found that a better fit with the data are reached, even though the relationship of VOSL/GDP with this particular variable has a p value of 0.068 (table 2). The inclusion of the developed/developing variable is important as its absence yields an omitted variables bias. Such bias is demonstrated in table 2, where both constant and coefficient for WTP method are overestimated (by 10% and 17%, respectively) when the regression is not controlled for country's development stage.
Regarding the methods carried out to evaluate serious injuries, we found three additional issues. First, the iRAP report ignores existing evidence such as the Hojman et al study (2005),11 where a more comprehensive estimation of serious injuries relative to VOSL was made. Second, this method assumes that the relationship of cost of a serious injury/VOSL is constant across countries. This assumption is controversial as developed countries might devote more resources to treat serious injuries (both in emergency medical services and long-term care), which might overestimate the figure for developing countries. Third, the probability of being injured as compared with dying from an injury for any given level of trauma is higher in developed countries, as emergency medical response systems are more organised and responsive and provide higher quality of emergency care. The latter is actually evidenced in the iRAP data by the comparably higher proportion of individuals in MAIS 5 in the USA and Hong Kong compared with Thailand (this fact is shown in figure 1, which displays data from McMahon and Dahdah, 2008; pp.10; table 8).3 Both facts imply that by inferring the results from the USA to any developing country, an overestimation of the costs of serious injuries is likely and, hence, these calculations must be contextualised to each study setting.
In this paper, we discuss the gaps found in the rule-of-thumb and methods being used for infrastructure assessments. We review the iRAP methods with the aim of providing constructive feedback that will improve the methodology of the cost-effectiveness analyses carried out in road safety specifically to encourage the explicit disclosure of the assumptions, use of sensitivity analysis and avoidance of omitted variables bias. By doing this, we aim to encourage the community involved in injury research to promote more transparent analyses, acknowledge the limitations of the methods used and take action to minimise the effects of those limitations in order to increase the ability of countries for taking optimal choices on road safety investments that can save lives.
iRAP has created a rule-of-thumb approach in order to rapidly estimate VOSL values for developing countries. VOSL calculations are useful to perform cost–benefit analysis on road safety interventions to generate estimates for decision making.12 However, the sources of the VOSL data used for such calculations are unclear and apparently important sources of data have been neglected. We have also demonstrated how the calculations performed might overestimate the impact of the assessed road safety investments as there is a strong omitted variables bias in VOSL calculations and costs for serious injuries are based on the distribution of serious injuries from the USA, which are unlikely to be similar to the distribution in any developing country as road safety, enforcement and, generally, survival to crashes is higher in the former. Previous literature13–16 has extensively discussed how VOSL calculations vary across development stages, countries, provinces or even socioeconomic groups and, hence, sensitivity analyses are strongly recommended to allow policymakers to model different scenarios that fit each country's circumstances.
As most of the countries with WTP measurements are developed nations, the results for VOSL might be overestimated. This possibility arises from previous literature on international benefits transfer which has found average variations of 38% on VOSL values between developed and developing countries.13 As previous research has used higher income elasticities (one example is the study of Ozawa et al which assumes an income elasticity of 1.517), we believe it is cautious to apply sensitivity analyses to the income elasticity value, which might help to show a more realistic picture of VOSL values across countries and allow policymakers to run different scenarios depending on each country's circumstances. Finally, including data obtained by HC techniques is not currently accepted as a valid tool to produce VOSL estimations.2 ,4
In conclusion, we have found a methodological issue in the way evidence might be generated in the road safety field, and we want to use the case of the iRAP rule-of-thumb as an example of how metrics on road safety can be improved. We recognise that performing research on VOSL is challenging, the benefits transfer approach is often the best method to use in countries that do not have WTP values and often the nature of the work in road safety requires the production of quick estimations with a margin of error. However, researchers in road safety should review existing approaches and explore ways of improving methods. We hope that this paper is a contribution to better measurement of VOSL in road safety, as well as a call for including sensitivity analyses in estimations, explicitly addressing the limitations of the analyses performed and providing clear sources of data.
What this study adds
This study provides a critical review of one type of a rapid assessment tool for value of statistical life, demonstrating that such measurements might yield biased estimates leading to non-optimal choices among decision-makers.
What is already known on this subject
Cost-benefits analyses are commonly used to estimate the return on investments for road safety interventions in road infrastructure. Rules of thumb to provide fast estimates on the value of statistical life (the unit of the “benefit” measurement of the cost-benefit analysis) are popular among decision-makers of road-safety and other type of interventions.
We thank Abdul Bachani from Johns Hopkins Bloomberg School of Public Health for his useful comments during the development of this manuscript. We also acknowledge the role of the anonymous reviewers of Injury Prevention who provided an outstanding feedback during the production of this constructive critique.
Funding This analysis was partly supported by the Global Road Safety Program of Bloomberg Philanthropies at the Johns Hopkins Bloomberg School of Public Health, USA. Adnan Hyder is also supported by grant #5D43TW009284 from the Fogarty International Center of the U.S. National Institutes of Health (Chronic Consequences of Trauma, Injuries, Disability Across the Lifespan: Uganda).
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.