Article Text

Download PDFPDF
Evaluation of the efficacy of an intersection conflict warning system at two-way stop-controlled rural intersections: difference-in-differences and triple-difference analytical approaches
  1. Disi Tian1,
  2. Susan G Gerberich1,
  3. Hyun Kim1,
  4. Andrew D Ryan1,
  5. Darin J Erickson2,
  6. Nichole L Morris3
  1. 1 Midwest Center for Occupational Health and Safety Education and Research Center, Division of Environmental Health Sciences, School of Public Health, University of Minnesota, Minneapolis, Minnesota, USA
  2. 2 Division of Epidemiology and Community Health, School of Public Health, University of Minnesota, Minneapolis, Minnesota, USA
  3. 3 HumanFIRST Laboratory, Department of Mechanical Engineering, University of Minnesota, Minneapolis, Minnesota, USA
  1. Correspondence to Dr Disi Tian, Midwest Center for Occupational Health and Safety Education and Research Center, Division of Environmental Health Sciences, School of Public Health, University of Minnesota, Minneapolis, MN 55455, USA; tianx229{at}


Objective Intersection conflict warning systems (ICWSs) have been implemented at high-risk two-way stop-controlled intersections to prevent right-angle crashes and associated injuries. This study involved investigation of the impacts of ICWSs on crash reductions.

Methods The study used a quasi-experimental design to analyse the potential causal relations between Minnesota’s ICWSs and various crash rate outcomes (including total, injury, non-injury, targeted right-angle and non-right-angle crashes) in pre-post analyses. A restricted randomisation method enabled identification of three controls to each ICWS treatment intersection, and included as many comparable intersection characteristics as possible. Annual crash rates (per year per intersection) were analysed over the same periods before and after system activation for treatment and control intersections in each matched group. Pre-crash data for 3 years and post-crash data for up to 5 years were included, ranging from 2010 to 2018. Negative binomial regression models with generalised estimating equations were applied to estimate the average, immediate and continuing treatment effects of ICWSs, through the difference-in-differences and difference-in-difference-in-difference approaches, respectively.

Results The ICWS treatment was significantly associated with a decreasing trend for targeted right-angle crash rates posttreatment. Although not statistically significant, most crash rate outcomes appeared to be elevated immediately after treatment (statistically significant for sideswipe crashes only). Pre–post differences in average crash rates (over entire periods), except for incapacitating injury-related crashes, were not statistically significant between treatment and control intersections.

Conclusions The study provided important insight into potential causal associations between intersection safety countermeasures and crashes at high-risk rural two-way stop-controlled intersections.

  • multiple injury
  • engineering
  • motor vehicle � occupant
  • epidemiology
  • rural

Data availability statement

Data are available upon reasonable request. Data may be obtained from a third party and are not publicly available. The data utilized in this study were not publicly available. Data pertaining to intersection characteristics and associated crash records may be accessed through the Minnesota Department of Transportation upon request.

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


In 2018, rural areas accounted for 45% of all crash deaths in the USA.1 The fatality rate was 1.68 versus 0.86 per 100 million vehicle miles travelled in rural versus urban areas.1 In particular, rural two-way stop-controlled intersections are among the most dangerous rural sites that are susceptible to serious right-angle crashes, primarily resulting from drivers’ misjudgments of appropriate gap size of the non-stopped and high-speed mainstream traffic.2 3 These types, compared with other control types of intersections, were also associated with greater average costs for crash casualties whether in rural or urban areas.4

In many US states, the intersection conflict warning system (ie, referred to as ‘ICWS’) has been deployed at selected rural two-way stop-controlled intersections to prevent traffic injuries. These systems typically incorporate visual displays, such as a combination of traffic signage and flashing elements, installed on either the minor, major or both approaches of an intersection. As such, drivers waiting at stop signs on the minor approaches can be timely informed when it is safe to proceed (ie, provided by the minor approach ICWS, when activated), whereas the major approach ICWS is designed to raise drivers’ awareness of the presence of stopped vehicles at minor approaches when they enter the intersections from major approaches.

A few studies have examined the effectiveness of various ICWS countermeasures from many aspects.5–13 Some positive impacts of the ICWS included reduced total and injury-only crashes,5 6 decreased vehicle approaching speeds,7–9 increased accepted gap sizes,10 11 etc. Other studies identified more crashes within a short period following ICWS deployments,12 or less compliance with the stop signs at treatment intersections.8 11 However, research has not been well established for investigating potential casual relations between the ICWS treatment (vs control) and crash outcomes, in pre–post analyses.

The difference-in-differences (DID) method has been used to estimate treatment effects of traffic laws, and policies or safety countermeasures in transportation-related observational studies.14–16 Gilpin14 examined the associations between different state-level graduated driver license provisions and teenage drivers’ licensing rates and crash deaths, using the DID and difference-in-difference-in-difference (ie, an extension of DID, or triple-difference) methods. Li et al 15 performed 18 DID models to identify potential causal impacts of congestion costs on crash occurrences by injury severity and crash type in London, England. Choi et al 16 applied the DID approach to evaluate effectiveness of a safety zone intervention on preventing crashes involving senior pedestrians.17 18

This study aimed to evaluate efficacy of the ICWS treatment in Minnesota, using a DID analysis approach. It was hypothesised that the ICWS treatment would reduce total crash occurrence, particularly targeted right-angle crash types, and those associated with injuries.


This study used a quasi-experimental study design to ascertain the relation between ICWS treatment and various crash rate outcomes. The Minnesota state registered crash data were obtained and analysed for years 2010–2018. This range ensured 3 years of pre-crash data and a minimum of 2 years’ post-crash data for every ICWS time point initiation. The Strengthening the Reporting of Observational Studies in Epidemiology guideline for reporting observational studies was applied to this study.

ICWS treatment intersections

A total of 56 Minnesota ICWS treatment intersections were investigated (see Minnesota Department of Transportation’s website for specific design configurations).19 Selection involved six criteria that were initially used to justify the ICWS installations: traffic volume; right-angle crash density rate; nearby curve; skewness; preceding stop sign; and commercial surroundings.20 The ICWS activation (or ‘turn-on’) dates ranged from 26 November 2013 to 1 December 2016.

Control selections

This study used a restricted randomisation approach to identify control intersections for each treatment intersection at a ratio of 1:3. This approach was considered appropriate for a relatively small sample size,21 and enabled adequate comparable characteristics between treatment and control groups for plausible causal inferences, particularly in the pre–post quasi-experimental studies.21 22

All control intersections were initially restricted to rural two-way stop-controlled intersections. Then, a reduced cohort of intersections for each treatment intersection was generated through exact matching to the control pool that involved a representative sample of Minnesota rural intersections.20 Matching characteristics included the six ICWS selection criteria, the number of intersection legs (‘X’ or ‘Y’ shaped) and major through lanes (two-lane or four-lane intersection).

Subsequently, three control intersections were randomly selected for each treatment intersection from the eligible reduced cohorts. The study also sought to ensure a maximally balanced distribution of unrestricted (not used for initial matching) intersection characteristics across treatment and control groups by adjusting the initial seed size in the simple randomisation procedure. There were 11 unrestricted characteristics available for both treatment and control groups, as identified in table 1. These characteristics have been recognised as essential impact factors for rural-intersection-related crashes,23–25 based on prior knowledge and professional judgments.

Table 1

Distributions of unrestricted intersection characteristics between the ICWS treatment and control groups

Finally, the deployment of a restricted randomisation approach for ascertaining controls and balancing of unrestricted characteristics assisted the comparability of baseline crash risk and satisfaction of a pretreatment parallel trend assumption at the population level.

Variable definitions

This study evaluated crash outcomes in five crash categories, including total, all-injury, non-injury, right-angle and non-right-angle. Specifically, the injury severity levels were classified by the KABCO crash-record categories (K=fatal injury (killed); A=incapacitating injury; B=non-incapacitating injury; C=possible injury; O=property damage only (PDO) or non-injury). Various other crash types were also examined as subcategories under the non-right-angle crashes. Crashes were defined as those that occurred within 300 feet of intersection centres (ie, crash buffer zone).

Independent variables included (1) treatment (1=treatment group, 0=control group) and (2) post (1=posttreatment period, 0=pretreatment period). Pre–post periods for crashes were exactly matched within each ICWS-control risk set, specific to individual ICWS activation dates. All crash data were removed during the first month, following the activation date (considered a washout period). For the triple-difference analyses, an additional dummy variable, Time, was created to indicate the nth year relative to the activation date (ie, n is a non-zero value ranging from −3 to 4). Activation_time, representing the calendar year during which ICWS was activated, was included as a covariate in all statistical analyses.

Statistical methods

Due to limited study sample size and crash observations, the study examined both aggregated and annual crash rates (per intersection per year) in pre–post analyses, using the DID and triple-difference specification, respectively. Relevant crash rates were also calculated within various subgroups, categorised by injury severity and crash type. To address the observed overdispersion of crash outcomes, the study used negative binomial regression models, with relevant offsets of time, to compare crash rates between the treatment and control groups, over the pretreatment and posttreatment time periods. Generalised estimating equations (GEEs), with an exchangeable covariance matrix, were applied to account for correlations between repeated measures on the same intersection, nested within each intervention-control risk sets.26

Specifically, the DID specification follows equation 1:

Log (aggregated_crash_counts)=β01 treatment+β2 post+β3 treatment × post+β4 covariates + log (offset_aggregated_years)+ε.

The triple-difference specification follows equation 2:

Log (annual_crash_counts)=β01 treatment+β2 post+β3 time+β4 treatment × time+β5 time × post+β6 treatment × post+β7 treatment × post × time+β8 covariates + log (offset_annual_year)+ε.

In equation 1, the relative crash rate ratios (RRRs) were obtained from the exp(β3) estimations of the interaction term, adjusting for the pre–post changes of crash rates within the control groups.27 28 Equation 2 considered the temporal trend of the treatment effects. Primarily, the exp(β6) and exp(β7) estimations were of interest for the measures of effect. Specifically, exp(β6) indicated the immediate treatment effect (ie, intercept change) and exp(β7) indicated the continuing treatment effect (ie, slope change) during the post-period, controlling for the temporal differences of crash rates between the treatment and control groups during the pre-period.29 30 All statistical analyses were performed using SAS V.9.4.


Table 2 identifies the aggregated crash rates (per year per intersection), during the pretreatment and posttreatment periods, separately, within treatment or control groups. Overall, a slightly lower rate was found for total, all-injury, non-injury and non-right-angle crashes during the post-period, among both treatment and control groups. Specifically, the aggregated post-rate for incapacitating injury (A) crashes increased at treatment intersections, but decreased at control intersections (table 2). Table 3 shows this pre–post difference was significantly different between treatment and control groups (adjusted RRR=4.75). Regarding crash type, the aggregated crash rates potentially decreased for left-turn, ran-off-the-road and head-on crashes, and increased for sideswipe and rear-end crashes, associated with the treatment; yet none of these average treatment effects (crude or adjusted) were statistically significant (table 3).

Table 2

Descriptive statistics of the aggregated crash rates (per year per intersection) between the ICWS treatment and control groups, during the pretreatment and posttreatment periods

Table 3

GEE estimations for crude and adjusted RRRs from DID analyses

On examinations of the triple-differences, an elevated intercept (ie, all exp(β6)>1) was identified but with a reduced slope (ie, all exp(β7)<1) of most annual crash rates, when comparing treatment and control groups at different time points, pretreatment and posttreatment (table 4). In particular, a significant increase in the sideswipe crash rate was found, immediately after the treatment (ie, exp(β6)=7.85, p<0.05, table 4). As illustrated in figure 1, the total crash rates consistently decreased within the treatment group, whereas this trend was reversed within the control group, during the posttreatment period. At treatment intersections, injury crash rates were observed to decrease, initially, and then follow a gradual upward trend (figure 1). There appeared to be a positive shift from injury to non-injury crashes for the treatment intersections, during the earlier phase of treatment, whereas the reductions in total crashes tended to be driven by reduction in non-injury crashes, during the latter phase of treatment (figure 1). Figure 2 illustrated reversed trends for right-angle crash rates between the treatment (decreasing) and control (increasing) groups. Consistent reductions in right-angle crash rates indicated a continuing protective effect by the treatment, which also demonstrated a significant result, with an exp(β7) equivalent to 0.64 (p<0.05, table 4).

Table 4

GEE estimations for the immediate and continuing treatment effects from triple-difference analyses

Figure 1

Illustrations of pre–post annual crash trends (in crude crash rates) between ICWS treatment and control groups, by injury severity. ICWS, intersection conflict warning system.

Figure 2

Illustrations of pre–post annual crash trends (in crude crash rates) between ICWS treatment and control groups, by crash type. ICWS, intersection conflict warning system.


This study evaluated the safety impact of ICWS treatment on various crash rate outcomes. Among all study hypotheses, only targeted right-angle crashes demonstrated a significantly negative association with treatment, as reflected by a decreased trend for annual crash rates posttreatment. This finding was consistent with previous studies, where similar ICWS installations also reduced the number of targeted crashes (right-angle crashes included).5 6 Other types of crashes, such as left-turn crashes, that could also have resulted from drivers’ poor gap judgments at two-way stop-controlled rural intersections, were also observed to decrease in association with the treatment in the present study.

Of particular interest are the greater post-rates (either aggregate or annual) for sideswipe crashes, observed at ICWS treatment intersections. One possible explanation could be a more exaggerated, evasive manoeuvre, adopted by drivers in the presence of ICWSs. Previous studies have consistently demonstrated the ICWSs’ effectiveness in reducing vehicle speeds on major approaches.7–9 This effect could allow more reaction time for drivers on both major and minor approaches to proactively switch directions to avoid more direct and severe angle impacts during crossing or left turns. However, the possibility of drivers being distracted by the minor road ICWS signs, and failing to search for mainstream traffic when making right turns, cannot be ruled out, completely.

No statistically significant changes were identified in annual total and all-injury crash rates associated with the treatment. However, directionality for all relevant estimations suggest a possible continuing effect of ICWS treatments for crash reductions, particularly among the more severe injury crashes. Evidence of a similar protective effect has been identified previously: Simpson and Troy5 used an empirical Bayes’ pre–post analysis to examine four categories of ICWSs in North Carolina with varied sign placements; ICWSs demonstrated greatest benefits on reducing total, targeted and injury crashes, when signs were placed on street-side posts, rather than overhead.5 Using a similar approach, Himes et al 6 investigated aggregated data of multiple ICWS strategies from three US states (Minnesota, Missouri and North Carolina). The crash modification factors were reported to be statistically significant for various crash types, ranging from 0.73 to 0.85.6 However, since these two studies included either none or limited numbers of Minnesota’s ICWSs, the results have less direct comparability to the present study.

In contrast with previous studies, the DID analysis suggested a significantly increased rate for incapacitating injury crashes posttreatment. This result should be interpreted with extra caution due to minimal crash numbers and wide CIs. Importantly, a new electronic system for crash reporting, launched in Minnesota in 2016, could also have affected the study results.30 Compared with the prior reporting system, the new system was shown to outperform in reliability, capturing more fatal and serious injury crashes.30 Potential random bias should also be considered.

Notably, yet not statistically significant, almost all annual crash rate outcomes in this study appeared to increase immediately after treatment. One reason may be drivers’ unfamiliarity with the ICWSs as a new countermeasure. Misuse or disuse of the system could have undermined its intended efficacy, potentially introducing additional gap-judgment errors and other risks. Based on an earlier version of Minnesota’s ICWSs, a field evaluation revealed a 13%–24% increase in intersection ‘roll-throughs’ (without stopping), within less than 9 months posttreatment.8 Another report documented a case investigation involving immediate increase in injury crashes associated with a poorly sighted ICWS implemented intersection in Minnesota.12

The current study suggested no statistically significant average treatment effects if data were analysed aggregately, using the DID specifications (except for the incapacitating injury crashes as discussed above). This result was supported by a recent Minnesota state report indicating that ICWSs did not significantly reduce crashes of any type, averaging over the years pretreatment and posttreatment (involving 66 ICWS treatment and 76 control intersections).13 Application of the DID and triple-difference approaches in this study was beneficial to better demonstrate the Minnesota’s ICWSs’ efficacy, overtime, and improve study comparability to other investigations of average treatment effects of ICWSs.

Several potential limitations are identified in the present study. First, there was a small sample size and limited crash outcomes, potentially restricting ability to detect a significant robust treatment effect. It is also essential to note that for the triple-difference specifications, a sufficient number of crash observations are generally required to obtain reliable causal inference.31 While crash data were initially available since 2006, the parallel trend for crash rates was only valid within 3 calendar years before the earliest ICWS activation date (2010–2013). One reason for the disparities in crash trends could have been due to other possible modifications on the ICWS intersection characteristics to counteract high crash occurrences, prior to the ICWS installations.

Although the crash periods were exactly matched within each ICWS-control risk set, it is inevitable that some of the intergroup variations still existed due to a relatively large variance of ICWS activation dates. For this reason, it is possible that biased results may arise from a less well-established parallel trend assumption for both DID and triple-difference analyses, with varying activation times. More advanced statistical analyses might be considered to minimise this bias in the future.

Importantly, possible selection bias may exist from restricted randomisation for control selections, such as increased risks of subversion and technical errors.21 Unmeasured differences between treatment and selected controls, and other sources of bias, including ‘regression to the mean’, could also potentially lead to biased estimations of treatment effects and misleading conclusions.32 Other common biases associated with police-officer-based reporting, such as misclassification of crash-injury severity, missing data, etc, also cannot be neglected.33 Finally, the study results may be generalisable to similar rural intersection countermeasures; however, careful examinations against various influencing factors are recommended when inferring potentially causal associations. These factors may involve, but are not limited to, different sign designs and placements, and other factors pertinent to drivers or rural intersections.5 11


Findings from this study suggested the Minnesota’s ICWSs demonstrated a potential protective effect against targeted right-angle crashes, continuously overtime posttreatment. Although this study revealed no significant average treatment effects on crash reductions, future examinations of the short-term and long-term efficacy of ICWSs on preventing injury or severe injury crashes are recommended, using larger crash databases. The study findings provided important insight into potentially causal associations between intersection safety countermeasures and crash outcomes at high-risk rural two-way stop-controlled intersections.

What is already known on the subject

  • Intersection conflict warning systems have been implemented at high-risk two-way stop-controlled intersections to prevent right-angle crashes and associated injuries.

What this study adds

  • The study enabled evaluation of the efficacy of intersection conflict warning systems (ICWSs) on reducing various crash rate outcomes, using the difference-in-differences and difference-in-difference-in-difference analytical approaches.

  • Results from the evaluation of ICWSs demonstrated a potential protective effect against targeted right-angle crashes, continuously overtime, posttreatment.

  • The study results provided important insight into potential causal associations between intersection safety countermeasures and crashes at high-risk rural two-way stop-controlled intersections.

Data availability statement

Data are available upon reasonable request. Data may be obtained from a third party and are not publicly available. The data utilized in this study were not publicly available. Data pertaining to intersection characteristics and associated crash records may be accessed through the Minnesota Department of Transportation upon request.

Ethics statements

Patient consent for publication

Ethics approval

All analyses of this secondary crash database received approval from the University of Minnesota's institutional review board (identification number: STUDY00012505).


The authors thank Mr Derek Leuer and Mr Ian Saari from the Minnesota Department of Transportation for their contributed efforts and support in data acquisition.



  • Contributors DT, SGG, HK and ADR conceived and designed the study, with SGG having a major advising role. DT, ADR and NLM contributed to the acquisition of data. DT performed data management and data analysis. HK, ADR and DE provided critical feedback to the statistical methodologies. DT, SGG and ADR contributed to the initial draft of the manuscript. All authors reviewed, revised and approved the manuscript.

  • Funding The research was supported, in part, by the Midwest Center for Occupational Health and Safety, Education and Research Center (T42OH008434), Pilot Project Research Fund through National Institute for Occupational Safety and Health, Centers for Disease Control, and Department of Health and Human Services.

  • Competing interests The contents and interpretations of this effort are solely the responsibility of the authors. There is no conflict of interest with the funding agency, Minnesota Department of Transportation, or other associated entities.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.