Article Text

Using synthetic control methodology to estimate effects of a Cure Violence intervention in Baltimore, Maryland
Free
  1. Shani A Buggs1,
  2. Daniel W Webster2,
  3. Cassandra K Crifasi2
  1. 1 Violence Prevention Research Program, Department of Emergency Medicine, University of California Davis, Sacramento, California, USA
  2. 2 Center for Gun Violence Prevention and Policy, Johns Hopkins University Bloomberg School of Public Health, Baltimore, Maryland, USA
  1. Correspondence to Dr Shani A Buggs, Violence Prevention Research Program, Department of Emergency Medicine, University of California Davis, Sacramento, CA 95817, USA; sabuggs{at}ucdavis.edu

Abstract

Objective To estimate the long-term impact of Safe Streets Baltimore, which is based on the Cure Violence outreach and violence interruption model, on firearm violence.

Methods We used synthetic control methods to estimate programme effects on homicides and incidents of non-fatal penetrating firearm injury (non-fatal shootings) in neighbourhoods that had Safe Streets’ sites and model-generated counterfactuals. Synthetic control analyses were conducted for each firearm violence outcome in each of the seven areas where Safe Streets was implemented. The study also investigated variation in programme impact over time by generating effect estimates of varying durations for the longest-running programme sites.

Results Synthetic control models reduced prediction error relative to regression analyses. Estimates of Safe Streets’ effects on firearm violence varied across intervention sites: some positive, some negative and no effect. Beneficial programme effects on firearm violence reported in prior research were found to have attenuated over time.

Conclusions For highly targeted interventions, synthetic control methods may provide more valid estimates of programme impact than panel regression with data from all city neighbourhoods. This research offers new understanding about the effectiveness of the Cure Violence intervention over extended periods of time in seven neighbourhoods. Combined with existing Cure Violence evaluation literature, it also raises questions about contextual and implementation factors that might influence programme outcomes.

  • firearm
  • public health
  • safe community
  • case-control study
  • programme evaluation

Data availability statement

Census block group-level data are available in a public, open access repository. Data on homicides, non-fatal shootings and arrests were obtained through both the Baltimore City Open Data portal (https://data.baltimorecity.gov/) and the Baltimore Police Department. Police Department data requests must be submitted to the agency.

Statistics from Altmetric.com

Background

Baltimore, Maryland has long been plagued by high rates of homicides and non-fatal penetrating firearm injuries (non-fatal shootings). The homicide rate in Baltimore, Maryland, has been one of the highest among the US cities with over 500 000 people, with the vast majority of these killings involving firearms.1 From 2003 to 2017, nearly 4000 Baltimore residents lost their lives to homicide and over 8000 were victims of non-fatal shootings.2 City leaders in Baltimore, as in many urban locales, have sought to implement community programmes to prevent violence among individuals at high risk for firearm violence.

Cure Violence is an intervention that employs community members to interrupt violence through conflict mediation, model and promote non-violence, and link individuals at high risk of violence to needed services.3–5 The Cure Violence model has been implemented and evaluated in multiple cities around the US.5–8 However, studies examining the programme’s impact on homicides and non-fatal shootings, while generally positive, have produced varied programme effect estimates across sites and cities, with some studies reporting no beneficial effects.4–10

The Cure Violence model was first implemented in Baltimore, Maryland, in 2007. Named ‘Safe Streets’, the programme was started in police posts whose total homicides and non-fatal shootings were in the top 25th percentile (see online supplemental figure 1 for a map with locations and operational periods for all programme sites). The impact of Safe Streets was first assessed in an evaluation of the original four programme sites in Baltimore using 2007–2010 operational data,11 and then again in 2018 using data on the then-seven Safe Streets sites operating between 2007 and 2017.12 Difference-in-differences regression strategies in both evaluations yielded wide-ranging effect sizes across sites in both homicides and non-fatal shootings. The 2018 evaluation found no aggregate protective effects of Safe Streets on homicides or non-fatal shootings. Models that estimated site-specific effects on homicides found that only one site (Cherry Hill) experienced a sizeable and statistically significant reduction, while the direction and effect sizes of homicide estimates for the other six Safe Streets sites ranged from null to a statistically significant increase in homicides in one site. Site-specific effects on non-fatal shootings showed that four of the seven sites had reductions in non-fatal shootings, while three experienced increases; none of the estimates were statistically significant.

The previous Safe Streets evaluations, like other Cure Violence evaluations, were limited in their ability to identify suitable comparison units for estimating programme effects because they used either traditional regression analyses or interrupted time-series approaches that do not use statistical methods to determine the most appropriate non-intervention controls for estimating the counterfactual for intervention areas. The objectives of the current study were to improve on the 2012 and 2018 evaluations of Safe Streets and extend the literature on the effectiveness of Cure Violence replications by using the synthetic control method (SCM), which has noted advantages over regression approaches for estimating intervention effects of non-randomised interventions.13 We hypothesised that by using the SCM method, we would generate more accurate estimates of intervention effects, allowing us to better assess the effectiveness of Safe Streets.

Methods

Design and setting

This study used a comparative case study design to estimate the impact of the Safe Streets programme on homicides and non-fatal shootings for each programme site operating in Baltimore, Maryland, during the years 2007–2017, using data from 2003 to 2017 to calculate preintervention and postintervention trends.

Data and measures

Incident-level data for homicides and non-fatal shootings occurring between 1 January 2003 and 31 December 2017 were obtained directly from the Baltimore Police Department2 or through Baltimore’s Open Data portal.14 Over 80% of all homicides in Baltimore are firearm-related, and Safe Streets is designed to address all violence regardless of weapon type; therefore, we included all homicides in our analysis. Data on the police posts where Safe Streets was implemented, as well as the sites’ dates of operation, were provided by the Baltimore City Health Department.

Our covariates, used to improve preintervention model fit, were selected based on prior firearm violence studies with similar aims and methodological approaches.15–17 Census block group-level data on the number of households, per cent of households below the poverty level, total population, per cent total male population, per cent males aged 15–34, per cent black, per cent white, median household income and per cent vacant housing were obtained from the 2005–2009 American Community Survey (ACS) 5-year estimates and averaged by police post.18 Arrests for drug possession, drug trafficking and weapon possession occurring between 2003 and 2017 were obtained from the Baltimore Police Department. All data were geocoded and aggregated to police posts by month and year.

The dependent variables were counts of homicides and non-fatal shootings within police posts by month and year. Given the volatility of monthly homicide and non-fatal shooting counts, we used 5-month moving averages to smooth the data.19 20

Analytic strategy

Finding appropriate comparison units can be difficult because areas where interventions such as Cure Violence are implemented often have substantially more violence than non-intervention areas. While strategies such as propensity score matching attempt to minimise the risk for omitted variable bias when treated and untreated units are quite different,21 identifying controls that most closely mirror the treated units, based on observable characteristics, should provide more valid estimates of the counterfactual than statistical approaches that generate estimates using non-treated areas that are much different from intervention sites.22

SCM is a suitable strategy for detecting the effects of an intervention in comparative case studies, when the treatment is applied in a non-experimental, non-randomised fashion.23–25 The SCM estimates outcome variable values to generate a counterfactual, or ‘synthetic control’, based on a vector of weights for non-intervention comparison units that minimises the root mean square prediction error (RMSPE) between the synthetic covariates and outcomes and those observed in the Safe Streets police posts prior to the intervention.21–25 The SCM captures trends in both observed and unobserved time-varying confounding factors throughout the study period.11 The method is not dependent on assumptions of regression analyses, such as parallel trends or the allocation of constant weights across all control units, that might limit prediction error minimisation.26 Whereas the root mean square error (RMSE) is a common measure of prediction error applied in studies using regression to help evaluate model fit,27 the RMSPE is similarly used in the SCM to assess model goodness of fit.23–25 Additionally, whereas regression analyses rely on a single model to estimate intervention effects across all sites, the SCM produces site-specific models designed to generate the best estimates for each intervention site.

The donor pool of controls for the SCM analyses was restricted to the 136 Baltimore City police posts without a Safe Streets programme during the study period. Twelve-month lagged averages for the dependent variables and arrest data for each preintervention year from 2003 forward were combined with the ACS 2005–2009 5-year block group estimates to generate preintervention trends for each model.

Although we were unable to include details on changes in Safe Streets leadership or staff or differences in intervention oversight by site as model covariates, those and other unmeasured neighbourhood-level factors may modify the effects of a local-level intervention such as Safe Streets in a way that could affect the intervention’s impact over time. Thus, 3-year, 5-year and 7-year programme effect estimates were generated for the two longest-running Safe Streets sites: McElderry Park and Cherry Hill.

During the study period, Baltimore experienced considerable civil unrest in April 2015 following the death of Freddie Gray, Jr, in police custody, and homicides and non-fatal shootings subsequently experienced sustained increases across the city. To assess how programme effect estimates were influenced by the unrest, we conducted a sensitivity analysis with the four Safe Streets sites in operation before and after the unrest by truncating the postintervention period to end in March 2015 and comparing those results with the primary model with data through December 2017.

Traditional tests of statistical significance are not computed via the SCM. To assess the likelihood that estimated effects were due to the intervention, ‘in-space placebo tests’, which treat each police post in the donor pool as if it received the intervention, were conducted to generate comparisons between the estimated per cent change associated with Safe Streets and the per cent change estimate derived from the placebo tests.23–26 We calculated the proportion of control posts in the placebo tests with an estimated per cent change in homicides and non-fatal shootings that was more favourable than the per cent change estimated in each Safe Streets post. This proportion, similar to a p value, provided an assessment of the strength of the associations found in the SCM.22 28

Geocoding of point data and aggregation to police post polygons were completed using ESRI Business Analyst 2015 software in ArcGIS Desktop V.10.4.1.29 Data management and analyses were performed in Stata/IC V.15.1 for Mac (64-bit Intel).30

Results

Table 1 compares the preintervention average monthly values of outcome variables and predictors in the treated police post, its synthetic control, and the unweighted pool of comparison units which were applied in the previous regression analyses. As illustrated by the larger variance between the unweighted comparison units and the treated sites in relation to the synthetic controls, the weighted approach used to generate the SCM provides closer estimates of the predictors than do the pooled unweighted comparison units. Furthermore, the prediction errors in the SCM, shown in table 2 as the RMSPE, were substantially smaller than the corresponding prediction errors from the regression models (RMSE), demonstrating that the SCM produces a better fit for estimating the counterfactuals and programme effects for each intervention site. This improved model fit can also be visually noted in graphs comparing the SCM preintervention predicted versus observed values for homicides (online supplemental figure 2A) and non-fatal shootings (online supplemental figure 2B) with the regression model preintervention predicted versus observed values for homicides (online supplemental figure 2C) and non-fatal shootings (online supplemental figure 2D). The graphs for the placebo tests showing the observed minus predicted values are shown in online supplemental figure 3A for homicides and online supplemental figure 3B for non-fatal shootings.

Table 1

Predictor balance averaged over preintervention periods for each Safe Streets site, their respective synthetic controls and the pooled unweighted comparison units

Table 2

Preintervention root mean square prediction error for synthetic control models and preintervention root mean square error for regressions in Safe Streets analyses

The SCM estimates of programme effects are shown in table 3 as per cent change increases or decreases in homicides and non-fatal shootings post programme implementation. Estimated effects varied widely by both site and outcome. For example, the Cherry Hill and Sandtown-Winchester sites saw reductions in homicides (−21% and −9%, respectively) but not non-fatal shootings (+11% and +15%, respectively). In contrast, the Madison-Eastend and Mondawmin sites experienced increases in homicides (+69% and +76%, respectively) and non-fatal shootings (+153% and +27%, respectively) during the study period. Lower Park Heights experienced non-significant decreases in both homicides and non-fatal shootings, while the homicide increases in Mondawmin and Elwood Park and the non-fatal shootings in Madison-Eastend were significantly different from the placebo test results.

Table 3

Estimated Safe Streets programme effects: per cent change, range of per cent change in control posts in placebo tests and proportion of control posts with better outcomes in placebo tests than Safe Streets posts

We found inconsistent programme effect estimates over time for the two outcomes in McElderry Park and Cherry Hill (table 4). In McElderry Park, the programme was associated with homicide reductions for the 3-year (−62%), 5-year (−48%) and 7-year (−24%) estimates, indicating attenuated protective effects on homicide over time, while non-fatal shootings increased relative to the counterfactual over the same time periods. In contrast, Cherry Hill saw relatively stable programme-related decreases in homicides and greater reductions in non-fatal shootings at each duration tested.

Table 4

The 3-year, 5-year and 7-year estimated effects for McElderry Park and Cherry Hill (proportion of control posts with better outcomes)

The results of the preunrest and postunrest analyses suggest that the civil unrest had varied impacts in the Safe Streets locations (table 5). For example, compared with its preunrest estimate for non-fatal shootings, Lower Park Heights appears to have had a postunrest reduction in non-fatal shootings relative to the estimated counterfactual, while Mondawmin, which was at the centre of the civil unrest, saw dramatic increases in both homicides and non-fatal shootings following the unrest relative to its synthetic control.

Table 5

Estimated Safe Streets programme effects for preunrest (through March 2015) and full study periods (through December 2017)

Discussion

This study estimated the effects of Safe Streets, a public health programme based on the Cure Violence model, in seven intervention sites in Baltimore, Maryland, using SCM. The findings indicate that Safe Streets has had disparate impacts on violence across implementation sites with more evidence of harm than benefit. Three sites—Madison-Eastend, Elwood Park and Mondawmin—experienced substantial increases in both fatal and non-fatal violence during the study period. These sites lost their funding after experiencing poor outcomes. While other sites did see programme-related reductions in at least one violence outcome, none of the seven Safe Streets sites experienced violence reductions outside the norm when compared with placebo tests.

The varied effect estimates provide additional context and nuance to the existing literature on Cure Violence interventions. The findings that Safe Streets in Baltimore was associated with significant increases in violence in three locations raise great concern, and when considered along with the null or negative findings in a number of past Cure Violence evaluation studies, the results of this research suggest that closer examination of implementation and contextual factors that influence programme effects is warranted. These findings may also suggest that programme enhancements or modifications are needed to achieve significant reductions in firearm violence in the neighbourhoods with exceptionally high rates of violence.

This study offered interesting comparisons with the 2018 Safe Streets evaluation. Both analyses showed that the effects of the programme varied widely by location. However, the regression analyses estimated that Cherry Hill experienced a significant reduction in homicide, while the SCM analysis in the current study found that, although Cherry Hill did experience homicide reductions, the effects were not significant. The two analytic approaches also yielded some differences in the magnitude of effect estimates. Given the model fit comparisons highlighted in this study, the findings from the SCM analyses are likely better estimates of Safe Streets’ site-specific effects than were the previous regression analyses.

Taken in aggregate, the evaluations of Safe Streets suggest that the initial potentially positive effects of the programme attenuated over time. Prior studies found that Safe Streets increased youth’s preferences for non-violent conflict resolution31 and was associated with increases in preferences for non-violent responses to interpersonal conflicts.32 Additionally, the Cure Violence model has been associated with reductions in youth’s willingness to use violence to settle conflict,33 improved confidence in police34 and increased confidence in a community’s ability to reduce firearm violence.9 However, it is important to note that community outreach and violence interruption programmes, regardless of branding, vary from site to site and city to city in their implementation, investment and management strategy. There is limited information on implementation differences across Safe Streets sites; each received the same training, city contracts and technical assistance.11 The differential programme effects found in this and other studies and the attenuation of early protective effects underscore the importance of strong implementation, appropriate oversight and meaningful support of community-level violence interventions like Safe Streets.2 3 Geographical and contextual differences across sites may also influence the model’s probability for success. For example, the Cherry Hill area is geographically isolated, which may offer unique protection that other sites do not, and a cross-sectional study of Baltimore found that Safe Streets-associated reductions in homicides were associated with a higher proportion of gang-related conflict mediations, while neighbourhoods without similar programme-associated homicide reductions saw more mediations for weapons and retaliatory conflicts.35 The underlying drivers of potentially violent interpersonal disputes vary by community, and they must be understood and appropriately matched with interventions that can adequately address those drivers. Future research should also examine how the theoretical concepts of the Cure Violence model apply to various communities in Baltimore and how the strategy might be adapted or enhanced to better achieve the desired outcomes.

Furthermore, promising findings from studies of Cure Violence programmes in New York suggest that programme success may be associated with increased financial, mental health and training resources for staff and programme participants, as well as the availability of wraparound services for the individuals engaged by outreach workers—things that have been lacking for Safe Streets Baltimore.2 9 33 34 Future research should examine distinct variations in implementation across sites and focus on how discrepancies in programme impact may be explained by factors such as programme oversight, outreach worker training and support, collaboration with community-based organisations, and the type, number, quality and utilisation of support and social services available to programme participants.

This study is the first known research to assess how civil unrest and increases in community violence, particularly following incidents of police violence, may influence the outcomes of violence interventions. While deeper investigation of unrest-related effects was beyond the scope of this study, the increases in violence in three of the four sites in operation when the unrest occurred suggest that Safe Streets may have been ill-equipped to address the flared tensions and increased violence that frequently occur in communities experiencing unrest following incidents of police violence.36 37 This finding importantly sheds light on similar challenges experienced during the summer of 2020, when dozens of cities across the US saw both civil unrest and coincident increases in violence, even in places where community-based violence prevention strategies existed but were hamstrung by the COVID-19 pandemic.

There are several study limitations to note. First, due to large differences in operational periods across the programme sites, the amount of data available to estimate preintervention trends or postintervention effects differed greatly and affected model fit. For instance, according to their respective RMSPEs, the models for the Safe Streets sites in Mondawmin and Sandtown-Winchester had better fit than did those for the longest-standing sites in McElderry Park and Cherry Hill. Conversely, the earlier sites benefited from much more postintervention data than did the sites that opened more recently. There was no appealing strategy for addressing this limitation. However, the RMSPEs were small for all sites, and the examination of programme effects over different time periods provided insight into variance across sites over time, irrespective of programme length.

An additional limitation is that the SCM is unable to account for breaks in the intervention. The intervention’s effects following the April 2015 civil unrest could not be isolated, although we attempted to account for the effects by estimating Safe Streets’ impact on violence preunrest and then for the entire duration of the study period. Similarly, temporary breaks in the programmes in McElderry Park and Mondawmin as a result of site reviews following programme worker arrests could not be modelled with the SCM. However, those programme interruptions were brief and thus unlikely to have had a major impact on the intervention’s effects in those areas.

Despite these limitations, the current study makes important contributions to the body of literature on community-level violence interventions. To our knowledge, this is the first study to use SCM to examine site-specific and time-varying effects of a Cure Violence programme model on violence outcomes. It is also the longest examination of a Cure Violence programme’s operation in multiple cities within one city, as well as the first to consider differential programme impacts following civil unrest spurred by police violence.

The current study illuminates challenges associated with achieving and sustaining reductions in violence using a programme like Safe Streets. It also elevates the urgent need to provide greater support for the workers who are tasked with outreach and violence interruption and risk their lives to reduce violence in communities. The model has demonstrated some success in other prior quantitative and qualitative evaluations, but variation in effectiveness highlights the importance of closer examination of various aspects of the programme. The results from this study emphasise careful implementation and continuous evaluation of violence prevention efforts. This research also points to a greater need to more deliberately identify and replicate critical components of effective community outreach and violence interruption work, and to more directly address systemic drivers of community violence, in order to reduce violence in communities nationwide.

What is already known on the subject

  • Cure Violence is a violence prevention model that has been employed to interrupt violence, facilitate service referrals and promote non-violence among individuals at high risk for violence involvement.

  • While studies of the intervention’s impact on violence have generally been positive, effect estimates across replication sites have varied widely, suggesting the need for further examination.

  • Prior analyses of Cure Violence have used either traditional regression analyses with panel data or interrupted time-series approaches that have limited utility for estimating the counterfactual for intervention areas and generating effect estimates.

What this study adds

  • The synthetic control method produces more accurate non-intervention comparisons for estimating programme effects than prior research.

  • Programme effects varied by site and outcome, suggesting that intervention effects are dependent on neighbourhood-specific and site-specific context.

  • Drivers of potentially violent interpersonal disputes must be understood and appropriately matched with interventions that can adequately address those drivers.

Data availability statement

Census block group-level data are available in a public, open access repository. Data on homicides, non-fatal shootings and arrests were obtained through both the Baltimore City Open Data portal (https://data.baltimorecity.gov/) and the Baltimore Police Department. Police Department data requests must be submitted to the agency.

Ethics statements

Patient consent for publication

Ethics approval

This study was deemed to be 'not human subjects research' by the Johns Hopkins Bloomberg School of Public Health Institutional Review Board.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Contributors SAB and DWW were responsible for the conception and design of the project, as well as interpretation of data. SAB and CKC developed the coding for analysis. SAB was responsible for data collection and management, data analysis and drafting of the article. CKC and DWW contributed to critical revisions of the article. SAB, DWW and CKC provided final approval of the version to be published.

  • Funding This research was supported by the National Institute on Drug Abuse (5T32DA007292-23 and 4T32DA007292-24), the Centers for Disease Control and Prevention (1U01CE001954-01A1), and the Johns Hopkins Center for Gun Violence Prevention and Policy. Funders had no role in the study design, statistical analyses, interpretations of data or drafting of this article.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.