Article Text
PDF
Abstract
Background Too little is known about the effectiveness of efforts to prevent firearm violence. We evaluated California’s Armed and Prohibited Persons System (APPS), which identifies legal purchasers of firearms who have become prohibited persons and seeks to recover all firearms and ammunition to which they have access.
Design and methods This cluster-randomised pragmatic trial was made possible by APPS’s expansion from a small pilot to a continuing statewide programme. We included 363 California cities, allocated to early (n=187) or later (n=176) intervention in blocks stratified by region within the state, and within region by population and violent crime rate. The study period began 1 February 2015; region-specific end dates ranged from 1 May 2015 to 1 February 2016. Analysis was on an intention-to-treat, difference-in-difference basis using generalised linear mixed models and generalised estimating equations with robust SEs. The population-level primary outcome measures were monthly city-level counts of firearm-related homicides, robberies and aggravated assaults. The primary model was adjusted for stratification variables; city-level population, population density, socioeconomic status and firearm purchasing; year; and month.
Findings Allocation groups were well balanced on baseline characteristics and implementation measures. In adjusted models, allocation to early intervention was not associated with statistically significant differences in any primary outcome measure; these findings were robust to multiple sensitivity analyses. There was some heterogeneity across regions.
Conclusions The APPS intervention directly affects a very small percentage of the population, limiting its potential for population-level effects. Individual-level analyses may provide a better estimate of the intervention’s effectiveness.
Trial registration number NCT02318732.
- Firearm
- Violence
- Randomized Trial
- Policy
- Enforcement
Data availability statement
No data are available. Data are not currently available as analyses are continuing.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Many firearm owners who become prohibited persons, often following events associated with an increased risk of future violence, retain their firearms.
WHAT THIS STUDY ADDS
In an intention-to-treat analysis, a programme to recover firearms from prohibited persons was not associated with beneficial effects at the population level.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
The intervention affects a small proportion of the population and, despite these findings, may have benefits for affected individuals.
Introduction
Firearm-related interpersonal violence is a major health and public safety problem in the USA, and rates are highest in urban populations. In mid-2022, an estimated 21% of American adults had experienced gun violence in the previous 5 years, either themselves or through a family member or close friend.1 In 2021, firearms were used in an estimated 20 966 homicides (81% of all homicides) and an estimated 326 894 non-fatal violent victimisations.2 3
One strategy for preventing firearm violence is to prohibit access to firearms by persons believed to be at high risk for committing violent acts. Background checks and denials of firearm purchases by prohibited persons, for example, can reduce the incidence of arrest for violent and firearm-related crime by at least 25% among individuals who are directly affected4 5 though they may not, by themselves, have similar benefits at the population level.6 7
Firearm access involves both acquisition and possession, however, and purchase restrictions have largely not been complemented by efforts to prevent retention of firearms by persons who acquire them legally and subsequently become prohibited persons. Transitions to prohibited-person status among armed persons are particularly concerning because they frequently arise from events associated with increased risk for future interpersonal violence or self-harm. Examples include convictions for violent crimes,8–10 involuntary hospitalisations for acute mental illness associated with dangerousness to self or others11 and domestic violence restraining orders (DVROs).12
Available evidence suggests that many legal firearm purchasers in the USA retain their firearms after becoming prohibited persons; California data from 2015 suggest that at least 100 000 have likely done so nationwide following prohibitions due to felony convictions,13 and no doubt others have remained armed following DVROs and other prohibiting events (see online supplemental file 1).
Supplemental material
California has implemented a systematic, large-scale effort to address this problem. The Armed and Prohibited Persons System (APPS), a programme of the California Department of Justice (CA DOJ), uses that agency’s archive of firearm sales and other transfers to identify firearm owners who have recently become prohibited persons under California or federal law. Specially trained APPS field teams then attempt to contact those individuals and recover all firearms and ammunition to which they have access.
APPS was first implemented in November 2006 as a pilot programme that conducted fewer than 2000 investigations each year.14 15 Following a series of mass shootings in 2012, the California Legislature required APPS to be implemented statewide beginning in 2013 and greatly increased its funding. The enabling legislation was adopted as an urgency statute, a measure ‘necessary for immediate preservation of the public peace, health, or safety’16; the bill took effect immediately on its enactment.17 Working with CA DOJ, we designed and implemented a pragmatic18 cluster randomised trial that allocated more than 1000 cities and census-designated places (CDPs) to early or deferred implementation in a staged roll-out of the programme, the protocol for which was published in Injury Prevention.19
This report presents findings for a population-level intention-to-treat analysis. There are precedents for such focused interventions, which directly affect only a small proportion of a population, to have beneficial population-level effects on interpersonal violence. Boston’s CeaseFire was associated with significant reductions in youth homicide and firearm assaults.20 Operation Peacemaker was associated with decreases in firearm violence in Oakland.21
Our aim here is to determine whether allocation to early APPS intervention is associated with changes in monthly counts of firearm and non-firearm violent crimes reported to law enforcement agencies. Our subjects are 363 California cities (crime counts are not available for CDPs).
Methods
Overview
The trial used a wait-list control design to evaluate an intervention that had been fully developed and was in operation before the trial began. Its design took advantage of the fact that, as APPS expanded, implementing the intervention across all of California simultaneously was impossible. The trial did not alter the content of the intervention and affected only the order in which cities received it. A detailed description of the intervention is in the online supplemental file 1.
The 363 included cities contained an estimated 11 000 APPS-eligible individuals (prohibited persons who retained their firearms) at the time of randomisation. Cities were stratified by geographical region within the state, and within region by population and violent crime rate, and were then randomised 1:1 within strata to early (group 1) or deferred (group 2) intervention. The trial start date was 1 February 2015; region-specific intervention end dates were defined as the month prior to the onset of the intervention in group 2 cities in each region.
The unit of analysis is the city month. The outcome measures (see the Outcomes section) are monthly city-level counts of reported crimes and the proportion of all violent crimes that involved firearms. The primary predictor is a time-dependent variable for the extent to which a city has experienced APPS implementation.
Data sources
Our primary data source was the APPS database, which contains identifiers for armed and prohibited persons, pointers to other CA DOJ records that provide grounds for the determinations that an individual owns a firearm and is a prohibited person, and information on the nature and duration of prohibitions. The linked APPS encounter database includes detailed reports of all field team encounters and attempted encounters with APPS-eligible individuals. APPS encounter records comprise detailed circumstantial data for all contacts and attempted contacts, a narrative describing encounters and their outcomes, and information on recovered firearms.
California’s Dealer’s Record of Sale (DROS) database contains electronic records for handgun purchases from 1996 onward and long gun (rifle and shotgun) purchases from 2014 onward. DROS includes private party transfers, which must be processed by licensed retailers in California. We used DROS records to generate a proxy measure for city-level firearm purchasing (see online supplemental file 1).
We obtained monthly crime counts from the California Criminal Justice Statistics Center’s archive of Return A and Supplementary Homicide Reports filings for the Uniform Crime Reporting (UCR) programme.
We used a categorisation scheme developed by the California Department of Social Services to divide the state into regions: areas of relative geographical and labour market continuity defined to optimise socioeconomic homogeneity (online supplemental figure 1).22 The 2010 Census and the 2009–2011 American Community Survey provided community demographic and socioeconomic characteristics. California’s Behavioral Risk Factor Surveillance System (BRFSS) surveys included measurements of household firearm ownership in 2001–2004, 2009, 2017 and 2018. We used these data to generate a supplemental proxy measure of household firearm ownership at the county level (see online supplemental file 1).
Participants
APPS is a mandatory programme under state law; we did not seek consent of cities for participation. In consultation with CA DOJ, we excluded three groups of cities prior to randomisation (online supplemental figure 1). Two sets of exclusions were at the county level. CA DOJ indicated that it would not be fiscally appropriate for APPS to intervene early in the study period in 24 isolated, sparsely populated counties located primarily in the northernmost and eastern parts of the state. (21 of 22 counties in the CDSS North and Mountains region were excluded for this reason; Butte County was assigned to the Central Valley region, which it borders, for randomisation.) We also excluded five counties where DOJ had recently completed extensive enforcement actions. Finally, we excluded the city of Los Angeles, which had implemented its own intervention similar to APPS. The effect of these exclusions on the study population is shown in online supplemental figure 1.
Measurement
During the trial, APPS intervention in a city typically progressed to completion over a period extending from several days to a few weeks. We created a continuous progress-to-completion (PTC) variable ranging from 0 to 1.0, set to 0 on the study start date (no individuals intervened with) for all group 1 cities and to 1.0 for each city when all interventions delivered in that city prior to its region-specific end date had been delivered. PTC for each city was updated at the end of each month. Progress was often slow at the beginning and end of the period in which a city’s intervention occurred, as the first and last few individuals were contacted. We therefore considered cities to be preimplementation until the end of the month in which PTC reached 0.1 and post implementation at the end of the month in which PTC reached 0.9.
Protocol deviation
A protocol deviation occurred for a group 1 city if interventions did not begin in that city until after they had begun for at least one group 2 city in that same region. Similarly, a protocol deviation occurred for a group 2 city if interventions began in that city before interventions had begun in all group 1 cities in the same region. (See online supplemental figure 2 for additional information.)
Monitoring
Investigators monitored fidelity of implementation through regular meetings with CA DOJ senior staff (biweekly during the intervention period), real-time APPS records monitoring and field observations (ie, ride-alongs). Internet searches and the CA DOJ meetings were used to monitor for contamination (unplanned exposure of a city’s population to the existence of APPS generally or its presence in that city through media publicity), parallel interventions and requests from communities for intervention outside the allocation scheme.
Outcomes
The outcome measures are monthly city-level reported crime counts prepared for the FBI’s UCR programme.23 The primary outcome measures are firearm-related homicides, robberies and aggravated assaults. Secondary outcome measures include those same three crimes where firearms are not involved. They also include violent crimes in aggregate (the categories just mentioned and simple assault) and non-violent crimes (burglary, larceny-theft and motor vehicle theft). Rape was excluded, as a nationwide redefinition for UCR reporting in 2013 produced a major discontinuity in reported counts.
Under UCR coding criteria, firearm-related aggravated assaults are not restricted to crimes in which a firearm is discharged. Crimes charged by local jurisdictions as assault and battery, disorderly conduct and brandishing a firearm, among others, are reported to UCR as aggravated assault.23
Outcome measures also included the proportion of violent crimes committed with firearms, a measure intended to capture possible weapon substitution. This was determined using a binomial generalised linear model with mixed effects in which the outcome events were defined by the UCR crime categories of ‘robbery with a gun’ and ‘assault with a gun’ and the Supplemental Homicide Reports category ‘murder with a gun’. Total events used the UCR categories ‘robbery total’, the sum of (‘assault with hands, feet, etc.’ + ‘assault with a knife’ + ‘assault other weapon’ + ‘assault with a gun’) and ‘murder’, respectively.
Randomisation
Randomisation was performed at the city level in a 1:1 ratio to early (group 1) or deferred (group 2) intervention. Cities were stratified for randomisation by CDSS region and then by whether they were above or below the regional medians for 2010 population and UCR Part I violent crime rate (average for 2009–2011). Randomisation was carried out using PROC SURVEYSELECT in SAS V.9.4 on 14 January 2015.
Allocation concealment
Treatment assignment was not masked to CA DOJ office staff and field teams or to the investigators; cities were not made aware of the study or of their allocation to treatment groups.
Statistical methods
Analyses were conducted in duplicate. We employed a superiority testing framework for all hypotheses and a difference-in-difference approach. Our primary null hypothesis was that any change in monthly crime counts in group 1 cities during the time when those cities received intervention and group 2 cities did not would not differ from changes in group 2 cities during that time period, when counts for both groups were compared with those for the preintervention period—1 January 2010 to 1 February 2015. This hypothesis was tested against the two-sided alternative that there would be such a difference.
We fit unadjusted and adjusted models; unadjusted models included the prerandomisation stratification characteristics (CDSS region, log-transformed population and violent crime rate above or below regional median). All models specified the log of the predicted count as a linear function of the primary predictor (intervention status) and covariates, with random effects for city. Analyses were conducted using generalised linear mixed models. In cases of non-convergence, models using generalised estimating equations were employed. Robust SEs were used to generate CIs (see online supplemental file 1).
Analysis population
Eight small cities (with a combined 2010 population of 21 770) that had been included in the randomisation were excluded from the primary analysis. No residents of these cities were eligible for APPS on the study start date or during the study period. A Consolidated Standards of Reporting Trials (CONSORT) diagram (figure 1) summarises the flow of study cities from assessment for eligibility to inclusion in the analysis.
Consolidated Standards of Reporting Trials flow diagram. *Here and following, the numbers in parentheses represent the number of cities included, the mean population of those cities, and the SD around that mean population. †Three cities (American Canyon, Laguna Woods, Buellton) do not have outcome data.
Adjustment for covariates
Anticipating residual heterogeneity following randomisation, we prespecified and investigated a wide range of covariates. Variables retained in the primary model included the prerandomisation stratification characteristics, log-transformed population in 2014, population density in 2010, year, month, DROS-derived estimated annual prevalence of firearm purchasing at the city level (a small-area proxy for firearm ownership) and a California-specific analogue to the Area Deprivation Index (ADI).24 The last two were included because the prevalence of firearm ownership and community characteristics such as those in the ADI have firmly established associations with rates of firearm violence.25 For the ADI analogue, we developed coefficients for individual ADI variables using principal components factor analysis on data for cities in California and categorised the ADI analogue in deciles.
Sensitivity analyses
Sensitivity analyses examined interactions between the prerandomisation stratification variables, the BRFSS-derived county-level estimate of household firearm ownership, exclusion of cities where no interventions occurred (though eligible individuals resided in those cities) or where intervention constituted a protocol deviation, alternative specifications of intervention start and end dates for individual cities, and exclusion of two outlier cities (see online supplemental file 1).
Secondary analyses
In secondary analyses, the null hypothesis was tested for primary and secondary outcomes at the regional level. Models with region × group interaction terms tested for heterogeneity among regions using p=0.20 as the threshold for significance, noting that this would allow us to gain a greater increase in power than the increase in type-1 error for the amount of heterogeneity we considered important to be able to detect.26 27
Statistical software
Analyses were conducted using SAS V.9.4 (PROC GLIMMIX for generalised linear mixed models and PROC GENMOD for generalised estimating equations) and replicated in R V.4.2.0.
Guidelines
This study is reported in accordance with CONSORT guidelines for randomised trials.28–30
Results
Participant flow
A total of 363 cities were randomised for inclusion: 187 to group 1 (early intervention) and 176 to group 2 (deferred intervention) (figure 1). Among group 1 cities, 94.7% (177) received intervention according to protocol; 69.3% (122) did so among group 2 cities, and the intervention was not discontinued prematurely in any city. Periods of observation were defined as the time in each region when group 1 cities were receiving intervention and group 2 cities were not. Periods of observation began 1 February 2015 in all regions but varied in duration from 2.9 months (Southern Farm region) to 15.9 months (Los Angeles County region) (online supplemental figure 2).
Baseline data
The two groups of cities were well balanced on a wide array of baseline characteristics (table 1), including population size and demographic composition, distribution across ADI deciles, location, violent and non-violent crime rates, prevalence of firearm ownership and purchasing, and proportion of the population eligible for the APPS intervention on the study start date. The proportions eligible were quite low in both groups, with a mean of just over 70 per 100 000 population.
Baseline characteristics of study cities
Numbers analysed
Eleven cities were excluded from the analysis; 180 group 1 cities and 172 group 2 cities were included. Summary measures describing the delivery of the intervention in group 1 cities are in table 2. About 25% of persons eligible for intervention (0.16 persons per 1000 adults in participating cities) had contact with intervention teams.
Measures of intervention as delivered in group 1 cities
Outcomes and estimation
In unadjusted models (online supplemental table 2), allocation to early intervention was not associated with differences in the incidence of most primary and secondary outcomes. Group 1 cities experienced relative increases in firearm-related aggravated assault (IRR 1.27, 95% CI 1.14, 1.42) and in the proportion of aggravated assaults involving firearms (OR 1.34, 95% CI 1.03, 1.75). Unadjusted region-specific models (online supplemental table 3) found no association between allocation to early intervention and most outcome measures; there were increases in firearm-related aggravated assault in the Los Angeles County and Southern California regions, in simple assault in the Southern California region, in all violent crimes in the Bay Area and Southern California regions, and in non-violent crimes in the Los Angeles County region.
In adjusted models, however, no significant differences between group 1 and group 2 cities were observed for any outcome measure (table 3, figure 2, online supplemental figure 2); the estimate for firearm-related aggravated assault (IRR 1.29, 95% CI 0.99, 1.68) was similar to that from the unadjusted model. In region-specific models (online supplemental file 4, figure 2, and online supplemental figure 3), the only significant difference detected was a decrease in firearm robbery in the Central Valley region. Unadjusted and adjusted estimates for firearm-related aggravated assault (online supplemental tables 3,4, respectively) differed substantially for the Bay Area, Central Valley and Southern California regions. Models including a region × group interaction term suggested heterogeneity among regions only for firearm-related aggravated assault (p=0.15) and all non-violent crimes (p=0.11).
Adjusted* incidence rate ratios for crimes by type and firearm involvement and proportion of violent crimes involving a firearm, comparing group 1 and group 2 cities
Combined and region-specific adjusted* incidence rate ratios for firearm-related violent crimes by type, comparing group 1 and group 2 cities. (a) Firearm homicide. (b) Firearm robbery. (c) Firearm aggravated assault. *Models include California Department of Social Services region (combined model only), violent crime rate (above/below regional median), non-violent crime rate (above/below regional median), population density, Dealer’s Record of Sale-based prevalence of firearm ownership, Area Deprivation Index decile, year, month and population (log-transformed). Cities were stratified according to the first three variables for randomisation. Incidence rate ratios were estimated using generalised estimating equations for firearm-related robbery in the Bay Area, Central Valley, Southern Farm and Southern California regions and for firearm-related aggravated assault in the Central Valley, Southern Farm and Los Angeles County regions.
Sensitivity analyses
Findings for sensitivity analyses were similar to those from the adjusted models; most yielded significant increases for firearm-related aggravated assault but not for other primary outcome measures. (Data not shown.)
Additional analyses
A region-specific and group-specific assessment of trends in firearm-related aggravated assault identified pre-existing trends that continued through the observation period and differed significantly between group 1 and group 2 cities in two of three regions (figure 2c, online supplemental table 4) where IRRs exceeded 1.0: the Los Angeles region (upward trends in group 1 and group 2 cities, but more steeply in group 1; group × time interaction p=0.02) and in the Bay Area region (downward trends in group 1 and group 2 cities, but more steeply in group 2; group × time interaction p=0.02) (online supplemental figure 4). No such interaction was seen in the Central Valley or Southern California regions, where IRRs were close to 1.0.
Discussion
APPS seeks to prevent firearm violence through systematic enforcement of laws regulating access to firearms. The subject population is precisely defined: persons who have legally acquired firearms in the past (and have acquisition records in the data used for the programme) and have subsequently become prohibited persons.
This study reported population-level estimates from the intention-to-treat analysis of a cluster-randomised trial of APPS’s short-term effects on firearm-related violent crimes and crimes of other types. In adjusted models, allocation to early intervention was not associated with statistically significant differences for any outcome measure; a non-significant increase in firearm-related aggravated assault achieved statistical significance in unadjusted models. There was suggestion of regional variation, and some estimates for firearm-related aggravated assault varied substantially with adjustment.
At least seven mechanisms, acting individually or together, could account for findings of no effect. One is inherent to the study design. Pragmatic intention-to-treat analyses such as this set the bar high; they assess effects of treatment allocation and not treatment as actually delivered.
The other six are specific to APPS and to this study; items 2–4 in this list, taken together, are likely the most important. The first arose from the legislative mandate for APPS to be implemented statewide urgently; the study’s period of observation (the time between onset of intervention in group 1 cities and onset in group 2) was shorter than expected. This reduced power to detect differences between group 1 and group 2 cities.
Second, only a small percentage of the population was eligible for the intervention. Even substantial reduction in risk in this small population might not produce a change in population measures of crime.
Third and similarly, failing to intervene with all eligible individuals would increase the likelihood of a null finding, and our estimate is that contact was made with 25% of individuals who were APPS eligible at the start of the study. Note that of the remaining 75%, some fraction—and perhaps many of those prohibited as a result of DVROs—would have had their prohibitions expire before APPS intervened in their cities. There are several other possible reasons for the low contact proportion. It was important that no advance notice of contact be given, for example; such notice would have maximised opportunities for prohibited possessors to take action to prevent APPS personnel from recovering their firearms. The study design also played a role. Recognising the implementation difficulties, APPS only closes cases if all known firearms have been recovered or all possible leads have been definitively exhausted. Our study design terminated observation for group 1 cities in any region when group 2 interventions began, with the result that any late contacts were not included. These limitations notwithstanding, if APPS interventions have a durable effect on the individuals directly affected, population-level effects might well emerge over time as the intervened-with proportion of the population increases.
Fourth, APPS does not intervene with some prohibited firearm owners at high risk of interpersonal violence: those who acquired their firearms illegally and whose firearm ownership is undocumented. Absent CA DOJ firearm acquisition records, these high-risk individuals cannot meet APPS’s eligibility criteria. They may constitute the majority of prohibited persons with firearms in California—nationwide, only 10.1% of persons incarcerated for firearm-related violence acquire their firearms from a retail source.31
Fifth and conversely, APPS does intervene with individuals who are likely not at high risk for the study outcomes. About 7% of persons intervened with during the study were prohibited only following hospitalisations for dangerousness arising from mental illness. Severe and acutely exacerbated mental illness is associated more strongly with risk for self-harm and suicide than with risk for commission of interpersonal violence.5
Finally, APPS relies on incomplete firearm ownership data and cannot identify firearm owners who legally purchased handguns only before 1996 or long guns only before 2014. If they subsequently become prohibited persons, they will not be detected by APPS’s screening processes.
The increase in firearm-related aggravated assault associated with allocation to early intervention that we found in some models was unexpected and remains a subject for future exploration. As mentioned, several region-specific estimates varied substantially with adjustment for prespecified covariates. Separately, increases in firearm-related aggravated assault beginning in January 2014 (a year before the study began) in group 1 cities relative to group 2 cities occurred in the Los Angeles County and Bay Area regions, two of the three regions in which non-significant increases were associated with allocation to early intervention in adjusted models. (Striking decreases in firearm homicide and firearm-related aggravated assault in Oakland, a group 2 city in the Bay Area region, have been examined previously.32) Separately, it bears repeating that, under UCR coding criteria, firearm-related aggravated assault may include misdemeanours such as assault and battery and brandishing a firearm. Variation in coding and reporting procedures may also account in part for our findings.
A modelling study has attributed strongly beneficial effect to APPS, estimating that the intervention prevented more than 100 firearm homicides per year from 2007 to 2016.33 Results from this real-world trial are quite different and, in our judgement, cast doubt on the validity of the estimates from that study. Separately, we note that APPS was a pilot programme during most of the intervention period included in the modelling study, conducting fewer than 2000 interventions per year.14 15 We do not consider it likely that such a small programme would have the effect estimated in the modelling study.
To our knowledge, no other randomised trials of firearm recovery interventions exist to provide a basis for comparison with our findings. Ecological and observational studies of extreme risk protection order (ERPO) laws provide the closest parallel. ERPOs have been used hundreds of times in efforts to prevent mass shootings, for example, and where outcome data are available, none of those threatened shootings have occurred.34 35 But mass shootings account for a very small percentage of deaths from firearm violence and could be eliminated completely without producing a detectable decrease in the population-level firearm violence mortality rate.
Individual-level studies suggest that ERPOs have benefits in preventing suicides.36 Population-level studies have yielded mixed effects,37 38 however, and a knowledgeable commentator has suggested that it is too soon for population-level evaluations of ERPOs.39 The same may have been true for this study of APPS, especially given the relatively brief period of observation. Experience with DVRO prohibitions, which have been available for decades, suggests that population-level effects emerge over time.12
In such situations, analyses relying on individual-level data may provide the best estimates of interventions’ effectiveness. Such an analysis is underway for APPS.
Limitations
Our study did not include all cities in the state. Cities in counties that were subject to the intervention shortly before the study began were excluded, as were cities in remote areas and the city of Los Angeles. As mentioned, the relatively brief period of observation prior to onset of APPS interventions in group 2 cities limited our ability to detect intervention effects. Delayed intervention in seven small cities on the relatively remote Central Coast resulted in an unexpected number of group 2 protocol deviations in the Farm and Southern California regions (online supplemental figure 2). Advance or concurrent notice of an intervention team’s presence in a city could have provided an opportunity for individuals to evade intervention, but our surveillance generated no reports of such contamination.
Conclusion
APPS is a unique and promising intervention that seeks to prevent violence by recovering firearms from prohibited persons. This intention-to-treat analysis of a large, pragmatic, cluster randomised trial provides a rigorous assessment of its effects on important population-level outcome measures. There was no clear evidence of benefit at that level. Several concurrent mechanisms may account for this, the most important, in our judgement, being that intervention was completed more quickly than planned for in the treatment (group 1) cities, that only a small part of the population was eligible for intervention, that many eligibles did not receive the intervention and that many persons at risk were not eligible. Further study of the effects of the intervention as delivered, effects on suicide and effects at the individual level is needed before recommendations can be made on replicating the programme in other states.
Data availability statement
No data are available. Data are not currently available as analyses are continuing.
Ethics statements
Patient consent for publication
Ethics approval
This study was approved by the University of California Davis Institutional Review Board (IRB-ID 553213). The University of California, Davis, in accordance with its FWA with the Department of Health & Human Services, adheres to all federal and state regulations related to the protection of human research subjects, including 45 CFR 46 (‘The Common Rule’), 21 CFR 50, 21 CFR 56 for FDA regulated products, and the principles of The Belmont Report and Institutional policies and procedures. In addition, the International Conference on Harmonization, Good Clinical Practice (ICH GCP) principles are adhered to insofar as they parallel the previously mentioned regulations and policies. This specific paper reports on population-level findings; its subjects are cities, and the outcome data are taken from public crime statistics reports. The trial as a whole evaluated a mandated law enforcement intervention (individuals did not have an option on participation). It relied on data collected by the California Department of Justice during the course of that intervention. See the protocol, which was published in Injury Prevention and is available at https://injuryprevention.bmj.com/content/23/5/358, for details.
Acknowledgments
The authors acknowledge with gratitude the support for this project provided by the California Department of Justice. The authors are also grateful for the contributions of Philip H. Kass, PhD, MPVM, MS, PhD, Pamela A. Keach, MS; Rocco Pallin, MPH; Aaron B. Shev, PhD; and Elizabeth A. Tomsich, PhD.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
X @Veronica_A_Pear
Contributors Wintemute: conception and design; acquisition, analysis, and interpretation of data; drafting of manuscript, funding. Tancredi: conception and design; acquisition, analysis, and interpretation of data; substantive revision of manuscript. Pear: conception and design; acquisition, analysis, and interpretation of data; substantive revision of manuscript. Li: conception and design; acquisition, analysis, and interpretation of data; substantive revision of manuscript. McCort: conception and design; acquisition, analysis, and interpretation of data; substantive revision of manuscript. Pierce: conception and design; acquisition, analysis, and interpretation of data; substantive revision of manuscript. Braga: conception and design; acquisition, analysis, and interpretation of data; substantive revision of manuscript. Wright: conception and design; acquisition, analysis, and interpretation of data; substantive revision of manuscript. Laqueur: conception and design; acquisition, analysis, and interpretation of data; substantive revision of manuscript. Kravitz-Wirtz: conception and design; acquisition, analysis, and interpretation of data; substantive revision of manuscript. Studdert: conception and design; acquisition, analysis, and interpretation of data; substantive revision of manuscript. Beckett: conception and design; acquisition, analysis, and interpretation of data; substantive revision of manuscript. Wintemute accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish. All authors agreed to the publication of the work.
Funding This study was funded by grant number 2014-R2-CX-0012 from the National Institute of Justice, by award number 14-6100 from the California Department of Justice, and by the Violence Prevention Research Program and the California Firearm Violence Research Center.
Disclaimer The opinions, findings and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect those of the funders.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.