Article Text
PDF
Abstract
Objective This study investigated the application of the Children’s Safety Network (CSN) Framework for Quality Improvement and Innovation in Child Safety through the Child Safety Learning Collaborative (CSLC).
Methods The CSN Framework was used by 26 state/jurisdiction teams that participated in cohort 1 of the CSLC, from November 2018 to April 2020. The aim was to strengthen child safety systems and the workforce to spread child safety evidence-based and evidence-informed strategies and programmes for children and adolescents ages <1–19 years.
Procedures Participating teams’ child safety system development, workforce development, engagement in the CSLC, challenges encountered and overall satisfaction with the CSLC were assessed through ongoing CSLC participation records and an end-of-cohort survey (survey response rate: 73.1%).
Results Teams showed an average change of 2.4-fold increase in the spread of evidence-based and evidence-informed child safety strategies and programmes, indicating improvement in child safety systems. Knowledge development on CSLC tools and strategies was reported by 77.8% of teams, with 55.5% reporting CSLC tools and strategies contributed to workforce development. Over two-thirds (70.6%) reported being satisfied or very satisfied with the CSLC, but identified some challenges, including staff turnover and the need to strengthen partnerships. All teams demonstrated engagement in the CSLC, based on participation in a virtual meeting, learning session or a monthly report submission.
Conclusions Despite challenges, teams continued to participate in the CSLC, recognising the importance of collaborative learning. The CSN Framework is helpful for state/jurisdiction teams to improve child safety systems and develop their workforce.
- Dissemination
- Implementation / Translation
- Process/impact evaluation
Data availability statement
Data are available on reasonable request. Deidentified data available on reasonable request from the corresponding author.
This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.
Statistics from Altmetric.com
Background
Over 40 000 infants, children and adolescents die each year in the USA, of which one-third are attributable to preventable injuries and violence.1 Injuries and violence are the leading cause of death for children/adolescents ages 1–19, and the third-leading cause of death for infants age <1 in the USA. Approximately 56% of injury deaths are due to unintentional injuries, 21% are suicides, 20% are homicides and 3% are undetermined deaths. Progress has been made in the field of injury and violence prevention and best practices have been identified,2–8 yet the need remains to increase the spread of child safety strategies/programmes. The Children’s Safety Network (CSN) is a national resource centre for state/jurisdiction health departments working to spread (ie, dissemination of best practices across a whole system) child/adolescent safety across state/local communities. CSN coordinates the national Children’s Safety Now Alliance of federal, state, local, hospital and academic agencies/organisations and provides training and technical assistance (TTA) in child safety to Title V agencies (ie, state/jurisdiction departments of health).
CSN framework for quality improvement and innovation in child safety
CSN developed the CSN Framework for Quality Improvement and Innovation in Child Safety (The CSN Framework)9 10 (figure 1), leveraging Foundation Strategy Group’s (FSG) collective impact approach, Institute for Healthcare Improvement’s (IHI) breakthrough series (BTS) and Associates in Process Improvement’s (API) model for improvement (MFI).11–14 The CSN Framework aligns with (1) FSG’s model in having a common agenda/aim, shared measurement, mutually reinforcing activities, continuous communication and a backbone organisation,13 (2) IHI’s BTS model in using structured activities, including learning sessions and action periods,12 14 and (3) API in using the MFI.14 BTS has been used in initiatives to improve healthcare,15 16 mental health,17 18 child welfare19 20 and early detection and intervention services21 22 for children and families. CSN adapted this model to address the complexities of change and collective action at state levels.
Children’s Safety Network Framework for Quality Improvement and Innovation in Child Safety.
Building on these models, the CSN Framework (1) incorporates a prominent component for leadership and management that provides support and resources for state leaders working in complex environments; (2) expands the expert community to include public health practitioners that inform feasibility of spreading evidence-based child safety strategies; (3) adapts the delivery of learning activities by adding blended learning and outputs for workforce development and systems improvement; and (4) adds distal population-level outcomes of improved child safety. The rationale for these adaptations and supporting evidence is described in previous work.10 The CSN Framework, used by the Child Safety Learning Collaborative (CSLC), puts forth consistent application of child safety expertise, leadership and management, and systems improvement methods results in strengthening child safety systems and the workforce, ultimately contributing to reductions in fatalities, hospitalisations and emergency department visits. The CSN Framework was first evaluated for feasibility in 2017 and results demonstrated the CSN Framework is a promising approach for learning collaboratives to achieve statewide child safety system improvement.10
Child Safety Learning Collaborative
Quality improvement learning collaboratives bring teams together to identify, test and spread evidence-based strategies/programmes through a structured process.23 24 While there is evidence on positive effects associated with learning collaboratives,24 literature on the strength of the approach and specific factors that contribute to outcomes remains limited.23 25 One meta-analysis found intact teams, participation, and continued data collection contributed to successful outcomes, while leadership support, time and resources did not.23 In another study participants reported collaborative faculty, quality improvement tools and resources (eg, plan-do-study-act (PDSA) cycles, change packages) and learning sessions advanced improvement efforts and knowledge acquisition.26
The CSLC was developed to bring Title V agencies together in their efforts to improve child safety through state/local improvements in child safety systems (ie, the interplay between policies, procedures, infrastructure, resources and human actions)27 and workforce development (ie, knowledge, skills and competencies of a workforce to carry out public health services, including delivery of effective programmes and interventions).28 CSLC cohort 1 took place from November 2018 to April 2020 with eighteen states/jurisdictions that formed a total of 26 teams. Applications were submitted by state/jurisdiction maternal and child health and/or injury and violence prevention directors and were used to assess applicants’ capacity and readiness to spread evidence-based approaches and quality improvement methods. Although all applications were accepted, variations were observed such that 50% of teams reported readiness to spread strategies. CSLC team membership was selected by states/jurisdictions and generally ranged from 3 to 12 members representing state/jurisdiction departments of health and selected partners (eg, other state agencies, hospitals, schools). Each state/jurisdiction chose to address one to three child safety topics: motor vehicle traffic safety, poisoning prevention, suicide and self-harm prevention, sudden unexpected infant death (SUID) prevention and bullying prevention.
CSLC topics were identified by CSN as those with the greatest impact on injury and violence prevention and most likely to be high priority for Title V agencies based on state/jurisdiction selected national performance measures. Table 1 provides prevalence data for bullying victimisation, fatality rates for CSLC topics, and CSLC team descriptive data. After a review of state action plans, CSLC teams self-selected evidence-based or evidence-informed programmes and practices to spread in their states/jurisdictions from a menu of programmes and practices provided by CSN (table 1).29 Participation expectations were to attend CSN coordinated monthly virtual meetings with teams across the CSLC to receive TTA on the CSN Framework, share challenges, successes and lessons learnt in a collaborative peer setting, and submit monthly data reports. Between virtual meetings, teams met within their state/jurisdiction team, setting their own agendas to advance the work.
Child Safety Learning Collaborative (CSLC) cohort 1 child safety topics and population
Methods
Twenty-six teams across 18 states/jurisdictions participated in cohort 1. One team dropped out due to staffing issues, yielding 96.2% CSLC participation rate.
Data sources
Participation records
CSN tracked attendance in monthly virtual meetings and two learning sessions and submission of monthly reports and PDSAs (ie, a structured approach to develop, test, implement and spread strategies through a planned learning process.30 Table 2 describes participation rates across participation record type. Each team designated a point-of-contact who completed self-reported, web-based monthly reports that collected quantitative and qualitative data. PDSA submissions were web based and described teams’ plans to test strategies. Teams decided when and how many PDSAs to submit, with 42% of teams submitting between one and six PDSAs, averaging two PDSAs per team. The quality of PDSAs submitted was inconsistent and CSN provided TTA each time a team advanced through the stages of a PDSA.
Response rates in participation records and end-of-cohort survey
End-of-cohort 1 survey
Nineteen of the 26 teams completed a self-report, web-based survey at the end of cohort 1 in May/early June 2020 (overall response rate: 73.1%) (table 2). Respondents were similar to non-respondents with respect to group size (respondents: 3–12 members per team vs non-respondents: 4–10 members per team), submission of at least one PDSA (36.8% respondents vs 42.9% non-respondents) and monthly report (100% respondents vs 100% non-respondents) and attendance at a minimum of one meeting (100% respondents vs 100% non-respondents) and one learning session (100% respondents vs 85.7% non-respondents). The survey was sent to the main point-of-contact for each team, who was asked to obtain input from all team members; however, CSN could not determine who contributed to the final responses. The survey contained multiple choice, Likert scale and open-ended questions to assess knowledge development on CSLC tools/strategies, effectiveness of CSLC tools/strategies on workforce development, and other measures, including staff turnover, TTA and overall satisfaction with the CSLC. Missing data were handled through listwise deletion; it ranged from 5.3% to 10.5% (1–2 teams) on workforce development, 10.5% (2 teams) on CSLC satisfaction and 15.8% (3 teams) on open-ended responses to identify challenges.
Narratives from selected teams
Two CSLC teams were invited by CSN to share success stories on child safety system development and workforce development to illustrate progress using the CSN framework.
Measures
Child safety system development was measured by teams’ self-reported progress in spreading strategies/programmes through a web-based monthly report. Teams were provided evidence-based/evidence-informed child safety strategies/programmes and spread measures (table 3).29
Spread of evidence-based and evidence-informed child safety strategies and programmes across teams
Workforce development was assessed using end-of-cohort survey data. Workforce development was measured through (1) knowledge development across CSLC child safety strategies/tools and (2) effectiveness of CSLC strategies/tools to use systems improvement tools/approaches, access and implement child safety evidence-based/evidence-informed strategies, and apply leadership and management practices (table 4). The survey also assessed which CSLC strategies/tools across the CSN framework were most helpful to spread strategies/programmes (table 4). Measures were identified based on the CSN Framework and literature review.10 23 26
Items assessing workforce development of Child Safety Learning Collaborative (CSLC) teams
Engagement was measured by participation in virtual meetings, learning sessions, PDSA submission, monthly report submission and/or requesting TTA from CSN. Engagement data were gathered by CSN in participation records, except TTA, which was self-reported by teams in the survey using yes/no responses. TTA corresponded with the CSN Framework: (1) Child safety expertise: Training models/education/resources to increase knowledge/skills across child safety topics; (2) Leadership and management: building capacity to leverage/sustain partnerships/collaborations and develop/implement organisational policy and (3) Systems improvement: Capacity building to develop child safety systems using quality improvement and implementation science, improve data collection/evaluation and improve practice/programme development to ensure spread of child safety strategies/programmes.
Overall satisfaction and challenges were measured using the end-of-cohort survey. Teams rated satisfaction with the CSLC on a 5-point scale, ranging from 1=very dissatisfied to 5=very satisfied. Teams responded to whether they experienced turnover in membership during cohort 1. Response options were ‘no change,’ ‘leadership changed,’ ‘less than 50% of members changed’ and ‘more than 50% of members changed’. Teams responded to an open-ended question on other factors and challenges that contributed to their work on improving child safety.
Analyses
Descriptive statistics were conducted to assess teams’ child safety system development, workforce development, engagement in the CSLC, staff turnover, TTA and overall satisfaction with the CSLC. Open-ended responses to challenges were coded manually using a focused coding or level-2 coding approach to group themes. A second coder independently coded open-ended responses for reliability.31
Patient and public involvement
There was no patient or public involvement in study design or analysis.
Results
Child safety system development results
Teams demonstrated positive child safety system development, measured as average change of 2.4-fold increase in the spread of child safety strategies/programmes (table 3). The largest positive average change was in poisoning prevention with two teams achieving a 65.7-fold increase. Following this result was an 8.4-fold increase in average change reported by four teams in the number of schools and organisations providing gatekeeper training for suicide prevention. Of note, 4123 parents/caregivers in historically underserved communities received safe sleep education through the work of two teams. Table 5 provides a narrative on child safety system development from a selected state/jurisdiction.
Examples of child safety development and workforce development from two Child Safety Learning Collaborative teams
Workforce development results
Table 4 shows workforce development reported by teams. Knowledge developed on CSLC strategies/tools ranged from 72.2% on improvement monitoring processes and scaling up evidence-driven practices/programmes/policies to 88.9% on systems thinking framework. Effectiveness of CSLC strategies/tools to support workforce development areas ranged from 41.2% for child safety system to 64.7% for programme/project organisation. Teams reported most helpfulness of virtual meetings and learning sessions (83.3%) and topic experts teaching about systems improvement, data collection and measurement strategies, and improvement techniques (72.2%) to spread programmes/strategies. Table 5 provides a narrative on workforce development from a selected state/jurisdiction.
Engagement results
All 26 teams participated in at least one virtual meeting and submitted at least 1 monthly report, 25 teams participated in at least one learning session (table 2). The percent of teams submitting PDSAs ranged from 28.6% to 66.7% (table 2).
All 19 teams completing the end-of-cohort survey received TTA. Most teams requested TTA on partnership/collaboration (78.6 %), followed by data collection/evaluation (60.0%), practice/programme development/implementation (37.5%), education/resources to increase public/stakeholder awareness (33.3%), systems development/implementation (33.3%); training models (25.0%); and organisational policy development/implementation (14.3%).
Overall satisfaction and challenges results
Most teams (70.6%) reported being satisfied or very satisfied with the CSLC. The majority (68.4%) reported turnover in membership over the course of cohort 1: 21.1% reported less than 50% membership change, 36.8% reported more than 50% membership change, and 10.5% reported more than 50% membership change and leadership change. Major challenges were staff turnover and staff capacity (31.3%); difficulty applying strategies, for example, identifying the population of interest (18.8%); weak partnership/collaboration (12.5%) and lack of funding (12.5%).
Discussion
Historically, prevention efforts across child safety areas are not consistently and widely spread. An innovative approach to spreading child safety strategies/programmes and developing the workforce is through the CSLC, which uses the CSN Framework for Quality Improvement and Innovation in Child Safety.10 CSLC cohort 1 findings demonstrate the three key areas of the CSN Framework—(1) leadership and management, (2) child safety expertise and (3) systems improvement—contribute to child safety system improvement and workforce development; and are congruent with the literature that subject matter expertise/identification of evidence-based strategies,14 the capacity to lead32 and implementation science14 are critical for scaling health interventions to large populations.
An important aspect of the CSLC is that all state/jurisdiction Title V agencies are allowed to join regardless of their knowledge, prior experience, infrastructure or resources to spread selected programmes. Accordingly, teams varied in the spread of evidence-based and evidence-informed child safety programmes/strategies. Teams were encouraged to set and modify spread goals as appropriate, but those goals were difficult to measure as teams worked in complex adaptive systems and the majority of teams (76.9%) did not report goals. In accordance with the literature, a majority of teams (77.8%) indicated participation in the learning collaborative aided in developing knowledge on various strategies/tools.25 Knowledge development was highest for systems thinking framework, an approach to understanding complex systems and a critical component of the CSN framework. Half the teams (55.5%) reported CSLC strategies/tools contributed to workforce development, especially related to programme/project organisation. Approximately two-thirds (66.7%) of teams reported CSLC strategies/tools, particularly virtual meetings and learning sessions on systems improvement, were instrumental in helping teams improve their child safety systems. Virtual meetings and learning sessions offer live TTA to teams and opportunities for teams to participate in peer learning, which are critical components of a learning collaborative model.12 TTA provides capacity-building support to teams as they learn and master new skills and pursue improvement.14
Across workforce development areas studied, CSLC strategies/tools were least effective in improving child safety systems. This finding is not surprising given systems are dynamic and transdisciplinary integration of approaches is often non-linear. Identifying and transforming processes to integrate elements in a system includes time delays between taking an action and seeing the effects of that action.14 In this context, teams work in complex systems on population health aims that require multiple resources, ranging from partnerships to subject matter expertise, expertise in data collection and analysis, and health communication and knowledge and behaviour change.32 A sampling of strategies and tools may fall short of what is required to impact such a system during the timeframe of cohort 1. Teams shared challenges with staff turnover, weak partnerships, lack of buy-in from leadership and funding. Needs for funding and support from networks, collaborations, teams and partnerships are commonly noted challenges to effective systems thinking and modelling in public health.25 33 Nonetheless, attrition in CSLC was low, with only one team leaving due to staffing issues.
Limitations
Limitations should be considered when interpreting the findings of this study. Two of three teams focused on bullying prevention did not report data; 33.3% in poisoning prevention; 33.3% in motor vehicle traffic safety; 42.9% in suicide and self-harm prevention; and 14.3% in SUID prevention. Nearly 27% of teams did not respond to the end-of-cohort survey. Staff time and resources were constrained by the COVID-19 pandemic, an example of an unanticipated influence on scaling up evidence-based strategies that requires strong partnership, real time data collection and problem solving to anticipate and mitigate negative impacts.25 34 Data were also self-reported and monitoring full fidelity to evidence-based programmes/practices was not assessed. Further, analysis of factors contributing to differential findings by safety topic (eg, bullying prevention, SUID prevention) and their relationship to spread was considered outside the scope of this study. Teams reported output data on the child safety system development measures (eg, number of schools and organisations providing education and number of individuals receiving education around specific evidence-based practice), but not data on the total number of schools, organisations, or individuals in their respective community/state/jurisdiction. Hence, the proportion of organisations or individuals adapting each spread strategy could not be computed. Further, due to the nature of this descriptive study with a lack of a control group or baseline data from teams, the most effective strategies and the effect size of the CSLC programme could not be determined. Future research on learning collaboratives may incorporate experimental design with a control group. In addition, future research should assess how gains in child safety improvement and workforce development are sustained23 35 and their relationship to health impact at the population level. Finally, whether teams received simultaneous support and TTA from other sources that could have contributed to workforce development and assisted to spread child safety strategies/programmes was not assessed.
Conclusions
This study contributes to the scant literature on the effectiveness of quality improvement learning collaboratives in population health by providing empirical evidence to a conceptual framework for improving child safety at the state level. Findings suggest the CSN Framework is effective in assisting states to improve child safety systems and develop their workforce, even during times of disruption, by engaging in quality improvement processes with peers and receiving supportive regular TTA from child safety, leadership and management, and quality improvement leaders.
Key messages
What is already known on this topic?
Quality improvement learning collaboratives are a widely used approach in healthcare settings, but there is scant literature on the effectiveness of these collaboratives on child safety work at the state level and in community settings.
CSN’s Framework for Quality Improvement and Innovation in Child Safety (The CSN Framework) is a promising approach for increasing the use of evidence-based practices by states/jurisdictions to promote child safety.
What this study adds
The CSN Framework is helpful to quality improvement learning collaborative teams in spreading evidence-based and evidence-informed child safety strategies and programmes across state and local communities.
Even in times of disruption (eg, staff turnover, natural disasters and a pandemic), states/jurisdictions are able to apply the CSN Framework and promote improvements in child safety systems and workforce development.
How this study might affect research, practice and/or policy
This study could inform how programmes design, implement and assess learning collaboratives for state-level complex systems, highlighting areas that contribute to workforce development and improved child safety systems.
Data availability statement
Data are available on reasonable request. Deidentified data available on reasonable request from the corresponding author.
Ethics statements
Patient consent for publication
Ethics approval
This study did not meet the definition of human subjects research under 45 CFR 46.102(f) and therefore was not subject to IRB review.
Acknowledgments
The authors would like to thank the states and jurisdictions who contributed case studies.
References
Footnotes
Correction notice This article has been corrected since it was first published. The open access licence has been updated to CC BY.
Contributors JBL and BA made substantial contributions to the manuscript’s conception, design and writing. JBL is resonsible for the overall content as guarantor. JS-C and MK contributed to the collection and analysis of data. JS-C contributed to the editing of the manuscript. All authors approved this version for publication.
Funding The Children’s Safety Network is supported by the Health Resources and Services Administration (HRSA) of the US Department of Health and Human Services (HHS) under the Child and Adolescent Injury and Violence Prevention Resource Centres Cooperative Agreement (U49MC28422) for US$5 000 000 with 0% financed with non-governmental sources.
Disclaimer The information or content and conclusions are those of the authors and should not be construed as the official position or policy of, nor should any endorsements be inferred by, HRSA, HHS, or the U.S. Government.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.