Article Text

Implementation fidelity of the ‘Stay One Step Ahead’ home safety intervention: a mixed-methods analysis
  1. Sabrina Stewart1,
  2. Denise Kendrick1,
  3. Michael Craig Watson2,
  4. Mike Hayes3,
  5. Elizabeth Orton1
  1. 1 Centre for Academic Primary Care, Unit of Lifespan and Population Health, School of Medicine, University of Nottingham, Nottingham, UK
  2. 2 Institute of Health Promotion and Education, Lichfield, UK
  3. 3 Child Accident Prevention Trust, London, UK
  1. Correspondence to Dr Elizabeth Orton, Centre for Academic Primary Care, Unit of Lifespan and Population Health, School of Medicine, University of Nottingham, Nottingham NG7 2RD, UK; elizabeth.orton{at}nottingham.ac.uk

Abstract

Objective To assess implementation fidelity of the Stay One Step Ahead (SOSA), a complex intervention which was delivered by health visiting teams, children’s centres, and family mentors and was aimed at preventing unintentional home injuries in children under 5 in disadvantaged communities.

Study design A mixed-methods evaluation of the implementation fidelity of the SOSA intervention.

Methods A conceptual framework for implementation fidelity was used to triangulate data from questionnaires and semistructured interviews with parents and practitioners, observations of parent and practitioner contacts, and meeting documents. Quantitative data were analysed using logistic regression and descriptive statistics. Thematic analysis was used for qualitative data.

Results Parents in intervention wards were more likely to receive home safety advice from a practitioner than those living in matched control wards. Monthly safety messages and family mentor home safety activities were delivered with greater fidelity than other intervention components. Content most frequently adapted included the home safety checklist used by health visiting teams, and safety weeks delivered at children’s centres.

Conclusion Consistent with similarly complex interventions, SOSA was delivered with variable fidelity in a challenging environment. The findings add to the body of evidence on implementation fidelity of home injury prevention programmes, providing important information for future intervention development and delivery.

  • Implementation / Translation
  • Child
  • Passive Safety
  • Home

Data availability statement

No data are available.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

WHAT IS ALREADY KNOWN ON THIS TOPIC

  • The Stay One Step Ahead (SOSA) multicomponent home safety intervention has been shown to improve parental home safety practices. Translating effective interventions into ‘real-world’ delivery can be improved if the fidelity of intervention delivery is understood. However, there is a general lack of information available regarding factors that affect the fidelity of child home safety programme delivery.

WHAT THIS STUDY ADDS

  • This study shows variable adherence to the SOSA intervention by practitioner groups and components are often adapted to fit local practices, resource and capacity demands. Fidelity is higher where activity is more highly prescribed and built into contracted delivery.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

  • The SOSA intervention is an effective programme that improves parental home safety practices and should be considered by health and care commissioners in areas of high disadvantage. However, programme specification, quality assurance and improvement approaches should be embedded into programme delivery to ensure fidelity to the tested intervention.

Introduction

Unintentional injuries in children under 5 are common, occur most often at home1 and can result in significant morbidity and mortality.2 Injury rates vary significantly with those living in disadvantaged circumstances most likely to be affected.3–6 Other factors associated with childhood injuries include living in a single-parent household and having older siblings.4 7 A 2012 Cochrane review demonstrated that education and provision of safety equipment are effective in improving home safety practices8 and national guidelines recommend home safety assessments.9 Home visiting programmes aimed at improving a range of parental and child health outcomes, including those delivered by lay workers, reduce child injury rates10 11 suggesting an important role of community-based practitioners supporting parents around home safety.12 Translating this evidence into effective home safety programmes for at-risk children and their parents remains a challenge.13–15

Fidelity is a key moderator of how likely an intervention is to achieve its intended outcomes.16 Fidelity can be defined as ‘a determination of how well the programme is being implemented in comparison with the original programme design’.17 The Medical Research Council emphasises the importance of assessing the quality and fidelity of implementation as part of conducting and reporting evaluations of complex interventions to improve the success of interventions ‘in the real world’.18 Fidelity assessments enable intervention replication,18 19 particularly in non-research settings.20 However, in a systematic review of public health behavioural interventions, only 44% of studies included a fidelity assessment when reporting intervention outcomes.20 Few programmes have assessed implementation fidelity specific to child home injury prevention.21 The aim of this study was to describe the implementation fidelity of the Stay One Step Ahead (SOSA) home safety intervention.

Methods

The intervention

The SOSA intervention is a multifaceted programme delivering evidence-based strategies for home injury prevention to parents of children aged under 5 by three community practitioner groups: family peer mentors (FM) (community members with experience of parenting), Children’s Public Health nurses in a 0–19 Nursing Service (health visiting team, HVT) and children’s centres staff (CC staff). Shared intervention components included signposting parents to local resources (eg, the Fire and Rescue Service for home fire risk assessments and charitable organisations for safety equipment) and the distribution of monthly safety messages (MSMs) on common hazards encountered within the home (table 1, online supplemental table S1). Practitioner specific components included practical safety activities contained in the FM manual, a home safety checklist for HVT members to use at routine child health reviews and to guide postaccident contacts, and home safety sessions delivered during safety weeks at CC.22 Further details of the intervention are given in the published protocol.23

Supplemental material

Table 1

Overview of SOSA intervention components

Patient and public involvement

SOSA was coproduced with ‘Parent Champions’ who were parents of young children, residents of the intervention wards, and part of the wider child health programme Small Steps Big Changes. Champions helped to develop parent recruitment and retention strategies and design data collection tools. They were also part of study oversight and dissemination.

Data collection

We used the conceptual framework for implementation fidelity proposed by Carroll et al 19 to evaluate the delivery of the SOSA intervention, focusing on adherence to the content of intervention components, the dose (frequency and duration of delivery) and reach (coverage) (figure 1). This conceptual framework was developed following a critical review of the literature on factors affecting implementation fidelity and helps researchers explore the relationship between components of fidelity. It has also been used in studies of complex interventions to promote home modification for safety improvement,24 25 fire safety promotion in CC21 and implementation of home-based parenting programmes.26

Figure 1

Fidelity assessment framework based on the conceptual framework for implementation fidelity.

Interviews

A purposeful sample of parents and practitioners from control and intervention wards participated in semistructured interviews, conducted face-to-face or by telephone. Potential participants were provided with detailed study information sheets explaining what was involved in interview participation and their right to withdraw at any time, prior to providing informed verbal or written consent for interview. The interview guide covered the experience of being trained in and delivering the SOSA intervention, including how and when resources were used, challenges to delivery, and how home safety promotion was received by parents. Interviews were digitally recorded and transcribed verbatim.

Activity data

Anonymised safety week attendance data were provided by CC throughout the study period. Aggregated electoral ward-level data on home safety checklist use by HVT members was collected from the children’s electronic medical records between September 2017 and 2020.

Quality assurance and child health review observations

Quality assurance (QA) observations of FM home visits by team managers took place throughout the intervention until March 2020 when home visits stopped due to COVID-19 social restrictions. Team managers used a proforma to assess fidelity to the FM manual activities and use of goal setting techniques. Informed verbal or written consent was gained from parents and practitioners prior to all observations and following provision of a participant information sheet explaining what was involved in the observations and their right to withdraw at any time. Researchers undertook observations of child health reviews conducted by HVT members assessing adherence to the SOSA home safety checklist, use of behaviour change techniques, use of intervention resources and signposting to home safety support.

Questionnaires

Questionnaires were completed by intervention and control ward parents at baseline (study recruitment), and at 12 and 24 months follow-up. Return on the questionnaires was incentivised with vouchers. Intervention practitioners completed questionnaires at 8 and 24 months follow-up and CC staff and HVT members from control wards completed questionnaires at the time of their recruitment to the study, and at 8 months follow-up.

Meeting documents

We analysed agreed minutes of meetings between the SOSA research team members, practitioners, practitioner managers and home safety champions as well as SOSA Steering Group meetings (involving SOSA research team members, intervention ward parent champions, practitioner team managers, the Fire and Rescue Service and managers from Small Steps Big Changes which was the wider organisation housing the intervention).

Data analysis

We used a mixed-methods approach to triangulate data from quantitative and qualitative sources (online supplemental table S2).

Qualitative Sources

Thematic analysis27 was performed for interviews, meetings and observations. Transcripts and other data sources were coded with a priori themes developed using the Conceptual Framework for Implementation Fidelity.19 Coding was undertaken by a researcher (SS) and independently validated by a second researcher (EO). The qualitative data management software, NVivo V.12.0, was used.28

Quantitative Sources

Data from parent questionnaires regarding receipt of home safety advice and resources from intervention practitioners were analysed using multilevel logistic regression models to quantify the reach and dose of the intervention and allow for clustering at ward level. Regression models controlled for baseline receipt of home safety advice and resources, matched wards, deprivation index, number of siblings, maternal age and whether the family was a single parent household. Practitioner questionnaire responses are reported as descriptive statistics. All quantitative analyses were conducted by using Stata V.16.29

Results

A total of 24 parents were interviewed: 12 each from control and intervention wards, and 29 practitioners: 9 FMs, 7 control HVT members, 7 intervention HVT members and 6 CC staff who worked in centres across intervention and control wards. A total of 22 QA observations of FM home visits and 5 observations of HVT-led child health reviews took place. Questionnaire responses were received from 537/720 (75%) parents at 12 months follow-up and 530/684 (77%) parents at 24 months follow-up. Questionnaire responses were received from 36/48 (75%) FMs at 8 months follow-up and 51/52 (98%) at 24 months follow-up, 29/55 (53%) intervention HVT members at 8 months follow-up, and 24/34 (71%) at 24 months follow-up, and 4/9 (44%) intervention CC staff at both 8 and 24 months follow-up. In control wards, 16/22 (73%) HVT members and 5/11 (45%) CC staff responded at 8 months follow-up.

Adherence to the SOSA intervention: shared intervention components

Provision of evidence-based home safety advice

Throughout the intervention period, HVT members in control and intervention wards reported always providing home safety advice during child health reviews with similar frequency, 88% (14/16) and 86% (25/29), respectively. This was consistent with data from intervention and control ward parents who reported a similar frequency of receiving home safety advice from HVT members across the study duration (table 2). At 8 months follow-up, HVT members from control and intervention wards reported discussing home safety with parents for a similar duration with a mean of 9.3 min (SD 3.8) and 9.6 min (SD 3.3), respectively, during 9–12 month reviews, and 9.3 min (SD 3.8), and 10 min (SD 3.4), respectively, during 2–2.5 year reviews. Time spent by intervention HVT members discussing home safety increased at 24 months follow-up with a mean of 12.5 min (SD 10.9) during 9–12 month reviews, and 13.1 min (SD 11.6) during 2–2.5 year reviews. Control HVT members did not complete questionnaires at this time point.

Table 2

Parent self-reported receipt of home safety advice from a practitioner source within the previous year at 12 and 24 months follow-up

At 8 months follow-up, 31/36 (86%) of FMs reported discussing home safety at every visit dedicated to safety, with 7 (23%, 7/31), discussing home safety at every visit irrespective of the visit purpose. A total of 73% (73/100) of parents with FMs reported receiving home safety advice from their FM.

At 12 months follow-up, parents living in intervention wards were significantly more likely to receive home safety advice from CC staff than those in control wards (p=0.02), however, this effect was not significant at 24 months follow-up (table 2).

Parents in intervention wards were significantly more likely to receive home safety advice from any practitioner source, namely CC staff, HVTs or FMs, and from two or more of these sources than parents living in control wards at both 12 and 24 months follow-up (table 2).

Monthly safety messages

Overall, 75% (38/51) of FMs, 50% (2/4) of CC staff and 38% (9/24) of HVT members reported using one or more MSMs. Intervention practitioners most frequently used MSMs to prompt discussion with parents in group sessions, child health reviews or on home visits, with a minority providing the MSM without discussion for parents to review in their own time (online supplemental table S3). In interviews, both approaches were described.

On home visits I will say this is the message of this month and we will talk about it and then I will leave them with that leaflet to look at. FM Interview

When we’re in a session and I literally go around to each individual parent, give them the information, while the children are playing, try and get them to fill [quizzes] in… those who get any wrong I do take to one side and have a proper discussion with them and find out why they have chosen the wrong answer and then tell them which is actually the right answer.” Intervention CC staff interview

I don’t go through [monthly safety messages] with them, I just leave them for parents with the erm RoSPA growth chart. Intervention HVT member interview

Parents in intervention wards were no more likely to have received home safety leaflets than parents in control wards at 12 months follow-up (p=0.50) (table 3) but were more likely to have received them at 24 months follow-up (p=0.02, table 4). In interviews, some parents recalled receiving leaflets pertaining to home safety but could not remember specific details to identify whether these were MSMs.

Table 3

Parents self-reported receipt of resources or support pertaining to home safety within the previous year at 12 months follow-up

Table 4

Parents self-reported receipt of resources or support pertaining to home safety within the previous year at 24 months follow-up

Signposting to support organisations

Intervention practitioners were more likely to signpost parents to organisations for home safety advice or resources with 90% (26/29) HVT members, 100% (4/4) CC staff and 86% (31/36) FMs signposting to one or more organisations. By contrast, 19% (3/16) HVT members and 1/5 (20%) CC staff in control wards reported signposting. Of the practitioners interviewed, most had signposted parents although it was not done routinely. No significant difference was reported by parents in intervention and control wards in signposting at 12 or 24 months follow-up (online supplemental table S4).

Adherence to the SOSA intervention: practitioner-specific intervention components

Health visiting teams

HVT members in intervention wards documented high usage rates of the SOSA Home Safety Checklist in electronic child health medical records (used at >80% child health reviews), except during quarters 2 and 3 of 2020 when most home visits were cancelled during the COVID-19 pandemic (figure 2). In questionnaires at 8 months follow-up, only 62% (18/29) reported using these checklists often or always, and 63% (15/24) at 24 months follow-up. Intervention HVT members received a pack containing supplementary education materials which were used with variable frequency: 72% consistently used a one-page height chart developed by Royal Society for the Prevention of Accidents (RoSPA), whereas 25% reported regularly using RoSPA’s ‘Keep Me Safe at Home booklet’.

Figure 2

Electronic medical record data of self-reported SOSA home safety checklist use at child health reviews by health visiting team members. SOSA, Stay One Step Ahead.

In observations of child health reviews, meeting minutes and interviews, HVT members’ use of the checklist varied. Some practitioners left the checklist with the parent to review in their own time whereas for others, it formed the basis of a verbal discussion:

[HVT members] are sometimes running out of time to do [the checklist] and are asking parents to complete it themselves. SOSA Home Safety Champions meeting

Yes, we obviously didn’t fill [the checklist] in, and I didn’t sign it I don’t think… but …I do remember reading through it…it was more of a verbal thing like what to look out for and if it was like a plan, I would probably say it was verbal, I didn’t write anything down. Intervention parent interview

In questionnaires, parents in intervention and control wards reported a similar frequency of completing a home safety checklist with an HVT member or CC staff (tables 3 and 4). Although the SOSA home safety checklist was only available in intervention wards, at the time of the SOSA intervention, HVTs were using checklists from other schemes to support child health outcomes. HVTs reported that postaccident contacts were conducted without reference to the guidelines developed as part of the intervention protocol, or use of intervention resources.

Face to face post-accident contacts using our home safety checklist are very rare. None of the champions have done any. SOSA Home Safety Champions meeting

Family mentors

All parents in intervention wards were offered the support of an FM, and 45% (100/222, ascertained at 12 months follow-up) of parents in the study accepted this offer. At 8 months follow-up, 24/36 (67%) FMs reported using at least three quarters of the home safety activities from the manual, rising to 38/51 (75%) at 24 months follow-up.

QA visits undertaken by the FMs’ managers found that during all 22 QA observations conducted, FMs used the home safety activities from their manual.

FM talked about making a fire escape plan with Mum, and as they pinpointed possible risks together FM wrote on the plan. FM QA observation visit 29

In interviews, parents recalled home safety discussions with FMs and participating in activities pertaining to home safety.

[My Family Mentor] has given me leaflets before and I have done little quizzes with her. Intervention Parent Interview

CC staff

Data on parent attendance at safety weeks were often incomplete (online supplemental table S5), but attendance appeared to vary considerably between centres and safety weeks. Some CC staff reported that the weeks had limited reach, which was echoed by parents.

The parents that we really wanted … to see in [children’s centres] they hardly attended… and a lot of the parents didn’t really go out to the groups…they weren’t able to engage in these really good activities about home safety because they did not come. Intervention CC staff interview

The majority of times and things that is on [at children’s centres] I am at work and then I always tend to find everything is on in the morning. Intervention parent interview

Safety weeks were consistently delivered but usually took the form of one or two sessions rather than activities throughout the week. In questionnaires, half (2/4) of CC staff reported using injury prevention briefing activities. In interviews, staff confused these activities with MSMs or were unaware of either resource.

Interviewer: Which activities from the injury prevention briefing have been used with parents at [Children’s Centre] over the last two years?

Interviewee: I have done [Monthly Safety Message] 21 this time, ‘trying to prevent strangulation from pull cords’, I did 20 on poisoning the last time and I think I did 15 the time before.” Intervention CC staff interview

Discussion

Main findings

Parents in intervention wards were more likely to receive home safety advice from a practitioner than those living in control wards at both 12 and 24 months follow-up. Adherence to the SOSA intervention contents varied with some components being delivered consistently, while others were adapted or used infrequently. The FM manual activities and MSMs were delivered with greater fidelity than other intervention components, although only 45% of parents in the study had accepted the support of an FM, limiting the reach of these practitioners and the resources they delivered. There was variation in the use of other physical resources, such as the home safety checklist, which was distributed to parents consistently but not always used as intended. The reach of activities at CCs was limited.

Strengths and limitations

The use of mixed methods and a range of data sources enabled a comprehensive triangulation of fidelity, comparing what parents and practitioners recorded and recalled. In addition, quantitative data from parents and practitioners usually converged. The assessment of adherence to intervention content through direct observation of child health reviews and FM visits provided insight to discussions of home safety but in the case of child health reviews was limited to a small number of observations, curtailed by the COVID-19 pandemic. There were lower questionnaire response rates for intervention HVT members and all CC staff compared with FMs limiting the generalisation of these results. The context of the intervention delivery included the COVID-19 pandemic, which restricted parent contact opportunities. In addition, there was also significant restructuring within CC and the Public Health Nursing service (the service within which HVTs are based) and high staff attrition rates from these teams. Such changes meant not all practitioners trained in the intervention were involved in its delivery, and new staff did not always receive training. In addition, staff turnover, service restructuring and the COVID-19 pandemic may provide some explanation for low follow-up questionnaire response rates among HVT and CC staff.

Although we have attempted to measure provision and receipt of home safety information, assessing the depth and quality of such information remains a challenge given the limited observation data available. However, data on parent home safety practices reported elsewhere shows intervention ward parents undertook more safety practices than control ward parents (in press) suggesting that information delivered by practitioners was effective in achieving such changes.

It was beyond the scope of this paper to explore factors which may moderate the fidelity of the SOSA intervention. Findings from this analysis will be presented elsewhere.

Comparisons with previous research

Our study demonstrates that the SOSA intervention was implemented with variable fidelity. To the authors’ knowledge, there are two studies analysing the fidelity of implementation of child unintentional injury prevention programmes, only one of which pertains to home safety,21 the other addressing road safety.30 Both of these studies demonstrated the intervention was delivered with a high degree of fidelity. The first study found that 75% (18/24) of CC implemented injury prevention briefing activities with high fidelity.21 The second study, the Buckle Up Safely programme, found all 13 CC in the intervention arm delivered a parent education session according to the intervention manual.30 Similar to the SOSA intervention, attendance varied considerably between sites with 6/13 centres reaching 10 or fewer parents at their session.30 Heterogeneity in intervention design and complexity may explain the difference in fidelity between these studies and ours. Both studies delivered their interventions within one setting only, involved one set of practitioners and were composed of fewer intervention components than SOSA. The intervention in the first study was delivered over a 12-month period as compared with 24 months, and some of the CC who achieved high fidelity received more intensive facilitation than the SOSA intervention. Furthermore, fidelity was assessed using provider activity logs and interviews whereas our study included observations and parent reports of receiving support and resources. In the second study, the intervention was delivered to parents in a single session and for those intervention components delivered with lower fidelity, information was not provided on whether these were core components of the intervention. Direct comparison with these studies is challenging due to differences in the interventions provided and the context within which interventions were provided.

Implications for policy, practice and future research

The challenges of translating evidence-based public health initiatives into practice are well known.13–15 Our study emphasises the importance of including a fidelity assessment in programme evaluations to ensure interventions are delivered as intended and effects are attributable to the intervention administered. Our findings also highlight the need for ongoing monitoring of the fidelity of intervention delivery when programmes are rolled out into routine service provision to ensure research findings translate to practice.

Data availability statement

No data are available.

Ethics statements

Patient consent for publication

Ethics approval

This study involves human participants and was approved by East Midlands—Leicester Central Research Ethics Committee, reference 17/EM/0240. Participants gave informed consent to participate in the study before taking part.

Acknowledgments

We would like to thank the Small Steps Big Changes Parent Champions (a team of parent volunteers) for their invaluable input into this project and the parents and practitioners who participated in the project, and Dr Michael Taylor for support with the statistical analysis.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Presented at Part of this study has been presented at the World Safety and Injury Prevention Conference, Adelaide, Australia, 2022. Orton E, Taylor M, Jones M, et al. 210 The ‘stay-one-step-ahead’ child home safety programme; effectiveness and cost-effectiveness. Injury Prevention 2022;28:A33. Published here https://injuryprevention.bmj.com/content/28/Suppl_2/A33.1

  • Contributors EO, DK, MCW and MH obtained funding for the study and designed the main study methods and data collection tools. EO, DK and SS planned and conducted the data analysis. SS and EO drafted the manuscript with revisions additionally from DK, MCW and MH. EO is the guarantor for the study.

  • Funding This project was supported by funding from Small Steps Big Changes, part of The National Lottery Community Fund A Better Start Programme. Award number: not applicable.

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.