Background Incorrect use of child restraints is a long-standing problem that increases the risk of injury in crashes. We used user-centred design to develop prototype child restraint instructional materials. The objective of this study was to evaluate these materials in terms of comprehension and errors in the use of child restraints. The relationship between comprehension and errors in use was also explored.
Methods We used a parallel-group randomised controlled trial in a laboratory setting. The intervention group (n=22) were provided with prototype materials and the control group (n=22) with existing instructional materials for the same restraint. Participants installed the restraint in a vehicle buck, secured an appropriately sized mannequin in the restraint and underwent a comprehension test. Our primary outcome was overall correct use, and our secondary outcomes were (1) comprehension score and (2) percent errors in the installation trial.
Results There was 27% more overall correct use (p=0.042) and a higher mean comprehension score in the intervention group (mean 17, 95% CI 16 to 18) compared with the control group (mean 12, 95% CI 10 to 14, p<0.001). The mean error percentage in the control group was 23% (95% CI 16% to 31%) compared with 14% in the intervention group (95% CI 8% to 20%, p=0.056). For every one point increase in comprehension, there was an almost 2% (95% CI −2.7% to −1.0%) reduction in errors (y=45.5–1.87x, p value for slope <0.001).
Conclusions Consumer-driven design of informational materials can increase the correct use of child restraints. Targeting improved comprehension of informational materials may be an effective mechanism for reducing child restraint misuse.
- motor vehicle occupant
- randomised trial
- product modification
Statistics from Altmetric.com
The use of restraints is an effective measure to prevent serious injury among children in passenger vehicles.1 2 However, optimal protection requires children correctly use restraint types that are most appropriate for their age/size.3 While effective interventions for increasing use of appropriate restraints have been identified,4 5 less is known about how to effectively counter incorrect use. The most recently available estimates in Australia indicate that 50% of children are incorrectly restrained,6 7 and similar rates are reported in North America8 9 and Europe.10
Instructions on how to use restraints are routinely supplied with child restraints in instruction booklets, labels on the restraint and increasingly, online videos. Child restraint users frequently report using instructions and labels accompanying restraints,11 12 and many jurisdictions regulate content and format of these through product safety standards. In a recent Australian survey, 90% of parents reported they had read instructions supplied with restraints,13 yet high rates of incorrect use continue.6–10 Studies have also reported a higher likelihood of incorrect use among parents who use available information than those reporting non-use of materials.12 14 This suggests instructional materials in their current form may not be effective in communicating how to use restraints correctly.
More broadly, there is a paucity of high-level evidence around effective interventions for reducing incorrect use; however, it appears that the most promising are those that include hands-on training either in person15 16 or through video17 or remote virtual assistance.18 Child restraint manufacturers are increasingly providing online video material as a supplement to written instructions and labels affixed to their products. However, the content of these supplementary materials are currently not covered by product safety standards, and no universal guidance is available for the development of these materials.
In other areas of health, it is common to involve consumers in the design of instructional materials to ensure comprehension and usability. A model for designing material developed by Sless and Wiseman in 199719 has become the gold standard for the user-centred design of health information materials.20 This model stipulates that users should be involved in every stage of design and testing, from user input in formative stages, through user testing in the optimisation of materials and finally in evaluation. It has been demonstrated to produce effective materials across a number of health disciplines,20 21 and is mandated in the European Union and incorporated in Australian guidelines for medicine labels and leaflets to ensure comprehension and usability.22
Using Sless and Wiseman’s model,19 we have developed a set of prototype instructional materials for correctly using child restraints. The objective of this study was to evaluate the effectiveness of these materials in terms of reducing incorrect use and increasing comprehension of the materials in a pilot laboratory trial. The relationship between comprehension and errors in use was also explored.
Figure 1 summarises the study design.
Participants and setting
Inclusion criteria were 18+ years and conversant in English. Those with a physical ailment precluding the installation of a child restraint were excluded. In the hope of achieving a broad cross section of participants, no further inclusion/exclusion criteria were set. The randomised control design aims to ensure a balance of individual participant factors between groups. Invitations to participate were distributed through electronic mail distribution lists, social media outlets, public noticeboards, community playgroups and through word of mouth. Participants were reimbursed AU$25 for travel expenses.
The study was conducted in a laboratory at Neuroscience Research Australia (NeuRA) in Australia between May 2017 and July 2018.
Intervention and control materials
The prototype materials (intervention) were developed for a single make and model of a commercially available rearward-facing or forward-facing convertible child seat via a multistage user-centred process. This Australian Standards-approved restraint is representative of convertible restraints currently available on the market and was designed for installation using the vehicle seat belt and top tether. The content and layout of draft materials were informed using procedural task analysis, review of human factors, instructional design literature, previous studies23–25 and a series of focus groups with users.26 These were then refined through iterative user testing (Hall et al, unpublished data) to deliver the final prototype materials used as the intervention. User testing was conducted prior to the trial and followed Sless and Wiseman’s model19 whereby groups of the target population were iteratively exposed to the materials, with the subsequent refinement of materials until a predefined criterion of a group average of 80% correct use, and 80% correct comprehension was achieved.
The final prototype intervention materials consisted of an A3 size instruction sheet (297×420 mm/11.7×16.5 inches), a set of four swing tags fixed to the restraint and an online video. The video was accessible via QR codes fixed to the A3 instruction sheet and the swing tags. Participants in the intervention group were provided with an iPad to access videos. The instruction sheet and video content covered the key tasks required to correctly prepare the restraint, install the restraint in the vehicle and secure the child in the restraint. This included correct use of the top tether and seatbelt. The online material included a complete video demonstrating the overall instruction and use process, as well as separate video snippets relevant to key tasks. The content of the swing tags addressed the most common errors in use. Online supplementary appendix 1 contains illustrations of the prototype materials. Control materials were the instructions and labels supplied by the child restraint manufacturer for the same make and model of the rearward-facing/forward-facing convertible child seat. No standard manufacturer instructional video was available for this restraint and the control materials did not include any links to online materials. Neither group had access to the vehicle instruction manual, removing this as a factor in participant performance.
On recruitment, participants completed an online survey to collect demographic and child restraint experience data. Participants then attended the laboratory where they were asked to install a single make and model convertible child restraint in the forward-facing mode in the left rear seat of a vehicle buck and then secure an appropriately sized mannequin within the restraint. The buck consisted of the occupant compartment of a Volvo S60 including driver’s seat (right side with no left front passenger seat), rear seat, luggage compartment and roof. The nearside (left side) doors were removed for ease of access. The same make and model convertible child restraint was used throughout. This was a commonly available restraint that had been used to develop the intervention materials.
On arrival to the laboratory, participants were asked to imagine they were about to take a child on a trip, they needed to install their child’s restraint in the vehicle and only had the information materials supplied to achieve this. Participants were given no time limit and were verbally encouraged to perform to the best of their ability. They were also instructed that the researcher would not be able to provide any guidance.
The restraint was initially set in the ‘as-purchased’ rearward-facing condition, that is, headrest was adjusted to the lowest setting, the stabiliser bar engaged, the recline bar was set to rearward-facing orientation, the tether strap was adjusted to the shortest length and excess tether strap stowed away.
After completing the installation, while participants still had access to the control and intervention materials, participants underwent a comprehension test. To minimise order effect, comprehension tests were conducted for the restraint in a rearward-facing mode. The decision on which mode to undertake the fitting trial and the comprehension test was arbitrary.
The primary outcome was overall correct use, and the secondary outcomes were percent errors observed in the installation trial and total comprehension score.
Overall correct use was scored as a dichotomous outcome, ‘Yes’, no serious error present or ’No’, serious errors were present. An error was defined as misuse not strictly in accordance with the manufacturer instructions. Serious and minor errors are defined in table 1 and this categorisation is the same used previously by Brown et al 27 based on observation of the influence of errors on crash protection during dynamic testing.28 29 As demonstrated in dynamic testing, a single minor error has little impact on crash protection; however, two or more minor errors can have a cumulative effect so that the impact on crash protection approaches that of a serious error.28 For that reason, at least one serious error or two or more minor errors were categorised as a ‘serious error, and one minor error or no errors were coded as ‘no serious error’.
For our secondary outcomes, errors observed in the installation and use trial were calculated for each participant as a percentage of possible errors using a predetermined 20-item checklist (table 1). This is a similar outcome measure used to that reported in studies examining the influence of enhanced labels on errors in use.23
The comprehension assessment was delivered verbally and consisted of 10 items relevant to critical tasks in installation and common errors in use (table 2). Each participant was required to locate information related to each item in the materials and to use the information to provide the correct answer. Participants were scored on the number of items they could locate (subscore 1) and the number of correct answers (subscore 2). As required in the method described by Pander Maat and Lentz,30 we multiplied the subscores together to derive the total comprehension score reflecting both understanding and usability of the materials.
The sample size was estimated for a three-arm parallel group randomised controlled trial and multiple analysis of variance to detect a 10% difference in mean total scores with 80% power. Due to ongoing commercialisation efforts involving the intervention tested in the third arm, only two arms are reported here. Based on this calculation, the sample size was set a priori at 22 in each group.
One researcher (AH) generated a simple randomised allocation sequence using an online programme. Another researcher (CH) recruited participants, sequentially assigned participant numbers and allocated participants to intervention or control group using the predetermined group allocation by participant number.
Researchers assessing installation outcomes were blinded to participant allocation. However, it was not possible to blind comprehension assessment as the control and intervention materials were intrinsic to the assessment. Participants were not told what group they had been allocated to, but were, unavoidably, likely able to identify standard or intervention materials, if they were familiar with child restraints.
All analyses were conducted in IBM SPSS Statistics for Windows, V.25.0.
Sample characteristics were described as percentages for categorical variables, or range, mean, 95% CI and SD for continuous variables. A χ2 test was used to assess differences between groups for overall correct use. Differences in mean percent error score and comprehension score were assessed using independent sample t-tests. The relationship between the mean percent error score and overall comprehension score was explored using linear regression.
Table 3 presents participant characteristics and shows the balance of characteristics between intervention and control groups. In general, the sample tended to be young, highly educated and naïve to child restraint use. Across the sample, 58% of participants reported never transporting children in cars. Note that demographic data are missing for six participants (two control, four intervention) as these participants failed to provide complete data in the pre-data-collection survey.
Across the entire sample, only 12 participants achieved no serious errors or less than two minor errors in use. There were 27% more participants achieving overall correct use in the intervention group (n=9, 41%) compared with the control (n=3, 14%) (c2=4.125, df=1, p=0.042). Single minor errors observed in the ‘no serious errors’ category were ‘harness low on the shoulder’, ‘minor slack in seat belt’, ‘minor slack in harness’, ‘non-stowage of excess top tether strap’ and ‘baby insert left in place’.
Comprehension scores ranged from 4 to 19 from a possible maximum score of 20, with a mean of 12 (95% CI 10 to 14, SD5) in the control group. In the intervention group, comprehension scores ranged from 7 to 20, with a mean of 17 (95% CI 16 to 18, SD3). The mean comprehension score in the intervention group was five points higher than the control group (p<0.001). As shown in figure 2, the distribution of comprehension scores varied between groups. For the intervention group, comprehension scores were positively skewed, while the control group were more evenly spread across the whole.
In the control group, the percent errors ranged from 0% to 55% with a mean of 23% (95% CI 16% to 31%, SD17%). In the intervention group, percent errors ranged from 0% to 45% with a mean of 14% (95% CI 8% to 20%, SD14%). Means between groups were not significantly different (p=0.056). As observed in the comprehension scores, the distributions between groups varied; however, percent errors were negatively skewed in the intervention group (see figure 2).
There was a significant linear relationship between total percent errors and overall comprehension (p<0.001, see figure 3). With every one point increase in comprehension, there was an approximate 2% (95% CI −2.7% to −1.0%) reduction in errors (y=45.5–1.87x, F(1,42)=18.328, p value for slope <0.001, R2=0.3). Adjusting for the trial arm (intervention/control) made little difference to this relationship (y=44.7–1.96x, p value for slope <0.001, R2=0.3).
In this pilot trial, user-driven intervention materials were associated with a 27% higher rate of correct use compared with the control group. Comprehension of key information was significantly higher (five points on a 20-point scale, a 42% increase) in the intervention group than controls. Furthermore, this work has demonstrated, for the first time, a significant linear relationship between comprehension and errors in child restraint use. However, the relatively low R2 for this association (0.3) means comprehension of instruction materials is not the only important factor in correctly using restraints.
Our primary measure of overall serious use tolerated one minor error on the basis of the impact of single and multiple minor errors on crash protection provided by child restraints.28 The objective of interventions to reduce misuse should be to reduce errors that significantly degrade crash protection. From this perspective, the 27% lower rates of incorrect use achieved with the intervention materials would have important implications at a population level.
Overall, we believe the method we used to measure errors is relatively stringent. While using a count of errors aligns with the approach taken in a previous similar study by Klinich et al,23 we assessed more potential errors (20 items compared with 8) than the previous study.23 Scoring each item as correct yes/no is also more stringent than some other approaches used previously. For example, in studies by Rudin-Brown et al 24 and Kramer et al,25 the outcome measure was based on a severity score cut-off, with severity scores calculated for each potential error and based on averaged expert rating. In that approach, a participant achieved correct use of forward-facing restraints, if any or all of the following errors were present: tether strap slack of less than or equal to 4 inches (10 cm), space between child restraint system and vehicle cushion of less than or equal to 1 inch (2.5 cm), chest clip not attached, chest clip too high/low, shoulder harness level too high/low, harness slack up to and including three fingers, shoulder harness strap twists and crotch strap twists. Conversely, in our study, all similar errors were counted as separate errors.
In our secondary analysis, we compared groups on this more stringent count measure but found no significant difference between groups. This may reflect a lack of power.
Results highlight the comprehension of materials as a key target in educational strategies to counter misuse. However, the spread of results presented in figure 3 and R2 value of 0.3 indicate other factors besides comprehension of materials also likely play a role in usage errors. Furthermore, there was a wide variation in comprehension scores among intervention participants that scored among the worst for errors in use. In some participants, the errors may be occurring because of poor comprehension, but in others, full comprehension was not translated into practical application. This study was not designed to examine the impact of other factors inherent to different participants on the effectiveness of the intervention so the mechanisms underlying variations in performance seemingly unrelated to comprehension remain unclear and bear further investigation.
It should be noted that this work was not designed to examine what mechanisms underlay improved performance in participants exposed to the intervention materials. Future work examining this may be useful. Similarly, a process evaluation examining the use of specific intervention components, and an error analysis similar to that presented by Giannakakos et al 17 might be useful to identify ways to further enhance instructional materials and remove residual errors in use.
The sample was not limited to participants with children and this may limit generalisability of results to the broader population of child restraint users. However, the mix of participants with and without previous exposure to child restraints is a strength in terms of assessing the intervention materials. Similarly, the use of only one restraint type may limit generalisability. However, there is no reason to believe that the user-driven process would deliver different results for different restraints.
There was also an inherent order effect where comprehension was always assessed after the installation trial. To address this, comprehension was assessed on items related to the rearward-facing mode while the installation trial used the forward-facing mode; however, this may have introduced a measurement bias. We cannot guarantee that the comprehension relevant to use of the restraint in forward-facing mode was the same as for rearward-facing mode, although similar approaches were used in their development.
This trial occurred in a controlled laboratory environment which may also impact real-world generalisability. On one hand, participants undertook the installation trial in the presence of the researcher, using a mannequin and were not subject to any time or practical barriers to correct use than might occur in the real world. This would likely encourage better performance than might occur in the real world. On the other hand, unfamiliarity with the test buck and lack of motivation that might exist when securing a mannequin versus a real child may have negatively impacted performance. In the real world, the prototype materials would also be presented in conjunction with other information as required by regulatory product standards, for example, the control materials. It is unknown if the effect of the intervention materials would be different if they were presented supplementary to the control materials. To address this, the results should be confirmed in the real world and a field-based randomised control trial is currently underway.
Finally, the distributions presented in figure 2 indicate some non-normality, particularly in data from the intervention group. While the independent sample t-test is fairly robust to non-normality in large samples, it is often argued this is not the case in small samples. To confirm the results presented here, the data were transformed (square root transformation for percent error data and Log10 inverse transformation for the comprehension data). Independent sample t-tests computed on these data achieved results identical to that obtained using the non-transformed data.
The results demonstrate the potential for consumer-driven development of child restraint instructional materials to significantly reduce errors in the use of child restraints and increase comprehension of instructions. Furthermore, better comprehension of materials is associated with reduced errors in use. However, inherent individual user factors not measured here may also impact on this relationship, and this is worthy of further study.
What is already known on the subject
Incorrect use of child restraints is a long-standing and widespread problem with few proven countermeasures. Instructions on how to use restraints correctly are provided routinely by manufacturers. In other areas of health, user-driven information design improves comprehension and usability.
What this study adds
The study demonstrates the efficacy of user-driven designed information to counter incorrect use of child restraints. It also demonstrates the relationship between comprehension of instructions and correct use, identifying comprehension as a key target for interventions designed to reduce incorrect use.
The authors thank Dr Raman Sran for the assistance with data collection.
Contributors ABH conceptualised the intervention, developed the intervention, conceptualised and designed the trial, designed data collection tools, coordinated data collection, collected data, carried out the initial analyses, drafted the initial manuscript, and reviewed and revised the manuscript. CH developed the intervention, collected data, carried out initial analyses, reviewed and revised the manuscript. BA developed the intervention, designed data collection tools, collected data, reviewed and revised the manuscript. LK conceptualised and developed the intervention, conceptualised and designed the trial, provided intellectual input into analyses, reviewed and revised the manuscript. KH developed the intervention, provided intellectual input into data collection tools and analyses, reviewed and revised the manuscript. JC conceptualised the intervention, developed the intervention, conceptualised and designed the trial, reviewed and revised the manuscript. AH conceptualised the intervention, conceptualised and designed the trial, provided expert oversight of analyses, reviewed and revised the manuscript. LEB conceptualised the intervention, developed the intervention, conceptualised and designed the trial, provided intellectual input into data collection tools, reviewed and revised the manuscript. JB conceptualised the intervention, developed the intervention, conceptualised and designed the trial, supervised design of data collection tools and data collection, carried out final analyses, drafted the initial manuscript, and reviewed and revised the manuscript. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.
Funding This study was funded by the Australian National Health and Medical Research Council (APP1124981).
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval The study was approved by the University of New South Wales’ Human Research Ethics Committee (HC17273).
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available on reasonable request.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.