Validation of a home safety questionnaire used in a randomised controlled trial
- Correspondence to: Michael Watson, Division of General Practice, University of Nottingham, Floor 13, Tower Building, University Park, Nottingham NG7 2RD, UK;
Objective: To measure the validity of self reported safety practices from a questionnaire, completed by families participating in a home safety randomised controlled trial.
Methods: The postal questionnaire was used to measure secondary outcomes in a randomised controlled trial. The answers to 26 questions that could be assessed by observation were checked by a home visit. Families were invited to take part in a “home safety check”; they were not told that the visit was part of a validation study. At the time of the visit the researcher was blind to the self reports in the questionnaires.
Results: Sixty four questionnaires were validated by visits to 64 households. Percentage agreement ranged from 58% to 100%. Sensitivity was high (68% or above) for most safety practices. The positive predictive value was also high for most safety practices (78% or above for 15 of the 16 practices).
Conclusions: This study found a fairly high degree of consistency between self reported data and actual observations. The findings from this relatively small study need confirmation from larger studies.
- self report
- home safety
- NPV, negative predictive value
- PPV, positive predictive value
- self report
- home safety
- NPV, negative predictive value
- PPV, positive predictive value
An important issue in injury research is the validity of self reported safety practices.1–3 Surveys of safety practices are frequently used in injury research, and self reported safety is often used as an outcome measure in studies evaluating the effectiveness of preventive interventions. There is concern that self reported safety practices might overestimate safe behaviour.1,2 It is therefore essential to be able to describe how well self reported practice reflects actual safety practice. Relatively few studies in injury research, particularly in the area of home safety have attempted to validate surveys of self reported safety practices. This lack of validation has been for a number of reasons including cost, time available, and feasibility.
One injury prevention area where there have been a number of observational validations carried out over the last 30 years is safety belt use.3–5 Findings in this area suggest that self reports generally overestimate safety belt use. Nelson suggests that the main reason for over reporting of certain behaviours is social desirability.3
Social desirability has been reported in many fields besides injury prevention and can relate to the over reporting of socially desirable behaviour or under reporting of socially undesirable behaviour.6 It is where respondents give the socially desirable response rather than describe what they actually think, believe, or do. With general health surveys, for example smoking and alcohol are often under reported and the amount of exercise people take is over reported.
Another injury prevention area where there have been a few validations carried out is in relation to safer cycling. Schieber and Sacks studying helmet use, carried out a narrative review of a small number of different types of self reports (mail, telephone, show of hands, and paper and pencil) and compared them to direct observations.7 They detailed the advantages and disadvantages of each method and concluded that observational surveys are the best method. They stated that for the other methods social desirability bias would be either “high” or “possibly high”. For observations they stated that there would be “none” of this type of bias. However, they did not discuss what difference it would make if the cyclists knew in advance that they were going to be observed.
In terms of home safety, there have been validation studies that have focused on smoke alarms and a broader study that covered general home safety. In relation to smoke alarms, there is over reporting of functioning ones. One study for example, compared self reports from a telephone survey with home tests and found that 22% of those who believed that they had a working smoke alarm did in fact have non-functional ones.8 Another study found significant differences in self reported rates of functional alarms between a telephone and household survey.9 Moreover, their household survey revealed a disparity between self reported and tested functional status. The results of these validation studies are consistent with operational surveys that have found between 23% and 34% of alarms are not functional when tested.10,11
The broader, general home safety study covered possession and use of safety equipment and safety practices.12 It was a relatively small study, where telephone self reports were validated by 20 home visits. Here, however, the researchers found a high degree of consistency between self reported practices and observed practices in the home,12 but this finding needs confirmation from larger studies.
Accordingly, we set out in this study to validate a postal questionnaire with observations in the home. The study was conducted in deprived areas (Townsend score >0) of Nottingham. The questionnaire was completed by families participating in a large randomised controlled trial of home safety advice and free safety equipment.13 The crucial function of the questionnaire in the trial was to measure secondary outcomes.
The questionnaire which measured outcomes for the trial was in the form of a booklet with 49 questions spread over eight sides, including questions on safety practices, satisfaction with safety equipment, and accident occurrence. Safety practices included the storage of potentially hazardous items and safety equipment possession and use. For example, in relation to smoke alarms questions covered possession, whether they were fitted (attached to some part of the house) and how many were working.
The answers to 26 questions that could be assessed by observation were checked by a home visit. Altogether 2000 families were sent the questionnaire and a random sample of 300 of those that responded to the initial mail out were invited to take part in a “home safety check”. The first 32 families in each of the intervention and control groups who responded to the “home safety check” invite and could be contacted were visited. The visits were organised as soon as possible after the questionnaire was returned; the average length of time between return of questionnaire and the home visit was 37 days. Each family taking part in the “home safety check” received a £5 voucher. At the time of the visit the researcher was blind to the self reports in the questionnaires and the families were not told that the visit was part of a validation study.
The sample size was calculated based on an estimated sensitivity of 0.8. To obtain a 95% confidence interval for the sensitivity of 0.65 to 0.95, and assuming a minimum of 40% of families were observed to have a safe practice for a specific safety practice, then 64 questionnaires required validation.
Data were entered onto an ACCESS database, verified by double entry and analysed using SPSS version 11.0 and StatsDirect. The answers to some questions were combined in order to ascertain whether or not certain practices were safe, giving 16 safety practice outcomes. For example, the kitchen was considered safe in relation to medicines if there was none there or if they were stored at eye level or above or if they were stored at another level in drawers or cupboards that had safety catches or locks. Similar rules were applied to cleaning materials and sharp objects in the bathroom and kitchen. For each safety practice, the sensitivity, specificity, positive (PPV) and negative (NPV) predictive values were calculated and their 95% confidence intervals using the exact Clopper-Pearson method (fig 1).14
Where more families under reported than over reported safe practice, PPV exceeds NPV. Where more families over reported than under reported safe practice NPV exceeds PPV.
In order to find out if there were any significant differences between the 64 families who had home visits and the rest of the families (n=1457) that completed questionnaires, comparisons were made of their self reported safety practices. The data were analysed using χ2 tests and by calculating odds ratio and 95% confidence intervals for each safety practice.
The response to the initial mail out of the 2000 questionnaires was 57% and the final response was 77%. The response to the 300 invitations for the “home safety checks” was 32%. The total number of households visited was 64. Table 1 shows sensitivity, specificity, PPV, and NPV and percentage agreement between the questionnaire and observations. The PPV was high for most safety practices (78% or above for 15 of the 16 practices). For five safety practices a higher proportion of families over reported than under reported safe practice (NPV exceeds PPV). For 10 practices a higher proportion of families under reported than over reported safe practice (PPV exceeds NPV).
Table 2 shows that for most of the safety practices there appears to be no significant difference between the questionnaire responses of the home safety check families and the rest of the families who returned the questionnaire. However, in relation to safety on the stairs, home check families were more likely to report that they either had no stairs or had one or more stairgates.
This research has provided information about the validity of self reported safety practices from a home safety questionnaire and highlights which questions could be used in the future. This study found that self reported safety practices were good at predicting observed safe practice for the majority of practices. Where discrepancies occurred between self reported and observed practices, under reporting of safety occurred more commonly than over reporting, which in terms of using self reported practices as outcome measures is encouraging. Over reporting of safety occurred most commonly with storage of sharp objects in the kitchen, possession of window safety catches and fireguards.
Disagreements between the questionnaire and the home visits may have been due to a number of reasons including movement of hazards or safety equipment such as sharp objects or fireguards between completion of the questionnaire and the home visit or providing socially desirable responses such as stating possession of a fireguard but not having fitted the fireguard, or stating place of storage of sharp objects, not where they are left while, or after, being used. The discrepancy in responses to possession of window safety catches became clear to us at the observations as families found it difficult to distinguish between handles, security locks, and catches to limit the width of opening. Further work is required to devise improved questions in this area.
An important feature of our method that needs highlighting is that we kept the time period between the respondents completing the questionnaire and the home visits as short as was practical. This could be seen as a strength of the design. However, choosing from the people who responded to the initial mailing of the questionnaire could also lead to bias. Respondents who reply early may be more organised, more efficient, and may also be so in the application of safety practices in their homes. Our study shows that most self reported safety practices do not appear to differ between the early responders and the rest of the group that responded.
Another point that has implications for the generalisability of our findings is that our study only dealt with responders. The published literature suggests non-responders differ systematically from responders in terms of sociodemographic characteristics, ethnicity, and interest in the topic covered in the questionnaire.15–28 In addition, there also appears to be differences in the reporting of safety practices.29
Although our study is fairly small, which led to wide confidence intervals, it was larger than the previous general home safety validation study.12 Although our study is in agreement with the other general home safety study,12 it is out of line with the single issue smoke alarm studies.8,9 The fact that in our questionnaire the smoke alarm questions were but a part of a general injury questionnaire might be one reason for the difference. It may also be due to the different research methods used.
In the field of injury prevention, the main concern in relation to the validity of self reported information has been the potential for over reporting. Our finding of more under reporting than over reporting is encouraging, but requires confirmation from larger studies.
Copies of the questionnaire used in this study are available from.
An important issue in injury research is the validity of self reported data.
Few studies have attempted to validate self reported home safety practices.
This study found a fairly high degree of consistency between self reported data and actual observations.
Further, larger studies are necessary to confirm the findings.
Our thanks to Trent National Health Service Executive who funded this study.