METHODS

Data

Data for this study were from a pediatric continuity clinic in a large, urban teaching hospital. Thirty one pediatric residents were randomly assigned to an intervention or control group. Families with infants from birth to 6 months were enrolled in the study (n=196) and assigned to the same study group as their pediatrician. Residents providing care to the families in the intervention group had been trained to provide enhanced anticipatory guidance about home safety at each well child visit. Residents providing care to the families in the control group did not receive the training and they provided routine anticipatory guidance to their families. Following The Injury Prevention Program of the American Academy of Pediatrics, routine anticipatory guidance should include discussion of all of the safety practices we evaluated. Follow up interviews were completed when the baby was between 12 and 18 months old, and home observations occurred two to four weeks after the interview. Families who completed both the follow up interview and home observation (n=136) are included in this analysis. Details of the intervention trial can be found in Gielen et al.⁶ The study was approved by the hospital’s institutional review board.

Measures of safety practices

In the follow up interview, the following four questions with dichotomous answer options (yes/no) were asked: (1) Do you have a working smoke alarm?; (2) Do you have a working smoke alarm on each floor?; (3) Do you have a stair gate or doors that block stairs?; and (4) Do you have syrup of ipecac? As part of the informed consent process, all participants were told that a home observation visit would be made at the end of the study.

During the home observations, the locations of smoke alarms were recorded. All were battery operated and were tested. Observers also recorded whether there were stairs in the home, and if so, whether there was a stair gate or a door at the top and bottom of every set of stairs. Parents were asked if they had syrup of ipecac, and if so, the observer recorded its expiration date. The criteria used to be counted as “safe” for each observed practice were having all stairs protected with gate or door, and having at least one unexpired bottle of syrup of ipecac. For smoke alarms two measures were constructed for “safe”, consistent with the two items asked: having any working smoke alarm, and having a working smoke alarm on each floor. The batteries of the smoke alarms were tested by the observers to confirm their status.

Analysis

The observation data are treated as the gold standard (that is, true safe, true unsafe). Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated separately for the intervention and control groups.⁸ Sensitivity of the self reported safety practice is calculated as the percent of all true safe cases that were reported as safe. Specificity of the self reported safety practice is calculated as the percent of all true unsafe cases that were reported as unsafe. PPV is calculated as the percent of self reported safe cases that were true safe cases. NPV is calculated as the percent of self reported unsafe cases that were true unsafe cases.

RESULTS

Characteristics of study parents

Of those enrolled in the study, 80 (67%) of the intervention group and 56 (74%) of the control group completed the home observation. Parents in both study groups were most often the child’s mother (96%) and were generally young (mean age 26 years). Ninety two percent of the parents were African Americans. Few of the mothers were employed (29% intervention group, 38% control group) or married (15% and 20% respectively), and approximately one third had less than a high school education (40% and 29% respectively) at the time of follow up interview. More than one third of families had a household income of less than $5000 (46% intervention group, 28% control group).

Prevalence of observed safety practices

Smoke alarms were observed in most homes (96% in both intervention and control groups); however, when tested by the observer about half of them were functional (59% in intervention group and 50% in control group). Smoke alarm prevalence was even lower when we defined safe as “having a working smoke alarm on each floor” of the home; 22% of the families in both study groups met that definition of safe. Nearly one third of the families had protected their stairs by either stair gates or doors (31% in intervention group and 30% in control group). Of the four safety practices observed, prevalence rates for syrup of ipecac differed the most between the two groups: 14% of the intervention group and 40% of the control group were observed to have syrup of ipecac.

Sensitivity and specificity

Sensitivities were high for both groups across the four safety practices. The highest sensitivity was observed for presence of any working smoke alarm; 100% in both intervention and control groups of those who had any working smoke alarm reported that they had one (table 1). The lowest sensitivity was observed for presence of a stair gate; 67% in intervention group and 71% in the control group reported they had protected their stairs by stair gates when they actually did.

Specificities were much lower for the smoke alarm questions (table 2). Only 9% in intervention group and 7% in control group who did not have any working smoke alarm reported the same. Similar specificities were observed for both stair gates and syrup of ipecac in both groups, ranging from 73% to 78% (table 2).

Positive and negative predictive values

The PPV for having a working smoke alarm was 61% in the intervention group and 52% in the control group (table 3). Ninety five percent of these families whose self report was not confirmed at home observation actually had a smoke alarm, but it was not working (data not shown). For having a working smoke alarm on every floor, the PPV dropped to 25% and 26% in the two study groups (table 3). The biggest difference in PPV between the groups was seen for syrup of ipecac—33% in the intervention group and 71% in the control group (table 3). In general, the NPVs were high in both study groups, more than 80%, for all of the safety practices observed (table 4).

DISCUSSION

This is the first study to our knowledge to provide evidence of the validity of self reported home safety practices, although the topic is one that has generated considerable debate recently.^9,10 Specificity and PPV varied considerably among the different safety practices. However, we found good sensitivities and NPV of the self reported interview responses for the four safety practices. The sensitivities and specificities were stable for measures of both stair gates and syrup of ipecac. However, the PPVs were low, due in part to the low prevalence of these two safety practices in our sample.

That participants tended to over-report the presence of a working smoke alarm is noteworthy. We cannot infer that this was simply a social desirability bias. It is more likely that respondents thought they had working smoke alarms because 95% of those whose self report was not confirmed actually had a smoke alarm, but it was simply not working. These results suggest that more detailed questions are needed to ascertain information about the functionality of smoke alarms (for example, asking when the last time it was checked or the batteries changed).

Program evaluators are often concerned that participants in an intervention group may be more likely to over-report positive behaviors because they know they are receiving extra services. We found no evidence of differential reporting between study groups, as virtually all of our findings were similar for the intervention and control groups. The only difference was in the PPV for syrup of ipecac, which could be due to the differences in prevalence in the two groups.

This study provides new evidence that relying on self reported safety behaviors may be problematic, both for clinicians as well as for program evaluators. For clinicians who are assessing their patients’ needs and risks, parents’ reports that they do not have a safety product or that they are not practicing a safety behavior appear to be quite trustworthy. However, reports of using a safety product or practicing a safety behavior should be interpreted cautiously, especially for smoke alarms. This is particularly disturbing because housefires are a leading cause of death for children in our area, as it is in many other urban areas. Program evaluations should incorporate observational data whenever possible to increase the accuracy of prevalence estimates. However, we did not find systematic differences in validity of self reports between the study groups. Therefore, if the main interest in an evaluation is on the relative difference between study groups, rather than the absolute value of the outcome measure, our results suggest that self reported data may be of acceptable validity.