rss
Inj Prev 8:248-251 doi:10.1136/ip.8.3.248
  • METHODOLOGIC ISSUES

How old is that child? Validating the accuracy of age assignments in observational surveys of vehicle restraint use

  1. S Moeller1,
  2. L Berger2,
  3. J G Salvador1,
  4. D Helitzer1
  1. 1Office of Evaluation, Department of Family and Community Medicine, University of New Mexico School of Medicine, Albuquerque, New Mexico
  2. 2Department of Pediatrics, University of New Mexico School of Medicine, Albuquerque, New Mexico
  1. Correspondence to:
 Dr Deborah Helitzer, Director, Office of Evaluation, Department of Family and Community Medicine, University of New Mexico School of Medicine Family Practice Building, Albuquerque, NM 87131, USA;
 helitzer{at}unm.edu

    Abstract

    Objectives: Many large scale observational studies of child restraint usage require observers to estimate the ages of the vehicle occupants. The accuracy of age assignments were assessed and possible methods to improve observational accuracy in research and field studies of child restraint use were identified.

    Methods: The validation study was performed at fast food restaurants in a metropolitan area. Three, two person teams observed 449 occupants of vehicles with at least one child passenger. The drivers were then interviewed to obtain the actual ages of the vehicle occupants. The primary outcome measure was the per cent of age assignments that were correct by age category (infant, toddler, school age, teen, adult). The observers had previously conducted a statewide child restraint observation study and were trained in estimating age categories.

    Results: A total of 391 (87%) of the 449 occupants were assigned to their correct age categories. Incorrect assignments were more common among infants (22% incorrect), but few infants (nine) were observed. The most frequent error was classifying adults (19 years and older) as teenagers (13–18 years).

    Conclusion: Trained, experienced observers approached 90% accuracy in their assignment of children to specific age categories. Additional study is required to determine whether these results are applicable to different age categories and observers.

    It is recommended that the National Highway Traffic Safety Administration, the Centers for Disease Control and Prevention, state and local agencies, and other sponsors of observational surveys consider observer competence as an important variable. The validity of age assignments can be assessed by randomly interviewing a sample of drivers. More accurate age estimates will improve decisions regarding prevention programs, funding, and policies.

    Observational studies of vehicle occupant restraint use are widely employed. They are used to justify new passenger safety programs and legislation,1–10 evaluate interventions,11–15 and influence funding decisions for state and local motor vehicle injury prevention efforts.16–18 Every two years, for example, the National Highway Traffic Safety Administration (NHTSA) conducts the National Occupant Protection Use Survey (NOPUS).19 Findings from NOPUS are often cited as the best available data on national trends in safety belt, child safety restraint, and helmet use. Observational surveys are used as the “gold standard” to assess the validity of self reported surveys of restraint use.20,21 A recent systematic review of programs to increase child occupant restraint use excluded studies which “did not use observed restraint use as one of the outcome measures”.11

    A key element in child restraint use surveys is the determination of occupant ages. One approach to obtaining accurate ages is to stop vehicles and interview drivers.3,4,12,22–24 This method is labor intensive, costly, poses potential risks to data collectors, and can affect the validity of the data because passengers have time to adjust their restraints. A more commonly used method, especially for large scale surveys, is to estimate occupant ages while observing vehicles stopped at traffic lights or intersections.19,25,26 Yet it is often not obvious whether a vehicle occupant is an infant or toddler, a pre-teen or adolescent, a teenager or adult. A literature search failed to reveal any studies of the accuracy of age estimation by observers in restraint use surveys, despite their importance in policy, research, and program evaluation. For the child safety restraint portion of NOPUS (where restraint use was observed for over 71 000 drivers and over 19 000 passengers), instructions to observers were “to use their best estimate to determine if the child was an infant or a toddler”.19

    As part of a statewide child occupant restraint use study, we developed training techniques for our observers to improve their estimates of occupants’ ages. We also conducted an assessment of the accuracy of their age assignments, using fast food restaurants as a safe and convenient venue for making observations and querying drivers.

    METHODS

    In 1998, New Mexico’s Department of Health and Traffic Safety Bureau commissioned a statewide restraint use observation study (RUOS). The purpose of the study was to obtain restraint use information on individuals in five age categories: infants (less than 1 year), toddlers (1–4 years), youth (5–12 years), teens (13–18 years), and adults (over 18 years). Six observers spent three months collecting data on 4893 occupants in 2013 vehicles in New Mexico. The six observers were recruited from a university community based on job experience requiring accuracy and attention to detail. They ranged in age from 29 to 48 years (mean age 33.8 years; SD 7.3). Although none of the observers were parents, all of them had previous experience caring for children and teens. The number of years of experience ranged from three to 33 (mean 14, median 7).

    Each observer completed an eight hour training program that covered the rationale and protocol for conducting the survey, types of occupant restraints, age estimation, and state restraint laws. Training methods included classroom discussion, a photo identification exercise, and on-site practice. The photo exercise used 30 color photographs of 49 people who ranged from 3 months to adults, were of different racial/ethnic groups, and appeared in both sitting and standing positions (but not seated in vehicles). The observers were given 10 minutes to examine the photos and assign age categories. The categories were the same as those in the statewide RUOS data collection forms. Actual ages were then revealed to the observers, who “graded” their own estimates. The overall correct score for the six observers was 82.5% (range 71%–90%, SD 7.3). On-site practice involved restraint use observations in the field. Teams were formed consisting of two observers who shared the roles of “caller” and “writer”. Data elements on the observation form included vehicle type, seating position, gender of driver, occupant age category, and type of restraint.

    After the field session and photo exercise, trainers and observers discussed issues encountered in estimating age and observing restraint use. For example, observers were alerted to not let car safety seat positioning (forward or rear facing) influence their estimation of whether a child was an infant or toddler, since car seats are often used incorrectly.22,23 The observers were encouraged to rely on a constellation of factors (such as head shape, clothing, and age specific behaviors) to improve their accuracy in estimating occupants’ ages.

    Two months after completing the RUOS, the six observers participated in the validation study to assess the accuracy of their age assignments. A validator joined the two person observation teams at the drive through area of three fast food restaurants. A sign informed patrons of a “child passenger safety” study. Observers were positioned near the order box. The validator was positioned between the order box and the pay window. Only vehicles with at least one child occupant (birth to 18 years old) were eligible to participate. Observations were conducted while vehicle occupants were placing their food orders. The observer called out the ages and seating positions to the writer, as in the RUOS study. When the vehicle moved forward to the pay window, the validator approached the driver, explained the purpose of the study, offered a discount coupon for participation, and asked the driver and occupants to report their ages. For matching purposes, the license plate number was noted on the data collection form by both observers and validator.

    Analysis of proportions by χ2 was used to determine the accuracy of the assignment of age categories. STATS VIEW 5 statistical software was used for the data analysis.27

    RESULTS

    A total of 173 motor vehicles containing 449 occupants were observed during the validation study (table 1). Of the occupants, 391 (87.1%) were assigned to their correct age category (percent correct assignment = number assigned correctly divided by the total number of occupants in that age group, as reported by the driver). There was no statistically significant difference in the accuracy of age assignments by the age category of the occupants (p=0.83). The overall accuracy of age assignments by the three teams was 86%, 88%, and 89% (p=0.73).

    Table 1

    Correct assignments by age category

    Table 2 shows the observed frequencies of the 58 incorrect age assignments by the direction of error. The direction of error is necessarily unilateral for both infants and adults: adults could not be assigned to an older age category, infants could not be assigned to a younger one. Infants were somewhat more likely than individuals in other age categories to be estimated incorrectly: of nine infants, two (22.2%) were assigned to the toddler category. Errors in both over and under estimation of age categories occurred for the intermediate age groups (toddler, school age, and teens). Not surprisingly, many (53%) of the inaccurate assignments involved individuals whose ages were close to “break points”. For example, there were seven 4 year old children who were assigned to the 5–12 year old category and three 5 year old children assigned to the toddler (1–4 year old) category. Eight adults who were 19 or 20 years old were classified as teens (13–18 year old).

    Table 2

    Incorrect age assignments by age category and direction of error

    DISCUSSION AND RECOMMENDATIONS

    Overall, our observer teams were very accurate in assigning age categories: 87% of the 449 vehicle occupants were assigned correctly. Possible factors for success were the observers’ innate ability to categorize ages, their experience working with children, the age estimation training, and three months of experience observing occupants during the RUOS. Being part of a team may have also enhanced the accuracy of age assignments by providing “second opinions” regarding occupant ages. That 83% of the age category assignments were correct in the photo exercise suggests that several of the observers were already very skilled in estimating ages before they received any specific training or experience in observation. This is consistent with results from experimental psychology, where untrained subjects often do well in age estimation tasks.23,28,29

    The high proportion (53%) of incorrect age assignments occurring at “break points” between age categories prompts a cautionary note for studies where specific ages, rather than age categories, are of critical importance. For example, observational studies of booster seats target 4–9 year olds30,31; studies of rear versus front seat position focus on children under 12 or 13 years of age32–34; and studies of graduated licensing target teenage drivers.35–37 Interviewing drivers to obtain the correct ages of vehicle occupants is not a perfect solution. For example, the driver may not be the parent and therefore not report the correct age of a child occupant.

    When selection bias is not a critical barrier, the accuracy of age assignments can be improved by using times and locations that attract specific age populations. For example, studies of teen seat belt use have been conducted at high school parking lots, infants can be observed at Women, Infants and Children’s Program sites, and toddler restraint use can be noted among children at day care programs.

    The small number of infants (nine) observed during the validation study limits any conclusions regarding this important age group. Designers of observational surveys should consider over sampling infants and toddlers, since these groups were only 4% and 12%, respectively, of the occupant population in our statewide survey. That two of the nine infants were assigned to the toddler age category suggests that age assignments may be particularly challenging for older infants (10–12 months of age).

    Another limitation of our study is that the fast food drive-through setting did not entirely duplicate field conditions. For example, observers were less subject to fatigue during the validation study because it required only two hours of observations, compared with six hours each day during the field survey.

    Further age validation studies should determine how generalizable our results are to other observers, populations, and age categories. The independent impact of training on improving age assessment could be assessed by randomizing observers to groups with and without training and then validating their observations. The photo examination exercise is simple and inexpensive, and can identify observers who have particular difficulty in estimating ages. The exercise could be strengthened by having the children in the photographs seated in vehicles to simulate observation conditions. Discussion of relevant issues in child development (such as when infants can grasp objects or what ages young people begin to smoke cigarettes) and common mistakes in estimating ages (for example, assuming that a forward facing car seat guarantees its occupant is a toddler) should be included in the training, in addition to practice in the field. Car seat “clinics” (where NHTSA certified individuals check for car seat misuse) offer another valuable training opportunity.38 At these clinics, observers can visit with many young children in different types of car safety seats and booster seats, and verify their ages with the parent or guardian.

    It is likely that observers improve with experience. This hypothesis could be tested by conducting an age validation study before and after observers participate in a large scale restraint use survey. Finally, safety recommendations for appropriate vehicle restraint use are often based on the weight of the child, rather than, or in addition to, the child’s age.39 The reliability of weight estimates of child occupants has never, to our knowledge, been investigated.

    We recommend that NHTSA, the Centers for Disease Control and Prevention, state and local agencies, and other sponsors of observational surveys consider observer competence as an important variable. The validity of age assignments can be assessed by randomly interviewing a sample of drivers to attain the true age of the occupants. More accurate age estimates will improve decisions regarding prevention programs, funding, and policies.

    Key points

    • Researchers and program staff need to use competent observers to acquire safety restraint data.

    • Trained, experienced observers demonstrated 87% accuracy in assigning vehicle occupants to five age categories.

    • The validity of age assignments can be assessed by randomly interviewing a sample of drivers to attain the true age of the occupants.

    • The most errors in age assignments occur at break points between age categories.

    • Comprehensive restraint use surveys should consider over sampling infants and toddlers, since they are a small percentage of the vehicle occupant population.

    • The accuracy of age assignments can be improved by using times and locations that attract specific age populations.

    • Further studies are necessary to maximize observer competence.

    Acknowledgments

    This work was supported by funding from the New Mexico Departments of Health and Traffic Safety. We would like to acknowledge the contributions of the data collection teams; McDonald’s stores in Albuquerque; Nancy Bondy and Wendy Wilson-Jones of the National Highway Transportation Safety Administration; and Lynne Fullerton-Gleason in the Center for Injury Prevention, Research and Education, Department of Emergency Medicine, School of Medicine, University of New Mexico.

    REFERENCES

    Free sample
    This recent issue is free to all users to allow everyone the opportunity to see the full scope and typical content of Injury Prevention.
    View free sample issue >>

    Don't forget to sign up for content alerts so you keep up to date with all the articles as they are published.