Article Text

PDF

Evaluating the acceptability and feasibility of the I Promise Program: a driving program for families with young new drivers
  1. E Votta1,
  2. M MacKay2
  1. 1CIHR Strategic Training Post-Doctoral Fellow, School of Nursing, University of Ottawa, Ottawa, ON, Canada
  2. 2Plan-it Safe, Child and Youth Injury Prevention Centre, CHEO, Ottawa, ON, Canada
  1. Correspondence to:
 Dr E Votta
 CIHR Strategic Training Post-Doctoral Fellow, School of Nursing, University of Ottawa, RGN 1118a, 451 Smyth Road, Ottawa, ON, K1H 8M5, Canada; lvotta{at}rogers.com

Abstract

Objectives: To evaluate the acceptability and feasibility of the I Promise Program (IPP), a driving program developed for families with young new drivers (YNDs).

Design, setting, and subjects: The IPP consists of a contract between parents and YNDs and a rear window decal (sticker). Program acceptability was assessed through four focus groups with 40 young new drivers (YND), two with 19 parents of YNDs, and two with 15 community members. To determine whether the program’s design, materials, and procedures were working as planned, 51 families participated in a six month pilot project. Telephone and in-person interviews were conducted at months 1 and 6, respectively.

Results: Participants had problems with the acceptability of the program’s underlying message; content, format, and language of the materials; program cost; and proposed participant incentives. Thirty eight (75%) families completed the six month pilot. Most YNDs (75%) and parents (85%) identified the contract as a useful communication tool. Despite positive initial reactions, 50% of YNDs did not recall the content of the contract after six months. Sixty eight percent of families had problems with the decal (for example, did not stay affixed, colors faded) and only 17% of YNDs reported a lasting impact on their driving. Only 20% of families chose to continue in the program after the pilot.

Conclusions: These results highlight the importance of formative and process evaluation in the development of a new prevention strategy to assess a strategy’s acceptability and feasibility. In response to participants’ feedback, revisions made to the program’s materials and delivery model included making its two key components—the contract and decal—available online, independent of each other, and free of charge.

  • IPP, I Promise Program
  • MVC, motor vehicle collision
  • YND, young new driver
  • young new drivers
  • motor vehicle collisions
  • driving contracts
  • formative evaluation

Statistics from Altmetric.com

Drivers aged 16–24 years are overrepresented in motor vehicle collisions (MVCs). Young drivers account for 14% of Canada’s population, yet account for 26% of traffic deaths and injuries.1 In 2000, 390 youth (16–19 years old) died and 29 674 were injured.2 Factors such as inexperience, risky driving behaviors, and risks associated with the driving environment are associated with youths’ increased risk.3 One possible approach to managing these risks is parent oriented strategies such as driving contracts.4,5

The I Promise Program (IPP) was developed to help families with young new drivers (YNDs) address the factors that relate most to teen MVCs. The program consisted of a parent-youth contract and a rear window decal (sticker) with a toll free number connected to a community monitoring service. Contract items included driving sober, refraining from using a cell phone, and limiting the number of passengers. After completing the contract families affixed the decal to any shared vehicle. Community members could file an anonymous report on the driver’s behavior with a professional call centre who then mailed the report to the vehicle’s registered owner. The cost of the program was $49(CAD)/year/family.

During program development, formative evaluation is used to test the appropriateness and acceptability of a program and its materials to its target audience.6 Process evaluation allows for the early evaluation of how program processes are working and/or reaching its target audience as planned.6 The IPP was developed and launched without the input of or pilot testing with its primary target audiences—YNDs and parents. The purpose of this study was to evaluate the acceptability and feasibility of the IPP to its target audience through focus groups and a pilot study.

METHOD

Focus group participants and procedures

To evaluate the IPP’s acceptability, eight 90 minute focus groups were conducted with the program’s three target groups: four with 40 YNDs, two with 19 parents of YNDs, and two with 15 community members. Participants were young drivers and parents of drivers aged 15–19 years who were preparing to get or had their learner’s permit (see table 1). Participants responded to posters seeking volunteers through schools and community centres.

Table 1

 Focus group participants

Participants were given a verbal description of the IPP and then asked about their initial perceptions by a moderator using a standardized script. Participants reviewed IPP materials and provided feedback on the original contract; a contract modified with more youth friendly language and formatting; the rear window decal; and the registration brochure with accompanying program information. Additional questions addressed program incentives; participants’ willingness to enroll in the program; strategies to increase participation; and, for YNDs, their anticipated peers’ reactions.

Pilot participants and procedures

To evaluate the IPP’s procedures, 51 families (28 urban, 23 rural) participated in a six month pilot (see table 2). Participants responded to posters at schools and community centers, television appeals, and newspaper advertisements seeking drivers aged 16–19.* Only one family who participated in the focus groups enrolled in the pilot. Families were enrolled in the IPP at no cost, mailed the program materials, and asked to notify the study coordinator of their start date.

Table 2

 Pilot participants

RESULTS

Focus groups

Focus groups were audiotaped, transcribed, and analysed by two raters using content analysis procedures.7 Both YNDs and adults described the IPP and its contract as useful tools to help families discuss driving related expectations and concerns. Although all participants appreciated the program’s concept of mutuality and recognition of parents as role models, YNDs felt targeted stating that adults also drive irresponsibly. Participants thought the program would work best for families where YNDs were responsible and had no communication difficulties. Participants stated the documents implied a safety guarantee, had too many contractual items, and used language that was not reader friendly.

Participants questioned if the decal would withstand climactic variation and if it would be seen at night or through tinted windows. There were concerns the decal did not contain sufficient information to indicate the driver was part of a program. To increase the possibility of positive reports, participants recommended changing the decal’s question from “How is my driving?” to “Am I driving safely?” Participants suggested large campaigns to ensure the public’s familiarity with the program and reporting process. Neither YNDs nor adults believed rewards (excluding insurance discounts) were necessary for participation and questioned the registration fee (“$49 seems like a lot for a piece of paper and sticker”).

Recommendations from focus groups were used to modify materials before continuing the pilot. Modifications included changing the decal to say “Am I driving safely?” and making the contract’s language more reader friendly and succinct.

Pilot

Three families withdrew in the first month of the pilot, three at three months, and seven could not be contacted at six months. Reasons for withdrawal were dislike of the decal, family no longer had vehicle, and family no longer wished to be in the pilot. At one month, phone interviews were completed with 48 (94%) families. At six months, in-person interviews were completed with 38 (75%) families. During both interviews, YNDs and parents were asked to comment on the contract and its usefulness; the decal; perceived impact on driving; family or friends’ reactions to their participation; insurance companies’ response to discount requests; and recommendations for improvement.

Interview data were analyzed using content analysis procedures.7 Results from the one and six month interviews were similar and thus aggregated. Families enrolled in the pilot because their YND was in the initial licensing stage process; YND was first child in family to get their license; insurance discount; follow up to driver’s education; and opportunity for parents to provide car safety information, heighten awareness of driving responsibility, and model good driving behaviors.

YNDs (75%) and parents (85%) identified the contract as a useful communication tool, with over 90% reporting they honoured the contract. Important items in the contract were restrictions on cell phone use, eating, and drug/alcohol use while driving; vehicle maintenance; emergency materials; and limits on the number of passengers. The latter item was noted as particularly important for families in rural areas as YNDs frequently car pool. YNDs noted that parents did not honor the contract item restricting eating while driving. YNDs reported that the contract did not add to information already covered by driver training and parents or prohibited as part of Ontario’s Graduated Licensing program.

Only 10 families included rewards in their contracts. Loss of driving privileges was the primary consequence for all YNDs. Parents living in rural areas admitted a personal impact of inconvenience when driving privileges were revoked from YNDs. YNDs stated that because consequences were not equal, the contract was not “mutual” and should not be emphasized as such. At six months, 45% of YNDs, 19% of mothers, and 19% of fathers could not recall contract specifics.

All YNDs were concerned about prank calls; 48% received “harmless teasing”. Others were embarrassed by the decal’s presence on their car, disliked a perceived “big brother” approach, were concerned it gave the impression they were not good drivers, and questioned if reports could be verified.

Families reported the following decal related problems: colors faded (13%); not visible due to tinted windows (13%) or window slant (13%); and did not stay affixed (13%). Consistent with focus group feedback, participants questioned the feedback loop’s feasibility, stating the decal was too small and not sufficiently reflective to be seen at night or in the sun’s glare. Participants believed the phone number was difficult to remember and recommended that it be replaced with a word. Other concerns included the amount of information required to file a report and the public’s lack of familiarity with the IPP.

During the pilot, two YNDs were involved in a motor vehicle crash; in one case, the decal was affixed. Two fathers received a speeding ticket and one YND received two speeding tickets during the pilot; the decal was affixed in two of the four instances. YNDs reported that any decal related impact on their driving was greatest in the first month. After time and no filed report, 83% of YNDs stated they were unaware of the decal. Although some parents reported feeling more accountable to drive safely and be role models, 54% of mothers and 69% of fathers concurred about the lack of sustained impact. Despite this, 67% of YNDs and 65% of parents stated both the decal and contract should remain in the program.

Parents (77%) and YNDs (60%) wanted to receive a report. During the pilot, only one report (positive) was received; family members identified the father as the driver. Pilot participants were enrolled in the program at no cost; although 33% of participants would have considered participating at a lower cost than the designated $49, 66% of YNDs and 53% of parents stated they would not have participated had they been required to pay the fee. In contrast to focus group results, all pilot participants identified an insurance discount as a critical incentive (“Any amount would help”). Many families inquired about an insurance discount, but only two received one.

Families completing the six month interview had the option of remaining registered in the program for the remainder of the one year registration period; 20% of families continued their participation and 29% indicated they would consider IPP for another YND in their family.

DISCUSSION

Given the magnitude of motor vehicle injuries and fatalities in Canada, injury prevention and road safety initiatives are numerous. In order for any injury prevention strategy to be effective, it must reach its target audience. This study demonstrates the pitfalls of developing and launching a road safety initiative without conducting a systematic formative and process evaluation. In response to both focus group and pilot participants’ feedback, numerous modifications were made to the program’s delivery model, materials, and cost following both the formative and process evaluation phases. For example, originally, families received the contract and decal in the mail after receipt of the registration brochure and fee. Following the pilot, the program developer made the contract and decal available online, independent of each other and free of cost. In the program’s first three years, the program drew 175 registrants. Since making the program free of cost, more than 200 contracts are downloaded per month.

Without evaluation, program developers would have been unaware of participants’ concerns about the community feedback loop. In addition to the 70% of participants who had problems with the decal (for example, colors faded), participants were concerned about prank calls, the veracity of reports, and the program’s “big brother” component. In the program’s three years, only one report was filed through the call centre—the perfect driving call made during the pilot study. Given participants’ concerns, the call centre component was removed as a mandatory program aspect. The decal is available to families who choose to use it. However, instead of an 800 phone number on the decal, families now place a personal phone number of their choosing, thus ensuring that calls received from a community member go directly to the family.

Families’ recall of contract items was poor at six months. Further, as an indicator of their program involvement, participants stated they wanted an interim report from the program and/or notification that another registrant had received a report. In response to and consistent with strategies used in other similar programs,4 interim communication features including reminders to revisit/recommit to the contract and status updates on the program itself were incorporated into the program.

Program developers assumed the decal would have a lasting impact on IPP participants regardless of whether a report was received. Instead pilot data indicated the decal had only a short term initial impact on all drivers. In some cases, the decal had no impact at all given that motor vehicle crashes occurred and tickets were received. Given the lack of a community advertising campaign to increase public awareness of the program, the long term impact of the decal in its current format was minimal. However, the lack of a report and program communication over the pilot period affected families’ decisions to terminate their participation, indicating that program communication in response to the decal appears to be an element in program sustainability.

Key limitations regarding the sample must be noted. First, participants were not randomly selected but instead were self selected thereby limiting the generalizability of the findings. Over 70% of participating families were of higher income households and over 75% of young drivers were female. This latter point is particularly noteworthy given that MVC related injury and fatality rates are higher among young males. Given the study’s self selection bias and lack of randomization, future studies should collaborate with local motor vehicle agencies4,11 as points of access from which to randomly select both males and females of all income levels to control for any potential sex or income effects.

Parents questioned if they were the group that would most benefit from the program because they had no communication difficulties with their teens and viewed them as low risk. This suggests the need for targeted strategies to engage and assess acceptability with families that are experiencing discordance, limit setting, and/or disciplinary difficulties.8,9,10

Parents in rural areas spoke to the freedom they lost when they imposed loss of driving privileges on their YND for contract violations. This issue may not have been realized without consulting target groups in rural settings. Future studies should thus employ quasi-experimental methods to test for geographical differences among drivers from urban and rural settings.

During recruitment, youth with G1, G2, and G licenses were included in the study. Youth with their G1 license must have a licensed driver with them at all times while G2 and G drivers can drive without such supervision. As 44% of YNDs in the pilot had their G1, youth may have been on their best driving behavior because they were being supervised. Future research should thus explore the effectiveness of the program among youth of varying license levels.

Lastly, although information obtained though focus groups is helpful and characteristic of a preprogram assessment with potential participants,11 given that only one focus group family enrolled in the pilot, it is unclear what, if any, impact the post focus group modifications might have had on the IPP. Why other focus group participants chose not to participate and why families who were eligible to participate chose not to after receiving the pilot package is unknown. Consequently, we are unable to determine if there were any differences between those who did and did not participate in the pilot.

CONCLUSIONS

Results highlight the importance of formative and process evaluation in the development of a new prevention strategy to assess its acceptability and feasibility. For initiatives requiring voluntary participation, target audiences must deem strategies acceptable—only if that occurs can participants be recruited and effectiveness be properly evaluated. Failure to take these steps can waste resources, lead to the development of unacceptable programs, and compromise a program’s potential effectiveness.

Based on this evaluation, we cannot recommend the IPP, in its original format, as an effective strategy to reduce driving related injuries and fatalities among youth. In its original format, there were many participant identified concerns with the program that were preventing families from supporting the program’s continuation. Problems with the decal and the lack of self reported impact on driving among young drivers and parents were key indicators of the program’s inability to produce behavior change. As noted, numerous modifications have since been made to the program and it is evident that another evaluation of the program in its modified form is important to determine whether the program is now able to produce sufficient behavior change to reduce driving related injuries.

Key points

  • This study highlights the importance of formative and process evaluation in the development of a new prevention strategy

  • Cost is a factor in individuals’ willingness to participate in a prevention program

  • Interim communication between a program and its participants must be a key element in any program that involves a lengthy contractual agreement

  • Participants in a program with a community feedback component want to receive feedback, and the delivery model and obligatory components of a program can affect individuals’ willingness to volunteer for program participation

  • Authors of this paper cannot recommend the I Promise Program, in its original format, as an effective strategy to reduce driving related injuries and fatalities among young drivers.

Acknowledgments

The authors would like to acknowledge Mr Gary Direnfeld, Executive Director of the I Promise Program, for his support of this independent evaluation. This work was funded by the Ontario Neurotrauma Foundation.

REFERENCES

View Abstract

Footnotes

  • * Graduated licensing lets new drivers get experience and skills gradually over a 20 month period. To become fully licensed, a driver must be at least 16 years of age, pass a vision test, test of knowledge of rules of the road and traffic signs, and two road tests (G1 and G2).

  • Competing interests: none.

  • Ethics approval was obtained from the CHEO Research Ethics Board (Ottawa, ON).

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.