Article Text
Abstract
Objective: To determine the reliability of reporting of information on risk factors from a standard accident report form used by ski patrols and a follow up mail questionnaire or telephone interview among injured skiers and snowboarders.
Setting: 19 ski areas in the Canadian province of Quebec between November 2001 and April 2002.
Participants: 4377 injured skiers and snowboarders seen by the ski patrol, who completed a follow up mail questionnaire or telephone interview.
Main outcome measures: κ and weighted κ statistics were used to measure the chance corrected agreement for self reported ability, age, skiing time on day of injury, lessons, type of practice, use of helmet at time of injury, and hill difficulty.
Results: The κ value for helmet use at the time of injury was 0.88 (95% confidence interval 0.87 to 0.90) and for other risk factors ranged from 0.45 (skiing time on day of injury) to 0.98 (age). Few differences were seen in reporting by body region of injury. Reporting consistency was lower for respondents who completed telephone interviews compared with those who completed mail questionnaires and those who responded more than four months after the injury.
Conclusions: Moderate to almost perfect agreement, depending on the risk factor, exists between ski patrols’ accident report forms and follow up information. Ski patrol reports can be a reliable and readily available source of information on risk factors for skiing and snowboarding.
- skiing
- snowboarding
Statistics from Altmetric.com
Many relatively recent investigations have used data reported by ski patrols to detail skiing and snowboard injuries.1–8 Little information on the quality of these data exists, however. Some researchers suggest that although classification of outcome by ski patrols may be imperfect in distinguishing between fractures and sprains, for example, the body region is reported accurately.9 Similarly, a recent study that examined the accuracy of injuries reported by ski patrols compared with doctors’ diagnoses confirmed that the data from ski patrols were correct or mainly correct in 89.5% of all cases.10 To our knowledge, no assessment has been made of the reliability of information on risk factors recorded by ski patrols. Therefore, the purpose of this study was to document the consistency of reporting of information on risk factors between a standard accident report form used by ski patrols and a follow up mail questionnaire or telephone interview.
METHODS
The data used in this investigation were part of a larger case-control study of helmet effectiveness in skiers and snowboarders. We included people who reported to a ski patrol member at one of the 19 largest ski areas in Quebec for an injury sustained while skiing or snowboarding. Twenty ski areas originally were asked to participate, but one area did not send any of their reports until long after the season and we thus excluded its data.
The Direction de la promotion sécurité of the Secrétariat au loisir et au sport is responsible for the Act Respecting Safety in Sports.11 According to the act, each ski area operator in Quebec must ensure that first aiders (that is, members of ski patrols) who meet standards set by regulation of the minister are present during all hours in which the ski area operates.12 Members of ski patrols in Quebec must be aged at least 16 years and hold first aid certification, which must be renewed every 15 months.12
In addition, members of ski patrols on all hills in Quebec are required by law to send standard accident report forms to the Secrétariat au loisir et au sport throughout the ski season.12 The accident report forms include sections for body part affected and type of injury for up to three injuries. In addition, the address and telephone number of each injured person is recorded, along with their age, sex, skiing ability, participation, contributing factor, environment, use of equipment, mode of leaving the hill, and information on destination after the injury.
We informed participating ski areas of the study and asked them to send their reports every 2–3 weeks. An employee of the Secrétariat au loisir et au sport then photocopied the reports and sent them by courier to the project coordinator at Montreal Children’s Hospital. After the coordinator received the report forms, they extracted the name, address, telephone number, participation, and injury information. Because many forms were wholly or partly illegible, and because we knew there would be problems with addresses based on ski patrol recording of a verbal description of personal information by the injured people, we confirmed the address and telephone information for each potential participant with the Canada Post (www.canadapost.com), Infospace (www.infospace.com), and Canadian YellowPages (www.yellowpages.ca) websites. Even with these confirmatory procedures, we could not find many addresses and telephone numbers, which made it impossible for us to follow up these potential participants. The absence of contact information did not exclude the participant from the study (that is, we included them in the analysis of response rates).
We then sent questionnaires to all injured people. The questionnaire was evaluated for content and clarity by adult and adolescent skiers and snowboarders, members of the Quebec ski patrol, members of the Quebec Ski Areas Association, and scientific peers before a pilot study in April 2001. The pilot allowed us to clarify further any misleading or equivocal questions in the mail questionnaire or follow up telephone interview. We then developed a final version of the questionnaire for the main study.
A postcard reminder was sent within two weeks of the initial mailing to remind people who had not returned their questionnaire to do so. The questionnaire was in French and English and was skier or snowboarder specific. It included questions about participation, personal characteristics, helmet use, risk taking attitude and behavior, equipment damage, follow up care, exposure, level of skill and experience, past injury, and other relevant information on the injury.
Parents answered the questionnaire for children aged <15 years, and next of kin were asked to answer for people with a severe brain injury and those who had died as a result of their injuries. A maximum of five follow up telephone calls was made on different days of the week and at different times of the day to those who had not responded to the questionnaire within three weeks of the initial mailing.
Injured people who were telephoned and indicated they had not received or sent in their questionnaire were asked to complete a telephone interview if the person (⩾15 years) or parent consented verbally. Trained bilingual interviewers provided the participants with information about the project and informed them (or their parents) of the confidential nature of the responses and their right to withdraw from the study at any time. The telephone interview sought the same information as the mailed questionnaire. It was impossible to blind the telephone interviewers as to the nature of the project (that is, use of helmets and prevention of head injury), as they had to read and sign the Commission d’accès à l’information du Québec form to indicate their willingness to maintain the confidentiality of the personal and other information on the accident report forms.
This combination of an initial mail questionnaire with a follow up telephone interview for non-respondents was used successfully in a study of bicycle helmets by Thompson et al.13–15 For people who did not list (or for whom the ski patrol did not record) a correct telephone number, we attempted follow up by mail.16 When ski patrol accident report forms had only a telephone number without an address, we conducted a telephone interview if consent to obtain the same information as the mailed questionnaire was provided.
The above protocol applied to people who lived in Canada and the United States. For skiers and snowboarders from other countries (mostly European), we attempted only a telephone interview because of the anticipated substantial delay in mail. We sent a questionnaire immediately, however, if they asked for a mail questionnaire when they were telephoned or if they could not be reached by telephone.
Analysis
We evaluated the agreement between the accident report form and the mail questionnaire or telephone interview information using κ statistics for helmet use and κ and weighted κ statistics for other variables.17 We used κ or weighted κ statistics to determine the chance corrected agreement between responses on the ski patrol accident report form and the follow up mail questionnaire or telephone interview.
RESULTS
Overall, 10 245 total ski and snowboard accident report forms were completed for the participating ski areas. We chose and attempted to contact all 1576 people with head, brain, face, or neck injuries and 4667 controls. In total, 69% of the cases and 71% of the controls returned a mail questionnaire or completed a telephone interview, which gave an overall response rate of 70%. The response rate varied by ski area, ranging from 55% to 85% (15 hills had rates >65%; eight had rates >70%). Response rates by age were 67% in those aged <9 years, 74% in those aged 9–14 years, 67% in those aged 15–25 years, and 70.4% in those aged ⩾26 years. Rates were similar for snowboarders (72%) and skiers (69%).
We received 3470 questionnaires by mail, 906 by telephone, and one by fax. Most of the mail questionnaires and telephone interviews were completed in French (n = 3167); the remainder were completed in English (n = 1210). Follow up time from injury (that is, the time after the injury that we received the mail questionnaire or conducted a telephone interview) ranged from 19 to 332 days but was ⩽6 months for 91% of respondents.
The κ value, which reflects chance corrected agreement, was 0.88 (95% confidence interval 0.87 to 0.90) for reported helmet use (table 1). The κ values for other agreement information ranged from 0.45 to 0.98.
Head, face, and neck injuries compared with injured controls
When we examined agreement for helmet use only among people with head, face, and neck injuries, the κ value increased to 0.93 (0.90 to 0.95) and remained stable at 0.87 (0.85 to 0.88) for helmet use in those with other injuries (table 2). Few differences were seen between injured cases and controls in terms of κ estimates for other variables.
Mail questionnaire compared with telephone interview
The κ value for helmet use at the time of injury was similar for participants who completed the mail questionnaire and those who completed the telephone interview (table 3). The κ values for self reported ability, skiing time on day of injury, lessons, and hill difficulty were in a lower category of agreement for the telephone interview than the mail questionnaire.
Time to return of mail questionnaire and completion of telephone interview
The κ values were compared for mail questionnaires and telephone interviews completed within and after four months of the injury. Little change was seen in κ values for reported helmet use depending on time from injury (table 4). Although the agreement rating for self reported ability and lessons was lower after four months, considerable overlap was seen in the confidence limits.
DISCUSSION
This study is one of the first to look at the issue of the quality of data on injuries reported by ski patrols. This seems surprising given that many relatively recent studies have used these data to identify risk factors for injuries during skiing and snowboarding,1–8 both of which are activities that can exact a large public health toll.
According to the Landis and Koch criteria, reliability of reported data was similar for those with head and neck injuries and those with other injuries, ranging from “moderate” to “almost perfect”.18 As expected, reliability declined with time after the injury event—a finding consistent with other studies that examined recall.19,20 Agreement also deteriorated when a telephone interview rather than a mail questionnaire was used to survey injured people. The greater inconsistency with time from injury and mode of obtaining information more likely reflects errors made on the mail questionnaire or telephone interview than the ski patrol’s recording of information on risk factors. This might be expected more for risk factors that may not relate directly to the injury event, such as skiing time on the day of injury and perhaps hill difficulty, than for characteristics related to the event, such as helmet use.
Perhaps most surprisingly, use of helmets was recorded with high reliability, even for people who did not have head injuries. This is a key finding, because it suggests that a potential source of bias may not be operating. For example, if a study of helmet effectiveness in skiers and snowboarders used people with non-head injuries as the control series, it could be argued that the protective effect was underestimated because the ski patrol would be less likely to record helmet use for these participants, deeming helmet use to be inapplicable information. The high reliability between the accident report forms and the mail questionnaires and telephone interviews for reported helmet use in non-head injured people in our study does not support this argument.
Study limitations
This study did not include people who were injured and for whom an accident report form was completed but who did not return the mail questionnaire or complete the telephone interview. Inclusion of these people may have changed the results. When we included non-responders in an analysis of helmet effectiveness on the basis of only the data from their accident report form, however, the results for this variable changed little.
In addition, if data were missing on an accident report form, mail questionnaire, or telephone interview, we could not use the responses in the analysis. Why these people did not have specific information recorded for the accident report form, mail questionnaire, or telephone interview is not known.
The ski areas were given information about the study and were asked to complete and send their accident report forms in a timely way. This may have influenced the reporting accuracy of the ski patrol. For example, ski patrol members may have been more diligent than normal in filling out forms during the study.
Many studies have shown that the total injury rate is underestimated with injuries reported by ski patrols as the numerator.21–26 A recent Canadian study found over four times as many injuries reported to a medical facility at the base of the ski area than to the ski patrol,23 although this likely reflects the uncharacteristic proximity of that medical facility. Evidence suggests that injuries that interfere with walking (for example, fractures of the lower extremities) tend to be reported more often to ski patrols, as are fractures and lacerations.22,24 For injuries that do not interfere with walking (that is, injuries not of the lower extremities), however, a strong correlation between injury reporting and severity has not been established.22,26
In terms of personal characteristics, women tend to report injuries more than men, as do younger age groups, those with lower ability, and those taking lessons.22,24,26,27 Factors not found to relate to injury reporting after adjustment was made for age and sex include time of injury, hours of skiing prior to injury, presence of previous ski injury, marital status, years of skiing experience, participation in previous ski classes, use of rental equipment, and collision as the cause of injury.22 Although many of the investigations into injury reporting issues were in the 1970s and 1980s, and only on skiers, no compelling reason suggests that the results would not apply similarly to present day skiers and snowboarders.
The key issue is not that we captured information on all injuries but that those who were included in the study were no different than those who were not in terms of agreement between the information on risk factors reported by their ski patrol and that from follow up questionnaire or telephone interview. This is likely the case. Furthermore, when we stratified by type of injury, the agreement ratings were similar, indicating that the results are robust.
Finally, reliability is not validity. That is, even though the results generally show consistent reporting between the accident report form and the mail questionnaire or telephone interview, the information might consistently have been incorrect. At least for face validity, however, this seems unlikely for the variables considered in this investigation.
Key points
-
Agreement between data reported by ski patrols and information obtained from a follow up questionnaire or interview ranged from moderate to almost perfect.
-
Reporting consistency did not differ between those who reported head, face, and neck injuries and those who reported other injuries.
-
Consistency declined with increasing time between injury and questionnaire or telephone interview mode of completion.
CONCLUSION
This study is the first we know of to address the issue of reliability of information on risk factors reported by ski patrols on accident report forms. The results suggest moderate to almost perfect agreement, depending on the risk factor evaluated, between information reported by ski patrols on accident report forms and a follow up mail questionnaire or telephone interview. Information reported by ski patrols thus can be a reliable and readily available source of data on risk factors for injury. In addition, when ski patrols’ accident report forms are used to identify an injured series for follow up, contact with each person in a timely way will likely reduce the level of misclassification in the data.
Acknowledgments
Research assistance of Genevieve Gore, Liam Gore, Helen Magdalinos, Gail Martin, Elise Mok, and Anna Sampogna was invaluable. We thank Nicole Marchand of Quebec Secretariat au loisir et au sport for her help in copying and sending the ski patrol accident report forms in a timely manner. We thank Quebec Ski Areas Association and participating ski areas for their cooperation with data collection.
Canadian Institutes of Health Research and Quebec Secretariat au loisir et au sport provided funding for this project. Alberta Heritage Foundation for Medical Research provided personal funding for BH.