Article Text
PDF
Abstract
The purpose of this study was to assess the content, quality, and type of internet resources available for safety education. Using 19 search engines with search strings targeting major forms of injury, identified resources were classified by audience group, accessibility, and authorship. Two independent reviewers rated each resource on the basis of its content and a set of quality criteria using a three point scale. Overall, 10 (18.2%) resources were of highest quality, four (7.3%) were intermediate, and 41 (74.5%) were not recommended. Eighteen months after the original search, 67.3% of all resources and 90% of the highest quality resources were still on the internet. This study provides a methodology for evaluating child safety resources on the world wide web and demonstrates that most internet resources for safety education are of dubious quality. A rating system such as the one developed for this study may be used to identify valuable internet materials.
- education
- health
- internet
- evaluation
- safety
- trauma
- world wide web
Statistics from Altmetric.com
Internet use has risen dramatically in recent years, a trend estimated to be three times faster than that of television, four times faster than personal computers, and nine times faster than radio.1 Parallel to this growth in the internet has been a growing recognition of the huge burden of injury worldwide.2 School teachers aided by public health professionals, parents, and community groups increasingly lead health education efforts and programming that depend on internet resources.3,4
Access to such a large and unregulated body of information, as exists on the internet, suggests a need for critical evaluation of health related internet resources.5 Although a number of studies have reviewed internet information for many different health conditions,6–8 there have been no similar evaluations of online injury prevention information. The purpose of this work was to assess the content, quality, and type of internet (web) resources that are available to supplement child safety education.
METHODS
Search strategy
Copernic 2001 Basic combined with Ask Jeeves, Overture, and Dogpile were used in July 2001 for the search strategy, with Microsoft Internet Explorer 5.5 as the web browsing software. Nineteen search engines were accessed; at the time of the search these included the top 16 search engines reported by Jupiter Media Metrix.9 Only the first 10 URLs were collected because we assumed that typical web users do not look beyond the first page of search engine. Non-English resources, resources that required payment, and resources that contained irrelevant information were excluded.
The websites were categorized by: (1) target audience; (2) the means of access to each resource; and (3) authorship.
Evaluation criteria of material found
Two raters independently evaluated the content and quality of each resource. Inter-rater reliability was assessed by percent agreement and Kendall’s tau. The following process was used for evaluation:
(A) Content assessment score
Content checklists specific to each safety theme and audience group were developed. These were intended to represent the content an ideal resource would contain (table 1).
A sample of two content checklists. A subjective assessment by two reviewers of whether each content item was present in the resource determined its content assessment score
Each reviewer assessed the presence or absence of each content item in the resource. The percentage of total content checklist items present in the resource constituted its content assessment score.
Our initial version of the checklists was drawn from the content contained in the Think First for Kids injury prevention curriculum,10 which is a comprehensive, internationally used school based injury prevention program that corresponds with the health and safety curricular expectations of all the Canadian provinces,11 and the “Kid’s Page” of the website of the United States Fire Administration.12 The initial checklists were supplemented by content items drawn from the first 15 resources included in the study to create a final comprehensive content checklist.
If a resource covered more than one safety theme, the theme that received the highest content assessment score determined the overall content rating of a resource.
(B) Quality assessment score
Quality criteria were drawn from those described by Worthen and Sanders,13 and the American Medical Association recommendations for websites.14 Upon completing our preliminary search strategy we felt that a more appropriate set of criteria should consider audience needs, and accommodate for the variable forms in which online information is presented. Teacher resources were therefore evaluated on criteria that considered their usefulness in teaching: clarity of the directions/lesson plan, level of engagement for children, number of outside materials required, references to source material, and currency (timeliness). Parent resources’ criteria were more typical of those used for evaluating websites: readability, aesthetic appeal/organization, and currency. Child resources were evaluated on the same criteria as parent resources, adding the criteria of lack of advertisements and navigability, since child materials tended to spread their information out across several web pages. Each criterion was rated on a scale of 1 (lowest) to 3 (highest).
As a result of different criteria and weighting, child, teacher, and parent resources were scored for quality out of 13.5, 12, and 7.5 points, respectively. These scores were converted to percentages, called the quality assessment score.
(C) Overall rating
Content and quality assessment scores were converted into overall ratings of “poor”, “fair”, or “excellent”. A “poor” rating was given for scores less than 50%, a “fair” rating for scores of 50% to less than 75%, and higher scores were deemed “excellent”. In determining the overall content ratings, only the content assessment score from the main safety focus of a resource was considered. Inter-rater agreement was calculated for this categorical scale.
RESULTS
We identified a total of 55 internet accessible resources. We classified 19 (34.5%) as child oriented resources, 25 (45.5%) as parent oriented, and 11 (20.0%) as teacher oriented. Non-profit (47.3%) and government authors (27.3%) produced a majority of the resources.
The number of resources that addressed each safety theme, starting with the most common, was: bicycle safety (n = 28), water safety (n = 24), car/seatbelt safety (n = 19), sports/recreation safety (n = 15), bus safety (n = 14), pedestrian safety (n = 14), violence (n = 14), and fire safety (n = 12). Falls were included in the sports/recreation and pedestrian categories. Some resources discussed more than one safety theme, so these numbers do not add to the expected total of 55.
Fifty one of the 55 safety resources contained web page content, 13 resources contained content requiring an additional download, two resources required a print out, and five resources allowed for free postal mail delivery of some materials.
Content ratings
Four child resources (21.1%), four parent resources (16.0%), and four teacher resources (36.4%) earned “excellent” ratings for content from both reviewers. The agreement between reviewers for the categorical poor, fair, and excellent ratings was 80.0%, and Kendall’s tau was 0.952.
Quality ratings
Both reviewers gave “excellent” quality ratings to 10 child resources (52.6%), 13 parent resources (52.0%), and six teacher resources (54.5%). The associated agreement on these ratings was 69.1%, with a Kendall’s tau of 0.947.
Overall recommendations
Ten (18.2%) of the 55 resources were rated as “excellent” by both reviewers for content and quality. Four (7.3%) resources fell short of this category because of one “fair” rating for either content or quality from one reviewer. Forty one resources (74.5%) were rated “not recommended” because they received more than one “fair” (or worse) rating on content or quality.
DISCUSSION
Our study found that content-rich, high quality web resources to supplement child safety education exist, but are hidden among a larger number of less desirable resources. From our data, a teacher or parent searching for safety education resources encounters material of questionable content, completeness, or quality about 75% of the time. Similar results—that excellent online resources make up a minority of what is available—have been found in other studies of web based health information.6,7,15
There are several limitations to the generalizability of our results. First, as new information is added to the ever expanding and ephemeral world wide web, older information is being removed, relocated, or becomes otherwise unavailable. For example, Li et al and Pandolfini and Bonati found that only a fraction of sites remained accessible after a period of up to two years.6,16 Our initial review was performed in July 2001 and two follow up “accessibility checks” were performed 12 months and 18 months later. While the initial review found 55 resources, the 12 month follow up search found 38 resources, and 18 months later there were 37 accessible resources. Parent resources were the most transient (declined from 25 to 15 over 18 months) while those that we rated as the best quality were most likely to persist after 18 months (90.0% remaining versus 58.5% for those not recommended).
A second limitation of our study is that the quality criteria we define may not fully describe the quality of a safety education resource for child, teacher, or parent audiences. A consensus on the definition of a high quality website is needed before meaningful evaluations can be conducted.
Perhaps the most difficult aspect of using the world wide web is simply finding relevant documents. Although we chose not to document the large amount of unrelated information encountered in our search, wading through extraneous results is a major obstacle that web users must face when seeking injury prevention resources. The output from today’s search engines often reveals little about the nature of the located documents. Users are frequently led to resources where the content does not correspond to the search query or intended audience, or the overall quality of the resource is questionable. This may be a result of search engines allowing companies to bid for prominent placements on search output screens, and consequently, the best, most relevant sites may not be seen. In our experience, brief descriptions provided by website authors for each of their links are often more helpful than the typical search engine output in determining the relevance of the linked document. Such links are often subjectively compiled by the website authors and are not rigorously evaluated. Before forming website links or using an online resource for safety education, we would encourage the use of rating systems such as the one we have developed to determine the value of the online document.
REFERENCES
