Background and objectives Four storage practices protect against unintentional and/or self-inflicted firearm injury among children and adolescents: keeping guns locked (1) and unloaded (2) and keeping ammunition locked up (3) and in a separate location from the guns (4). Our aim was to mimic common Google search strategies on firearm/ammunition storage and assess whether the resulting web pages provided recommendations consistent with those supported by the literature.
Methods We identified 87 web pages by Google search of the 10 most commonly used search terms in the USA related to firearm/ammunition storage. Two non-blinded independent reviewers analysed web page technical quality according to a 17-item checklist derived from previous studies. A single reviewer analysed readability by US grade level assigned by Flesch–Kincaid Grade Level Index. Two separate, blinded, independent reviewers analysed deidentified web page content for accuracy and completeness describing the four accepted storage practices. Reviewers resolved disagreements by consensus.
Results The web pages described, on average, less than one of four accepted storage practices (mean 0.2 (95% CL 0.1 to 0.4)). Only two web pages (2%) identified all four practices. Two web pages (2%) made assertions inconsistent with recommendations; both implied that loaded firearms could be stored safely. Flesch–Kincaid Grade Level Index averaged 8.0 (95% CL 7.3 to 8.7). The average technical quality score was 7.1 (95% CL 6.8 to 7.4) out of an available score of 17. There was a high degree of agreement between reviewers regarding completeness (weighted κ 0.78 (95% CL 0.61 to 0.97)).
Conclusions The internet currently provides incomplete information about safe firearm storage. Understanding existing deficiencies may inform future strategies for improvement.
Statistics from Altmetric.com
Firearm-related injuries remain a leading cause of mortality in children and adolescents in the USA. In 2013, 1806 firearm-related deaths were reported in US children of age 0–18 years, ranking second only to motor vehicle collisions as a cause of injury-related death in this age group.1 ,2 Non-fatal firearm-related injuries were even more frequent, with 9718 reports in US children of age 0–18 years in 2013.3 Contributing to these statistics is the availability of firearms,4–8 which are common in American households, with 32.6%–38% possessing at least one firearm.9 ,10 Prevalence of firearms in US households with children and adolescents under age 18 years is similar,11 ,12 resulting in exposure of approximately 22 million US children to home firearms.11
As stated by the American Academy of Pediatrics (AAP), the most effective measure to prevent suicide, homicide and unintentional firearm-related injuries to children and adolescents is the absence of guns in homes and communities.13 An observational study supported the link between firearm injuries and storage practices.14 In households with firearms, the four practices of keeping guns locked, either in a locked location or with an extrinsic lock (1), and unloaded (2) and keeping ammunition locked up (3) and in a separate location from the guns (4) are each associated with decreased risk of unintentional and/or self-inflicted firearm injuries among children and adolescents younger than 20 years.15 A large multicentre office-based survey of parents of children presenting for well child care across the USA, Canada and Puerto Rico found that only one-third of firearm owners reported safe firearm storage,16 which is similar to results from prior work.11 ,17 ,18 In this context, families may benefit from better education on the best practices for safe firearm and ammunition storage.
Paediatricians and other healthcare professionals have been encouraged by the AAP to inquire about the presence of firearms in a child's environment and to counsel caregivers about these practices.13 Yet, recent legislative activity has challenged physicians’ right to inquire and counsel on this topic in some US jurisdictions. Specifically, passage of the Florida Firearm Owners’ Privacy Act of 2011, which was recently upheld by the US Court of Appeals for the 11th Circuit, prohibits specific inquiry or record-keeping about a patient's firearm ownership status except in limited situations.19 Additionally, a national survey of AAP members in 2000 indicated only a minority reported adequate training in firearm safety (32.7%) or sufficient time in health supervision visits to deal with firearm safety (27.5%). In the same survey, only 15.2% of AAP members reported that they always identify families who have firearms at home during health supervision visits.20 Thus, it is important to examine other potential sources of firearm safety messaging to which families may be exposed.
Parents and other potential consumers of firearm storage materials may use the internet as a source of information. Nationally representative survey data have indicated that 59% of US adults have looked online for health information in the past year; of these, 77% say they began their last search for health or medical information at a search engine such as Google, Bing or Yahoo.21 In the USA between May 2014 and April 2015, Google was the most frequently used search engine (78%–80% monthly), ahead of Bing and Yahoo (8%–10% each monthly).22
To our knowledge, there have been no prior studies evaluating commonly available consumer-directed internet content on firearm and ammunition storage. Our primary aim was to mimic common consumer Google search strategies on this topic and assess whether the resulting web pages accurately and completely provided recommendations consistent with those supported by the literature. Second, we aimed to assess readability and technical quality of the web pages. Additionally, based on findings from studies of internet content related to other health topics,23–26 we hypothesised that web page content and readability would vary depending on the type of web page specifically with worse performance on retail sites. As groundwork to target the specific needs of potential future partners for collaboration to improve web page content in this area, we sought to describe the performance of retail web pages with respect to each of the above measures, using non-retail web pages for comparison.
This study was submitted to the University of Michigan Institutional Review Board and assigned ‘not regulated’ status for research using publicly available data sets. Our framework for methods of web page identification and assessment was modified based on previously published criteria for quality and rigour of methods used in studies that assess health information for consumers on the internet.27
Subject/web page identification
A priori, authors assembled a list of search terms related to firearm or ammunition storage based on synonymous terms identified in group discussion. Then we used the publicly available tool Google Trends, which compares relative frequency of search terms in Google by region and time period. Several additional synonymous terms were added to the list following initial search in Google Trends. The final list of potential search terms included the nine terms: ‘gun storage’, ‘gun safe’, ‘gun cabinet’, ‘gun vault’, ‘gun box’, ‘gun lockbox’, ‘gun lock’, ‘gun safety’, ‘safe gun storage’ and each of the preceding terms replacing the word ‘gun’ with each of the eight words ‘firearm’, ‘handgun’, ‘rifle’, ‘pistol’, ‘weapon’, ‘ammunition’, ‘ammo’ and ‘bullet’. From the list of 81 potential search terms, we identified the 10 most commonly used terms in Google within the USA from 2004 until 30 September 2014 to approximate the most common search criteria used to identify information on this topic (see online supplementary eTable 1). We then found the top 10 web pages by rank order presented in a Google search of each of these terms conducted on 1 October 2014. Using the top 10 web pages was intentional, as analysis of actual click patterns during internet searches has shown that displayed position is more important than relevance in determining whether a page will be viewed.28
We saved copies of each complete web page for analysis between 1 October 2014 and 18 October 2014, and excluded duplicate pages and pages with completely unrelated content from analysis. We analysed only the initial web page linked from the Google search; analysis did not include information linked to this page even if contained within the same website. Observation of internet users searching for health information has demonstrated a median of 37 s spent on each website.29 If the first page does not contain information of interest to the user, he/she will likely try another site.28 By limiting analysis in this way, we believe that we captured the information most readily available and likely to be viewed by consumers.
Web page assessment measures
Two non-blinded independent reviewers assessed the pages for type of web page (eg, government, organisation, retail, education, news, etc.) according to previously published criteria, reproduced in part with examples from this study in online supplementary eTable 2,23 and adherence to 17 general web page technical criteria reflecting proper disclosures, sources of evidence and features that improve site usability such as an internal search engine. The technical criteria were derived from a previously published review of studies assessing the quality of health information for consumers on the internet27 and are listed in online supplementary eTable 3. Reviewers discussed initial disagreements until arriving at consensus. A non-blinded reviewer additionally downloaded content for each page and tested it for readability using the Flesch–Kincaid Grade Level Index, which assigns a US grade level to English language text, for comparison with published guidelines for target grade level for patient education materials.30 Subsequently, the non-blinded reviewer deidentified the content by removing all information that would identify authorship, ownership and sponsorship and changing text to a consistent size and font. Two separate, blinded, independent reviewers then assessed this content for accuracy and completeness according to the questions listed in box 1. Disagreements were discussed until the reviewers reached consensus.
Does the site say that guns should be stored locked?
Does the site say that guns should be stored unloaded?
Does the site say that ammunition should be stored locked?
Does the site say that ammunition should be stored separately?
Does the site contain inaccurate assertions regarding gun storage? If so, what? (Example: “Avoid sleeping with a loaded gun if you sleepwalk”.)
How many of the above questions (1–4) are answered ‘yes?’
The web page was the unit of analysis in this study. After excluding one irrelevant page and 12 duplicate pages, the search resulted in a total of 87 unique web pages from 46 websites.
The accuracy, completeness, grade level index, technical quality and overall technical quality scores of 87 web pages were calculated and compared between retail and non-retail web pages.
The categorical and continuous measurements are described in terms of proportions and means with their 95% confidence limits, respectively. We compared retail and non-retail page completeness ordinal scores using an exact trend test. We compared retail and non-retail page accuracy and individual technical quality measures using Fisher's exact test. We compared grade level index and overall technical quality scores using the Wilcoxon rank sum test. Also we assessed for the absence of disagreement between reviewers by Exact McNemar's test for technical quality and accuracy items and determined the degree of initial agreement by weighted κ test for completeness score. All inference testings were performed using SAS V.9.4 (SAS Institute, Cary, North Carolina, USA). A p<0.05 was considered significant.
The majority (74%) of 87 web pages were retail. Non-retail pages were categorised as follows: 39% organisational, 26% educational, 26% news, 4% government and 4% sponsored link.
The web pages described on average fewer than one of four accepted firearm and ammunition storage practices (mean 0.2 (95% CL 0.1 to 0.4)). Only two web pages (2%) correctly identified all four practices. Recommendation of individual storage practices ranged from a low of 3% for storing ammunition separately from guns to a high of 7% for both storing guns locked and storing guns unloaded. Only two web pages (2%) made assertions that were not compliant with best practice evidence; both stated or implied that loaded firearms could be stored safely. Flesch–Kincaid Grade Level Index for all web pages averaged 8.0 (95% CL 7.3 to 8.7). The average overall technical quality score was 7.1 (95% CL 6.8 to 7.4) out of available 17.
With respect to the secondary analysis of the subset of retail pages, completeness scores were worse among retail when compared with non-retail pages (95% vs 74% with completeness score 0; exact trend test p=0.02). Retail pages were significantly less likely than non-retail pages to state that guns should be stored unloaded (2% vs 22%, Fisher's exact test 0.005). There was a trend toward lower rates of recommending the three other storage practices on retail pages when compared with non-retail pages, but this did not reach the level of statistical significance. Full accuracy and completeness results are available in tables 1 and 2.
Grade level index was lower for retail pages compared with non-retail pages (mean 7.5 (95% CL 6.7 to 8.3) vs 9.5 (95% CL 8.1 to 10.8), p=0.006). The overall technical quality score was slightly higher for retail when compared with non-retail pages (mean 7.2 (95% CL 7.0 to 7.3) vs 6.9 (95% CL 5.6 to 8.1), p=0.02). Readability and overall technical quality results are available in table 3. With respect to individual technical criteria, retail pages performed worse on several features related to assuring quality of information. Specifically, retail pages were significantly less likely to disclose authors’ affiliation, statement of purpose, date of creation or date of last update or to make sources of information clear.
With respect to inter-rater reliability, using Exact McNemar's test, we found no significant tendency for web pages to be assigned differently for accuracy or technical quality measures based on reviewer (p>0.05) except for disclosure of authors’ affiliation and site sponsorship (p=0.007). There was a high degree of agreement between reviewers with respect to completeness (weighted κ 0.78 (95% CL 0.61 to 0.97)).
Prior studies have evaluated internet guidance on many other paediatric and young adult health maintenance topics; these studies have generally reported low content and/or accuracy scores but exact results have varied depending on the specific study and topic.23–26 ,31 In light of legislative action and other barriers to physician inquiry about household firearms, we became interested in alternative and/or complementary sources of messaging regarding firearm and ammunition storage. In addition to reaching firearm owners who attend primary care office visits but do not receive counselling in this setting, the internet also holds potential as a source of messaging for firearm owners who never attend these visits.
To our knowledge, our study is the first to evaluate the accuracy, completeness, readability and technical quality of content consumers would easily find on the internet regarding firearm and ammunition storage. The most striking finding of our study was that overall completeness of information is very low. While only a small percentage of websites had inaccurate information, the lack of guidance regarding known safe storage practices was notable. While we did not perform a formal analysis of the specific content of each web page, informal review noted large numbers of advertisements and pictures of products available for sale without instruction regarding safe use and storage. Studies of web content on other topics have noted variability based upon the type of website, with worst performance on sponsored sites25 ,26 and commercial or news sites.24 Our study results are largely consistent with this prior work, showing better completeness among non-retail, when compared with retail pages. The majority of pages were retail, however, making this group an attractive target for potential partnership and collaboration to expand safety messaging. Recommendations for safe storage techniques would largely be consistent with sales of locks, cabinets and other safety products and may therefore be mutually beneficial to retailers.
As further groundwork to improve safety messaging, future study should address possible barriers to display and acceptance of recommendations. Specifically, keeping firearms in the home for personal protection has been associated with unsafe storage practices in a survey of primary care patients,32 and belief that a firearm could scare off a burglar or keep someone in the family from being hurt was associated with less favourable attitudes towards keeping firearms locked in a national survey of married women with children.33 Our study did not specifically assess the frequency of messaging that supports this perception, but it is possible that retailers using this perception for marketing may find it contradictory to display complete safe storage recommendations.
In addition to improvement in content, improvement in readability would also be an important target. The average Flesch–Kincaid Reading Level of all web pages in our study was 8.0, which is similar to recently reported mean reading level of information on weight loss available on the internet24 but above published guidelines for target grade level for patient education materials.30 Flesch–Kincaid Reading Level does not fully capture all factors that contribute to reading comprehension, such as writing style, appropriate use of formatting and cultural sensitivity, but it does provide one easily standardised, measurable marker of readability.
Specific technical quality features were also lacking although patterns were not uniform, consistent with prior analysis of web pages from pooled studies of health information on the internet which revealed significant heterogeneity for many technical quality measures.27 The overall profile of retail pages was more consistent with features related to sales and customer service, such as sponsorship, advertising, copyright, internal search engine and feedback mechanism, when compared with features related to quality of evidence. Providing retail partners with quality safe storage information would be important.
Potential limitations of this study include lack of clarity regarding how frequently families use the internet to obtain information on firearm and ammunition storage and how closely our search method and analysis reproduced the information these families would encounter. We selected search terms to mimic a consumer's search experience, but Google Trends does not distinguish between terms used by caregivers for children and those used by others within the USA. It is possible that families searching for information would use slightly different search terms and possibly reach a slightly different group of web pages. Focus group or survey data from families regarding search terms could possibly have augmented the choice of search terms. Messaging received by gun owners without children is also relevant, however, as children visit the homes of neighbours and relatives without children, and prior study has suggested that these homes may be more likely to have guns stored unlocked.34 We also did not separate search terms and results by region across the USA, which could be a useful future direction for local efforts given substantial regional variation in firearm prevalence and storage practices.9 Our decision not to include information on web pages ranking below number 10 in the initial search or on pages linked to the initial page was purposeful based on data regarding internet search behaviour, but it did possibly underestimate the amount of information (whether accurate or inaccurate) consumers would access through this type of search. Consumers who pursue a more diligent search than assumed in this study may be rewarded with better information. Future work to evaluate completeness and accuracy of messaging throughout entire websites could determine if there is potential for interventions to improve ease of access to existing but poorly viewable material.
Additional important strengths of our study design included dual-rater analysis for everything other than Flesch–Kincaid Grade Level Index, as well as blinding for content analysis to limit bias based upon the source of the web page. Our evaluation tools for accuracy, completeness and technical quality performed very well with respect to inter-rater reliability.
Commonly used internet search strategies are unlikely to yield optimal guidance for caregivers on safe firearm and ammunition storage. Improved strategies for information dissemination are crucial. One novel approach may be collaboration with site owners to improve existing internet guidance. Future study needs to evaluate how caregivers in households with guns obtain information about firearm storage as well as which strategies most effectively deliver information and positively modify behaviour in the large number of US households with improperly stored firearms and ammunition.
What is already known on the subject
Household firearm storage is common in the USA, including in households with children, resulting in exposure of approximately 22 million US children to home firearms.
The practices of keeping guns locked and unloaded and ammunition locked up in a separate location protect against unintentional and/or self-inflicted firearm injury among children and adolescents.
Among parents who are firearm owners, only about one-third report safe firearm storage.
What this study adds
We reviewed 87 web pages regarding firearm/ammunition storage.
On average, web pages described fewer than one of four accepted storage practices.
Only two web pages identified all four practices. This shows limitations of the internet to provide accurate information about firearm safety.
Special thanks to Kristen R. LeFevre, PhD, for her guidance on publicly available Google tools and to Shane C. Quinonez, MD, and Janet R. Gilsdorf, MD, for their reviews of the manuscript. KLF had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Contributors KLF substantially contributed to the study concept and design, acquired data, drafted the initial manuscript and approved the final manuscript as submitted. MSS, JPS and BAM acquired data, reviewed and revised the manuscript critically for important intellectual content, and approved the final manuscript as submitted. NRK carried out the data analysis, reviewed and revised the manuscript critically for important intellectual content, and approved the final manuscript as submitted.
Funding Statistical analysis was performed with financial support from ‘National Center for Advancing Translational Sciences’ from grant 2UL1TR000433.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Complete original data are available on request.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.