Article Text

PDF

Routine narrative analysis as a screening tool to improve data quality
  1. S J Jones,
  2. R A Lyons
  1. Department of Epidemiology, Statistics and Public Health, University of Wales College of Medicine, Cardiff
  1. Correspondence to:
 Dr Sarah Jones, Department of Epidemiology, Statistics and Public Health, University of Wales College of Medicine, Heath Park, Cardiff CF14 4XN, UK;
 jonessj3{at}cardiff.ac.uk

Abstract

Aim: To determine whether narrative information in emergency department surveillance systems can be systematically interrogated to improve our understanding of the causes of injury.

Methods: Screening algorithms for location, intent, and activity were developed from structured analysis of narrative data from 98 999 records. The algorithms were then tested on a 50 000 record database containing entries in both of the two narrative fields. A proxy gold standard was defined as the total extract using both code and narrative. Sensitivity and specificity of the emergency department coding and narrative algorithms was calculated.

Results: The proportion of records carrying an informative emergency department code was higher in records containing narrative—the percentage of causes coded “not know” dropped by 28.3%. The sensitivity of coded data varied from 42% to 98 % and from 33% to 99% for narrative data. Narrative analysis increased the percentage of home injuries identified by 19%, assaults by 26%, and rugby injuries by 137%.

Conclusions: Using a small amount of narrative is a practical and effective means of developing more informative injury causation data in an emergency department based surveillance system. It allows for internal validation of the codes and for the identification of emerging hazards without adding more “tick boxes” or further burdening data entry clerks.

  • injury causes
  • screening
  • surveillance data
  • AWISS, All Wales Injuries Surveillance System
  • CHIRPP, Canadian Hospitals Injury Reporting and Prevention Program
  • HASS, Home Accidents Surveillance System
  • LASS, Leisure Accidents Surveillance System
  • RTA, road traffic accident
  • injury causes
  • screening
  • surveillance data
  • AWISS, All Wales Injuries Surveillance System
  • CHIRPP, Canadian Hospitals Injury Reporting and Prevention Program
  • HASS, Home Accidents Surveillance System
  • LASS, Leisure Accidents Surveillance System
  • RTA, road traffic accident
View Full Text

Statistics from Altmetric.com

The need to reduce injuries has led to the development of surveillance systems to fill the gap between local surveys and national morbidity or mortality statistics.1 Furthermore, such data rarely reflect local conditions. Survey data that satisfy the “local” criteria are likely to be too expensive for widespread adoption.

In contrast, injury surveillance systems in emergency departments that collate routinely collected computerised patient management data yield large amounts of data. These data usually include the nature of the injury, along with some causation data, but often fall short of including data needed for prevention.

Some emergency department based systems (for example, CHIRPP, the Canadian Hospitals Injury Reporting and Prevention Program; HASS, the Home Accidents Surveillance System; and LASS, the Leisure Accidents Surveillance System) are separate from routine patient management. HASS and LASS use dedicated staff to collect detailed information from patients. This approach can be labour intensive and often can only sample a proportion of all relevant attendees.

Patient management data tend to collect all factors related to the causes of injury—locations, mechanisms, or activities—in one field, with coding carried out by staff. However, pressures of time or a lack of appropriate codes results in many injuries being coded as “other”.

More causative information using such systems may be obtained if narrative information is included. Narrative provides meaning and context in medicine2 and electronic text searching can be applied to surveillance.3 Collecting a detailed description of the injury incident may not be possible, but even a short line of text can be informative.

In January 1999 emergency department records received from participating hospitals by the All Wales Injuries Surveillance System (AWISS)4 included injury incident narrative in two short fields. A review of the two narrative fields—“how and where did the injury happen” and “what is the injury”—suggested that emergency department based coding was underestimating some injuries, for example, school and sports, because the existing categorical structure combined locations and activities but only allowed one entry. This prompted us to explore the possibility of analysing routinely collected narrative information. The aim was to determine whether structured, automated analysis of narrative is possible and, if so, if this improved the completeness and accuracy of causation coding.

METHODS

The AWISS is a population based surveillance system currently covering 80% of the 2.9 million population of Wales. It collates routinely collected patient management data from 13 of the 17 emergency departments in Wales centralising this information to facilitate local and regional injury prevention efforts.4 However, AWISS only collects limited information about causes and the inclusion of narrative in the emergency department record seemed an appropriate tool for improving the quality of causation data.

Development of routine analysis procedures

We developed algorithms for the automated analysis of narrative data and tested them on a MS Access dataset of 98 999 records covering January 1999 to June 2000. The intention was to supplement, not replace, causation codes and to permit automatic coding of key terms within the narrative to facilitate rapid analysis. The process is described here for four locations (home, work, school, traffic (road traffic accident, RTA)), assault injuries, and the two most injurious sports in Wales, rugby and soccer. These causation codes were copied from the AWISS source field into one of three new fields in the same record and headed “location, activity, and intent”. For example, home, school, work, and RTA codes were pasted into the location field; rugby and soccer into the activity field, and assault into the intent field.

We then validated these codes against the narrative with modifications made in “new” fields, where necessary. Data in the “incident type” and “patient group” field were not altered. Manual addition of other causation codes was also carried out, for example, “fell down stairs at home”, originally coded as a home injury, was supplemented by “stairs”. “Pedestrian knocked over by van”, originally coded only as an RTA, was enhanced to “pedestrian RTA”. “Playing rugby at school”, originally coded as a school injury, was also activity coded as “rugby”.

The process began using only the causation codes listed. However, the narrative soon suggested new items for inclusion. For example, where “bed” was included in the narrative the injury was recorded as having occurred at home.

Once the entire dataset had been validated and recoded, we generated a list of all records, including the narrative fields. We then identified keywords and generated screening algorithms for automated analysis.

These algorithms were a series of keywords entered as *keyword* which extracts the term from anywhere in the narrative string. The “home” algorithm was the most complex, using over 80 search terms, but covering all rooms as well as home repair activities. The “rugby” and “soccer” algorithms used four search terms each.

Semantics and precise use of language are important in correctly understanding causes. “Knocked” usually indicates individuals bumping into objects and injuring a specific body part, but “knocked over” may be a pedestrian injured in an RTA. “Kick” and “bite” alone are relatively uninformative, but “kicked” and “human bite” indicate assault injuries.

Testing routine analysis procedures

These algorithms were tested on a random sample of 50 000 records not used in the developmental phase, with narratives in both fields. Causation fields were added in the same way as during development. After extraction, codes were automatically entered into the causation fields and records manually screened to identify false positives. The improved dataset of 50 000 test records was then analysed. Comparison between this dataset of all narrative records and one containing no narrative was carried out. Sensitivity and specificity were calculated based on the assumption that the total coding and narrative extraction represents the gold standard. Increased record capture was described as “added value” and measured as the percentage of records added to the original number after narrative extraction.

RESULTS

The inclusion of narrative added to the proportion of informative causation codes. In the dataset containing no narrative, 67.5% were coded as “other” or “not known” compared with 49.5% in the dataset containing narrative in both fields. Increased “capture” of records occurred for each of the four main injury locations (table 1).

Table 1

Comparison of incident location coding in two datasets: one supplied to AWISS without any incident narrative, the other containing incident narrative in addition to standard codes

Sensitivity and specificity of the categorical codes and narrative derived codes were compared (table 2). Excepting work injuries, neither emergency department coding nor narrative only codes have high levels of sensitivity. For workplace injuries, however, emergency department coding was almost three times more sensitive than narrative because most workplace injuries are listed under the name of the business and are not suited to keyword extraction. For school, rugby, and soccer injuries, the narrative additions were more sensitive than emergency department code alone because they permitted recording both locations and activities. Narrative only coding led to lower specificity than categorical only coding of home injuries because potential false positives are extracted to maximise the algorithm sensitivity.

Table 2

Sensitivity and specificity of coded information and narrative derived coding when compared with a “gold standard” of total records extracted by both coding and narrative derived codes

Sensitivity and specificity analysis cannot demonstrate the full, additional value of narrative analysis. The total number of correct records extracted by the combination of emergency department coding and narrative analysis was calculated as: total records extracted by code alone + total records extracted by code and narrative + total records extracted by narrative alone. The percentage of records extracted by the narrative alone is shown in table 3.

Table 3

Breakdown of the number of records extracted by code alone and by narrative derived code alone, demonstrating the “added value” provided by narrative data extraction

The value added by the narrative was greatest for causes where the algorithm was most sensitive: school (14.7% added), rugby (137.2% added), and soccer (86.8% added). There were also increases in record capture for all other listed causes.

DISCUSSION

The use of structured narrative analysis to describe the circumstances of injury can substantially improve the completeness and quality of injury surveillance data. These analyses better describe the injury problem by increasing the number of records with specific locations or activities and by the breadth of activities recorded. Narrative is not an alternative to emergency department codes, but supplements and validates this information. It “adds value” to an emergency department dataset and although a “gold standard” would ideally be created using the original paper record or patient interviews, such an approach is often impractical. However, the proxy gold standard used here—taking the total number of records extracted by both coding and narrative—seems an appropriate alternative. The sensitivities, specificities, and added value demonstrate substantial improvements in data quality, particularly with respect to better understanding the causes of injury.

Adding tick boxes is not practical for improving causation data. It adds to the number of items that staff must consider and the location of a particular code may not be immediately obvious. Clerks also have difficulties where a record could fit more than one category because hierarchies are rarely defined and would often be inappropriate. Furthermore, our analysis of AWISS data has shown that hospitals supplying the fewest fields have the best data completion, averaging above 80%. As the number of fields increases, the quality of data drops markedly. Narrative analysis identifies hazards without adding to the burden upon clerks and medical staff and without compromising dataset quality. Decision making by clerks is reduced to entering the patient's description of the injury incident.

This is the beginning of a process to improve causation data collection with the minimum of economic and staffing costs to a population based surveillance system. These methods would be relatively easy to apply to other emergency department based systems.

REFERENCES

View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.