Article Text

Download PDFPDF
Evaluation of text mining to reduce screening workload for injury-focused systematic reviews
  1. Melita J Giummarra1,2,
  2. Georgina Lau1,
  3. Belinda J Gabbe1
  1. 1Epidemiology and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
  2. 2Caulfield Pain Management and Research Centre, Caulfield Hospital, Caulfield, Victoria, Australia
  1. Correspondence to Dr Melita J Giummarra, Epidemiology and Preventive Medicine, Monash University, Melbourne, VIC 3004, Australia; melita.giummarra{at}monash.edu

Abstract

Introduction Text mining to support screening in large-scale systematic reviews has been recommended; however, their suitability for reviews in injury research is not known. We examined the performance of text mining in supporting the second reviewer in a systematic review examining associations between fault attribution and health and work-related outcomes after transport injury.

Methods Citations were independently screened in Abstrackr in full (reviewer 1; 10 559 citations), and until no more citations were predicted to be relevant (reviewer 2; 1809 citations, 17.1%). All potentially relevant full-text articles were assessed by reviewer 1 (555 articles). Reviewer 2 used text mining (Wordstat, QDA Miner) to reduce assessment to full-text articles containing ≥1 fault-related exposure term (367 articles, 66.1%).

Results Abstrackr offered excellent workload savings: 82.7% of citations did not require screening by reviewer 2, and total screening time was reduced by 36.6% compared with traditional dual screening of all citations. Abstrackr predictions had high specificity (83.7%), and low false negatives (0.3%), but overestimated citation relevance, probably due to the complexity of the review with multiple outcomes and high imbalance of relevant to irrelevant records, giving low sensitivity (29.7%) and precision (14.5%). Text mining of full-text articles reduced the number needing to be screened by 33.9%, and reduced total full-text screening time by 38.7% compared with traditional dual screening.

Conclusions Overall, text mining offered important benefits to systematic review workflow, but should not replace full screening by one reviewer, especially for complex reviews examining multiple health or injury outcomes.

Trial registration number CRD42018084123.

  • text mining
  • systematic reviews
  • injury
  • road trauma
  • transport injury
  • research methods
View Full Text

Statistics from Altmetric.com

Footnotes

  • Contributors MJG designed the study, conducted the literature search, screened citations and full-text articles, analysed the performance of Abstrackr and text mining, and prepared the first and final versions of the manuscript. GL screened citations and full-text articles, and assisted in the interpretation of the results and drafting of the manuscript. BJG assisted in the design of the study, interpretation of the results and drafting of the manuscript.

  • Funding This research was funded by an Australian Research Council (ARC) Discovery Early Career Research Award to MJG (DE170100726). BJG was supported by an Australian Research Council Future Fellowship (FT170100048).

  • Competing interests None declared.

  • Patient consent for publication Not required.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.