Original Articles
Data collection instrument and procedure for systematic reviews in the guide to community preventive services1

https://doi.org/10.1016/S0749-3797(99)00122-1Get rights and content

Abstract

Introduction: A standardized abstraction form and procedure was developed to provide consistency, reduce bias, and improve validity and reliability in the Guide to Community Preventive Services: Systematic Reviews and Evidence-Based Recommendations (the Guide).

Data Collection Instrument: The content of the abstraction form was based on methodologies used in other systematic reviews; reporting standards established by major health and social science journals; the evaluation, statistical and meta-analytic literature; expert opinion and review; and pilot-testing. The form is used to classify and describe key characteristics of the intervention and evaluation (26 questions) and assess the quality of the study’s execution (23 questions). Study procedures and results are collected and specific threats to the validity of the study are assessed across six categories (intervention and study descriptions, sampling, measurement, analysis, interpretation of results and other execution issues).

Data Collection Procedures: Each study is abstracted by two independent reviewers and reconciled by the chapter development team. Reviewers are trained and provided with feedback.

Discussion: What to abstract and how to summarize the data are discretionary choices that influence conclusions drawn on the quality of execution of the study and its effectiveness. The form balances flexibility for the evaluation of papers with different study designs and intervention types with the need to ask specific questions to maximize validity and reliability. It provides a structured format that researchers and others can use to review the content and quality of papers, conduct systematic reviews, or develop manuscripts. A systematic approach to developing and evaluating manuscripts will help to promote overall improvement of the scientific literature.

Introduction

The independent, non-federal Task Force on Community Preventive Services (the Task Force) will make recommendations about health promotion and disease prevention interventions in the Guide to Community Preventive Services: Systematic Reviews and Evidence-Based Methods (the Guide).1 These recommendations will be based on systematic reviews of the evidence of effectiveness, other positive and negative effects of the interventions, applicability of the effectiveness information, economic evaluations and barriers to implementation of the interventions.2 Fifteen topics are currently being reviewed, and each chapter will cover a single topic and include reviews for 10–20 interventions.2, 3 A multidisciplinary team (i.e., the chapter development team) coordinates development of each chapter and consists of Task Force members, a coordinating scientist, and several topic experts.2 The chapter development team defines the scope and intent of each chapter and selects a set of interventions for inclusion in the chapter using predefined criteria.2 To evaluate the effectiveness of the intervention, the team conducts a systematic review of the scientific literature. The systematic review methods include: identifying the potential links between an intervention and relevant outcomes, using specific inclusion criteria to search for studies, evaluating effectiveness of the interventions, and evaluating the content and quality of each study.2

Conducting systematic reviews for development of the Guide involves multiple coordinating scientists and participants, reviews of interventions in highly variable topics (e.g., sexual behavior, cancer, motor vehicle occupant injuries), a range of intervention types (e.g., education, environmental change, policy development), and the inclusion of all types of comparative study designs (e.g., experimental studies with allocated control groups or observational studies with concurrent or historical control groups). These features of the development process have the potential for introducing inconsistency into the Guide. The use of a standardized data collection instrument and procedure is one way to reduce inconsistencies within and between chapters.

In this paper we describe the instrument and procedure used to collect and evaluate data from individual studies of intervention effectiveness, a key step in the methods used to develop the Guide. The form illustrates the Task Force’s approach to categorizing information about study design, content, and quality of the scientific literature. This approach will be useful to others for the purposes of reading the scientific literature, writing scientific manuscripts, designing evaluation studies or teaching epidemiology and evaluation methods.

Section snippets

Data collection instrument

In developing the data collection instrument, we considered its six main purposes:

  • Tracking the article review process. The form collects information needed to monitor the status of screening, reviewing and summarizing of each article by multiple reviewers.

  • Developing tables that summarize the body of evidence. The form captures detailed descriptive data about the intervention and evaluation; this data is used to develop summary evidence tables for each intervention.2, 4

  • Classifying other key

Data collection procedures

Data is collected from each study by two independent reviewers. If the reviewers report different information for a question, the chapter development team reconciles the two reviews. Members of the chapter development team, graduates of Masters of Public Health degree programs, doctoral candidates and physicians in preventive medicine training programs serve as reviewers. Selection of reviewers is based on experience in content areas, experience in conducting evidence-based reviews and

Discussion

Systematic reviews of four topics (tobacco use, physical activity, motor vehicle occupant injury and vaccine preventable diseases) in the Guide have been conducted using the data collection instrument and procedures. Over 400 papers with designs as varied as randomized controlled trials, time series studies and cross-sectional studies have been reviewed by more than 40 reviewers, all of whom have provided feedback and suggestions for improving the instructions and format of the form.

The

Acknowledgements

The authors gratefully acknowledge David Atkins, MD, MPH (Agency for Health Care Policy Research); Donna Higgins, PhD (CDC); David McQueen, PhD (CDC); Mike Stoto, PhD (George Washington University); and Steven Woolf, MD, MPH (Medical College of Virginia) for their helpful reviews, comments and suggestions for improvements to the data collection instrument.

References (16)

There are more references available in the full text version of this article.

Cited by (0)

1

The names and affiliations of the Task Force members are listed on page v of this supplement and at http://www.thecommunityguide.org

View full text