The role and interpretation of pilot studies in clinical research

https://doi.org/10.1016/j.jpsychires.2010.10.008Get rights and content

Abstract

Pilot studies represent a fundamental phase of the research process. The purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to be used in a larger scale study. The roles and limitations of pilot studies are described here using a clinical trial as an example. A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and implementation of the novel intervention.

A pilot study is not a hypothesis testing study. Safety, efficacy and effectiveness are not evaluated in a pilot. Contrary to tradition, a pilot study does not provide a meaningful effect size estimate for planning subsequent studies due to the imprecision inherent in data from small samples. Feasibility results do not necessarily generalize beyond the inclusion and exclusion criteria of the pilot design.

A pilot study is a requisite initial step in exploring a novel intervention or an innovative application of an intervention. Pilot results can inform feasibility and identify modifications needed in the design of a larger, ensuing hypothesis testing study. Investigators should be forthright in stating these objectives of a pilot study. Grant reviewers and other stakeholders should expect no more.

Introduction

Over the past several years, research funding agencies have requested applications for pilot studies that are typically limited to a shorter duration (one to three years) and a reduced budget using, for example, the NIH R34 funding mechanism (NIMH, 2010). Pilot studies play a key role in the development or refinement of new interventions, assessments, and other study procedures. Commonly, results from pilot studies are used to support more expensive and lengthier pivotal efficacy or effectiveness studies. Importantly, investigators, grant reviewers, and other stakeholders need to be aware of the essential elements, appropriate role, and exceptional strengths and limitations in the interpretation of pilot studies.

A pilot study is, “A small-scale test of the methods and procedures to be used on a larger scale…” (Porta, 2008). The fundamental purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to ultimately be used in a larger scale study. This applies to all types of research studies. Here we use the randomized controlled clinical trial (RCT) for illustration. Prior to initiating a full scale RCT an investigator may choose to conduct a pilot study in order to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and/or implementation of the novel intervention. A pilot study, however, is not used for hypothesis testing. Instead it serves as an earlier-phase developmental function that will enhance the probability of success in the larger subsequent RCTs that are anticipated.

For purpose of contrast, a hypothesis testing clinical trial is designed to compare randomized treatment groups in order to draw an inference about efficacy/effectiveness and safety in the patient population, based on sample results. The primary goal in designing such a study is to minimize the bias in the estimate of the treatment effect (Leon et al., 2006, Leon and Davis, 2009). That is, the trial is designed to ask the question, “Is the treatment efficacious, and if so, what is the magnitude of the effect?”. Features of RCTs that help us achieve this goal are randomized group assignment, double-blinded assessments, and control or comparison groups.

This manuscript will focus on pilot studies, those used to shape some, but not all aspects of the design and implementation of hypothesis testing clinical trials. It is the feasibility results, not the efficacy or safety results, that inform subsequent trials. The objective of this manuscript is to elaborate on each of these points: efficacy, safety, and feasibility. We discuss both the design of a pilot study and the interpretation and application of pilot study results. What is discussed here applies to pilot studies, feasibility studies and proof of concept studies, terms that have been used somewhat interchangeably in the literature and henceforth are referred to here as “pilot studies”.

Section snippets

What a pilot study can do: assess feasibility

Pilot study results can guide in the design and implementation of larger scale efficacy studies. There are several aspects of RCT feasibility that are informed by conducting a pilot study. A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, and implementation of the novel intervention and each of these can be quantified (Table 1). Study components that are deemed infeasible or unsatisfactory should be modified in the subsequent

Sample size determination in designing a pilot study

A pilot study is not a hypothesis testing study. Therefore, no inferential statistical tests should be proposed in a pilot study protocol. With no inferential statistical tests, a pilot study will not provide p-values. Power analyses are used to determine the sample size that is needed to provide adequate statistical power (typically 80% or 90%) to detect a clinically meaningful difference with the specified inferential statistical test. However, power analyses should not be presented in an

Exceeding the limitations of a pilot study: what pilot studies cannot do

Although a pilot study will undoubtedly incorporate relevant outcome measures and can serve a vital role in treatment development, it is not, and should not, be considered a preliminary test of the intervention hypothesis. There are two fundamental reasons that hypothesis testing is not used in a pilot study: the limited state of knowledge about the methods or intervention in the patient population to be studied and the smaller proposed sample size.

Discussion

The primary role of a pilot study is to examine the feasibility of a research endeavor. For instance, feasibility of recruitment, randomization, intervention implementation, blinded assessment procedures, and retention can all be examined. Investigators should be forthright in stating these objectives of a pilot study and bravely accept the limitations of a pilot study. Grant reviewers should expect no more.

The choice of the intervention for a pilot study should be based on theory, mechanism of

Role of funding

Dr. Leon’s funding for this study was provided by grants from the National Institute of Mental Health (MH060447 and MH068638). Dr. Davis received research support from VA, Department of Defense, National Institute of Mental Health and Astra-Zeneca. None of these sources of funds had a further role in study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the paper for publication.

Contributors

Dr. Leon conducted the literature searches and analyses. Dr. Leon wrote the first draft of the manuscript. Drs. Leon, Davis and Kraemer made substantial contributions to subsequent drafts. All authors contributed to and have approved the final manuscript.

Conflict of interest

Dr. Leon has served on Independent Data Safety Monitoring Boards for Astra-Zeneca, Dainippon Sumitomo Pharma America, Pfizer. He served as a consultant to Cyberonics, Eli Lilly, FDA, NIMH, MedAvante Noven, Roche, Schering Plough, and Takeda. He has equity in MedAvante.

Dr. Davis served as a consultant to Eli Lilly, received speaker’s honorarium from Astra-Zeneca, and received research support from VA, Department of Defense, National Institute of Mental Health and Astra-Zeneca.

Dr. Kraemer reports

Acknowledgements

Portions of this manuscript were presented at the Convergence Workshop sponsored by the National Institute of Mental Health, June 9, 2008, Arlington, VA, at the annual meeting of National Institute of Mental Health, New Clinical Drug Evaluation Unit (NCDEU), July 1, 2009, Hollywood, FL and at the 5th Annual Scientific Meeting of the International Society for CNS Clinical Trials and Methodology (ISCTM), March 3–5, 2009, Arlington VA.

References (13)

There are more references available in the full text version of this article.

Cited by (0)

View full text