Reproducing Statistics Performed in Published Randomized Controlled Trials

Authors

Naila H Dhanani
Oscar A Olavarria
Cynthia S Bell
Julie L Holihan
Mike K Liang

Publication Date

12-1-2022

Journal

Journal of General Internal Medicine

Abstract

INTRODUCTION: Randomized controlled trials (RCT) represent evidence at the lowest potential risk for bias. Clinicians in all specialties depend upon RCTs to guide patient care. Issues such as statistical discordance, or reporting statistical results that cannot be reproduced, should be uncommon. Our aim was to confirm the statistical reproducibility of published RCTs.

METHODS: PubMed was searched using "randomized controlled trial." Studies were selected using a random number generator. Studies were included if the primary outcome could be reproduced using the data and statistical test reported in the manuscript. The reproduced p-value from our analysis and the published p-value were compared. Primary outcome was the number of studies that reported p-values that differed in statistical significance (crossed p-value=0.05) from the reproduction analysis. Assuming an alpha of 0.05, a beta of 0.80, an estimated rate of statistical discordance of 5% for RCTs, a total of at least 568 studies were required.

RESULTS: Overall, 572 RCTs were selected involving six specialties. Of these, 45% were positive (p

CONCLUSION: Less than 5% of published RCTs reported a discordant p-value that crossed the "p=0.05" threshold. Although the occurrence is uncommon, the existence of even one RCT publishing nonreproducible results is concerning. Future studies should seek to identify why some RCTs report discordant statistics and how to prevent this from occurring.

Keywords

Humans, Randomized Controlled Trials as Topic, Research Report, Bias

Comments

PMID: 36163528