Perhaps once he has had enough of complaining about advertising-equivalent exposure and Britney Spears’ breasts, Ben Goldacre could take a look at this article.
The lengthly title, Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols belies the simple and slightly alarming findings.
This “research about research” considered seventy published randomised controlled trials, comparing the protocols and the corresponding publications of results. The authors considered the proportion of protocols that did not provide information about sample size calculations and statistical methods, and the proportion of trials with discrepancies between information presented in the protocol and the publication.
The authors find numerous such discrepancies in the protocols and resultant publications, from the framework of the study to the methods of statistical analysis chosen.
Does this matter? Well, yes – different statistical methods explore the raw results in different ways, and can give different impressions of the outcomes. If these statistical methods are not specified in advance, choosing which tests to include could potentially be influenced by explaratory analyses of the data. The authors end their piece with a call for documentation of sample size calculations and full plans for data analysis before a trial gets underway, and faithful adherence to this published protocol, or necessary amendments to be explicitly described.
Only with fully transparent reporting of trial methods and public access to protocols can the results be properly appraised, interpreted and applied to care of patients.
Just as Richard wonders if anyone takes that junk seriously, we can question whether clinicians (and researchers, journalists and the public) read, or will begin to read, reports of randomised controlled trials with the scrutiny that this article suggests is warranted.
—
A.-W. Chan, A. Hrobjartsson, K. J Jorgensen, P. C Gotzsche, D. G Altman (2008). Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols BMJ, 337 (dec04 1) DOI: 10.1136/bmj.a2299
