NEW PAPER 21,041 clinical trials were analyzed to determine whether...
Kevin Bass@kevinnbass
6 views
Jun 30, 2025
~3 min read
1
NEW PAPER
21,041 clinical trials were analyzed to determine whether they followed basic principles of experimental design and reporting
These studies are the core scientific pillar informing the care that patients receive
The results are shocking👇
21,041 clinical trials were analyzed to determine whether they followed basic principles of experimental design and reporting
These studies are the core scientific pillar informing the care that patients receive
The results are shocking👇
2
Fewer than 10% of published studies even explained how patients were assigned to each group
Since these are RANDOMIZED controlled trials, this is hard to fathom
There is no information provided about the most important design feature in >90% of trials
Since these are RANDOMIZED controlled trials, this is hard to fathom
There is no information provided about the most important design feature in >90% of trials
3
Fundamentally, this means we don't know if we can really trust the results in >90% of randomized controlled trials
This is because we do not have any information about how these results were actually obtained
This is because we do not have any information about how these results were actually obtained
4
Were patients really assigned randomly? What biases were there? We don't know.
5
It gets worse
Among the biggest blockbuster studies that have most influenced modern medical practice, fewer than 20% explained how patients were assigned
Among the biggest blockbuster studies that have most influenced modern medical practice, fewer than 20% explained how patients were assigned
6
And, even though guidelines for reporting this basic information were published in 1996:
The biggest blockbuster studies published in the past 10 years omitted this information more than 75% of the time
The biggest blockbuster studies published in the past 10 years omitted this information more than 75% of the time
7
For basic reporting of ALL trial design parameters:
Trials that fell under formal FDA oversight did WORSE
FDA trials reported only about 61% of basic trial details
Unregulated trials reported 67%
Trials that fell under formal FDA oversight did WORSE
FDA trials reported only about 61% of basic trial details
Unregulated trials reported 67%
8
Trials having something called a "data-monitoring committee" didn't do any better
Papers with a committee actually scored several percentage points lower on reporting trial details than trials without one
Papers with a committee actually scored several percentage points lower on reporting trial details than trials without one
9
Trials published in prestigious journals like Lancet, NEJM, or BMJ only did slightly better than papers published in the least prestigious journals
They averaged 67% of the basic trial details
Bottom quarter: 59%
They averaged 67% of the basic trial details
Bottom quarter: 59%
10
In other words, a NEJM RCT was almost as likely to omit basic information about study design as an RCT published in an obscure scientific journal
11
As mentioned, the randomization method was reported in only 9.7 % of papers
But other critical methodological features were also routinely omitted:
Allocation-concealment: 15.3 %
Publicly published protocol: 2.22 %
Any discussion of external validity discussion: 1.6%.
But other critical methodological features were also routinely omitted:
Allocation-concealment: 15.3 %
Publicly published protocol: 2.22 %
Any discussion of external validity discussion: 1.6%.
12
Without knowing these, we cannot know whether to really trust the trial findings
These are all CORE methodological features in randomized controlled trials: almost universally omitted from published papers
These are all CORE methodological features in randomized controlled trials: almost universally omitted from published papers
13
There has been some improvement over time
From 1964-90, 73% of details were missing
Today, it's 44%
From 1964-90, 73% of details were missing
Today, it's 44%
14
The best performing fields were urology/nephrology and critical care
These reported on average 63% of basic details
The worst field? Pharmacology at 35%
That's right
Drug trials did the worst
These reported on average 63% of basic details
The worst field? Pharmacology at 35%
That's right
Drug trials did the worst
15
The set of studies analyzed overlaps heavily with Cochrane CENTRAL
This means that under-reported trials are used into evidence syntheses that feed clinical guidelines
This means that under-reported trials are used into evidence syntheses that feed clinical guidelines
16
In other words, medical care is determined by evidence from trials we do not know we can trust
Because these trials do not provide the basic information that allow us to determine whether the findings have scientific integrity
Because these trials do not provide the basic information that allow us to determine whether the findings have scientific integrity
17
The current evidence base for medical care is therefore determined by studies whose findings we cannot be confident about
18
The majority of modern medical science essentially boils down to "trust me, bro" with no possibility of external verification
19
This should be incredibly alarming to anyone who has to see a doctor
20
Modern medical science is in desperate need of reform
21