Preprint out today that tests o1-preview's medical reasoning...

In this case the title says it all:
Superhuman performance of a large language model on the reasoning tasks of a physician
Link: arxiv.org/abs/2412.10849
A 🧵⬇️
x.com/omarsar0/statu…
The first is that we have reached benchmark saturation with LLMs in medical reasoning. We spent 100s of physician hours developing and grading these benchmarks and for $20 a month an AI model crushes them.

The fact that there are not 30 different benchmarks from different organizations in medicine, in law, in advice quality, etc. is a real shame, as people are using systems for these things, regardless.
And the second implication -- it's time for clinical trials.




