How do reasoning models solve hard math problems? We asked 14...
Epoch AI@EpochAIResearch
118 views
Jun 09, 2025
~2 min read
Advertisement
1
How do reasoning models solve hard math problems?
We asked 14 mathematicians to review o3-mini-high’s raw, unsummarized reasoning traces on 29 FrontierMath problems. Here’s what they found:
We asked 14 mathematicians to review o3-mini-high’s raw, unsummarized reasoning traces on 29 FrontierMath problems. Here’s what they found:
2
o3-mini-high is extremely knowledgeable, and it’s not pure memorization. It fairly reliably invokes relevant techniques and results from the mathematical literature, even when problems were designed to obscure them.
3
It also does a lot of informal and heuristic-driven reasoning akin to a physicist, with many steps lacking rigorous justification. One mathematician described the model as a “vibes-based inductive reasoner”, employing rough, intuitive leaps rather than precise proofs.
4
But this lack of precision has its own issues. o3-mini could “cheese” through problems, guessing the final answer heuristically without going through most of the intended steps, and this happens to a moderate extent out of the 13 reasoning traces with correct solutions.
5
The biggest weakness was a lack of creativity and deep understanding. This is perhaps most aptly captured by a quote from one of the mathematicians:
6
Besides this, many of the reasoning traces contained hallucinations. One mathematician noted that "while [o3-mini] often recalls the names of relevant formulas, it fails to reproduce them accurately, frequently inserting placeholders like "(...)" where it cannot recall details."
7
Opinions varied on how much o3-mini reasons like a human mathematician. It shows human-like curiosity and informal exploration, yet it’s overly verbose, repetitive, and sometimes “neurotic” about confirming answers, repeating the exact same calculations over and over again.
8
Overall, we can pithily summarize o3-mini-high as an “erudite vibes-based reasoner that lacks the creativity and formality of professional mathematicians, and tends to be strangely verbose or repetitive”.
9
It also leads to some important questions. Why do models reason in this way, and how will this change over time? Will the use of synthetic data lead to substantially different reasoning traces? We hope to look more into these questions in the future.
10
We’d like to thank OpenAI for sharing o3-mini-high’s raw reasoning traces with us, and all the mathematicians who helped us analyze them.
11
You can read the full analysis here: epoch.ai/gradient-updat…
