The "reasoning doesn't exist" Apple paper drives me crazy. Take...
Kevin A. Bryan@Afinetheorem
16 views
Jun 09, 2025
~2 min read
Advertisement
1
The "reasoning doesn't exist" Apple paper drives me crazy. Take logic puzzle like Tower of Hanoi w/ 10s to 1000000s of moves to solve correctly. Check first step where an LLM makes mistake. Long problems aren't solved. Fewer thought tokens/early mistakes on longer problems. 1/11
2
But fundamental problem is that there are foundation *models*, and then there is *the way we train these models to cutoff thought*. Imagine you say "hello". Try it on o3 and 4o. o3 takes longer because it "thinks" about other things you could mean. Unnecessary in this case. 2/x
3
ALL LLMs just predict next token (~word) based on distribution of words in training data plus reinforcement learning. "Thinking models" just RL in "thinking tokens" to check assumpions more carefully before responding. If you don't think that can create "reasoning", fine. 3/x
4
The trick here is "how long should you think to respond to a problem". We as humans realize sometimes we need to write things down (counting a large set, multiplying big numbers), sometimes we answer write away, sometimes that is a mistake. We "RL ourselves" on this. 4/x
5
But if you tell me to solve a problem that would take me an hour of pen and paper, but give me five minutes, I'll probably give you an approximate solution or a heuristic. THIS IS EXACTLY WHAT FOUNDATION MODELS WITH THINKING ARE RL'D TO DO. 5/x
6
As a matter of underlying statistical properties of a transformer, we can of course program an LLM to spit out millions of tokens in response to "good evening", and RL it to iterate creatively on all sorts of possible interpretations, then collate, then brainstorm more, etc. 6/x
7
When the models *don't* do that, not b/c they *can't*, it's because we use post-training to stop them from doing something so crazy. This does mean that in some cases (e.g., "A boy goes to the hospital" h/t @goodside below), it *should* think longer. 7/x x.com/Afinetheorem/s…
8
We know from things like Code with Claude and internal benchmarks that performance strictly increases as we increase in tokens used for inference, on ~every problem domain tried. But LLM companies can do this: *you* can't b/c model you have access to tries not to "overthink". 9/x
9
The team on this paper are good (incl. Yoshua Bengio's brother!), but interpretation media folks give it is just wrong. It 100% does not, and can not, show "reasoning is just pattern matching" (beyond trivial fact that all LLMs do nothing more than RL'd token prediction...) 10/x
10
You can read the setup here: machinelearning.apple.com/research/illus… Paper is very short so I think what I'm saying above is essentially uncontroversial. 11/11
11
PS - This is a nice writeup where the mathematicians who provided FrontierMath novel questions were asked to analysis o3-mini's (not even today's frontier model, to be clear!) "reasoning". They did *not* believe it was pure regurgitation of heuristics.
Epoch AI@EpochAIResearch
How do reasoning models solve hard math problems?
We asked 14 mathematicians to review o3-mini-high’s raw, unsummarized reasoning traces on 29 FrontierMath problems. Here’s what they found:
We asked 14 mathematicians to review o3-mini-high’s raw, unsummarized reasoning traces on 29 FrontierMath problems. Here’s what they found:
12
To briefly follow up on the argument above:
Kevin A. Bryan@Afinetheorem
Just to clarify on the Apple reasoning paper, consider the simple problem "Output x(i)=2(i)+2, for i=0...10000. don't use code or search. State first 10000 outputs. Be COMPLETE. BE SURE TO STATE THE FULL LIST!" Every LLM can do this. None of them WILL do it. 1/4
