To address the limits of deep learning and avoid stalling, the...
François Chollet@fchollet
80 views
Aug 03, 2026
~2 min read
Advertisement
1
To address the limits of deep learning and avoid stalling, the field of AI started by applying patch (1), which started being demoed 9 months later in December 2024 and has now become completely ubiquitous. However, long term, it is simply inevitable that AI will move to patch (2).
François Chollet@fchollet
There are essentially two main options to remedy this:
1. Find ways to perform active inference, so that the model adapts its learned program in contact with a new data distribution at test time. Would likely lead to some meaningful progress, but it isn't the ultimate solution, more of an incremental improvement.
2. Change the training mechanism to something more robust than SGD, such as the MDL principle. This would pretty much require moving away from deep learning (curve fitting) altogether and embracing discrete program search instead (which I have advocated for many years as a way to tackle reasoning problems...)
1. Find ways to perform active inference, so that the model adapts its learned program in contact with a new data distribution at test time. Would likely lead to some meaningful progress, but it isn't the ultimate solution, more of an incremental improvement.
2. Change the training mechanism to something more robust than SGD, such as the MDL principle. This would pretty much require moving away from deep learning (curve fitting) altogether and embracing discrete program search instead (which I have advocated for many years as a way to tackle reasoning problems...)
2
Worth noting that to this day, base LLMs (no test time compute) *still* perform poorly on the ARC 1 benchmark from 2019 (on unseen tasks) despite continued massive scaling (~100,000x since 2019). We can say that without the switch to the test-time compute paradigm, AI would still not be capable of the kind of advanced reasoning that sota systems display today. Scaling the single-pass, static, next-token prediction paradigm of the GPT-2 through GPT-4 era was running into a capability asymptote. To bypass this plateau, test-time adaptation was a necessary evolutionary step.
3
I know it feels very tempting to dunk, and that's fair, but to be clear, my past criticism of base LLMs does not apply to TTA systems, in the same way that criticism of the limits of steam trains would not apply to current-day electrified bullet trains (even if they feel similar and use the same platform, the principles at work are substantially different). I have first updated my views in December 2024 -- by that point I no longer believed the LLM tech platform (with TTA) would stall.
If you want to assess how my past assessment of the limits of base LLMs has fared with hindsight, compare to current-day base LLMs (yes, my points still apply, even with a few years of scaling added).
If you want to assess how my past assessment of the limits of base LLMs has fared with hindsight, compare to current-day base LLMs (yes, my points still apply, even with a few years of scaling added).
4
My December 2024 takes were "there will be no wall", "if the only bottleneck is test-time search, we will see continued scaling in the future", and "the world is once again about to run out of GPUs"
To this day, I do expect post-training scaling and TTA scaling to continue to deliver. At least 10-100x from current levels.
To this day, I do expect post-training scaling and TTA scaling to continue to deliver. At least 10-100x from current levels.
François Chollet@fchollet
The limitations of specific techniques are predictable and correspondingly lead to plateaus for those techniques. But there is always the next technique, building on top of the pile that's already available.
There is enough research investment that there will be no wall.
There is enough research investment that there will be no wall.
