MASSIVE claim in this paper. AI Architectural breakthroughs can be...
Rohan Paul@rohanpaul_ai
97 views
Jul 27, 2025
~5 min read
Advertisement
1
MASSIVE claim in this paper.
AI Architectural breakthroughs can be scaled computationally, transforming research progress from a human-limited to a computation-scalable process.
So it turns architecture discovery into a compute‑bound process, opening a path to self‑accelerating model evolution without waiting for human intuition.
The paper shows that an all‑AI research loop can invent novel model architectures faster than humans, and the authors prove it by uncovering 106 record‑setting linear‑attention designs that outshine human baselines.
Right now, most architecture search tools only fine‑tune blocks that people already proposed, so progress crawls at the pace of human trial‑and‑error.
🧩 Why we needed a fresh approach
Human researchers tire quickly, and their search space is narrow. As model families multiply, deciding which tweak matters becomes guesswork, so whole research agendas stall while hardware idles.
🤖 Meet ASI‑ARCH, the self‑driving lab
The team wired together three LLM‑based roles. A “Researcher” dreams up code, an “Engineer” trains and debugs it, and an “Analyst” mines the results for patterns, feeding insights back to the next round. A memory store keeps every motivation, code diff, and metric so the agents never repeat themselves.
📈 Across 1,773 experiments and 20,000 GPU hours, a straight line emerged between compute spent and new SOTA hits.
Add hardware, and the system keeps finding winners without extra coffee or conferences.
AI Architectural breakthroughs can be scaled computationally, transforming research progress from a human-limited to a computation-scalable process.
So it turns architecture discovery into a compute‑bound process, opening a path to self‑accelerating model evolution without waiting for human intuition.
The paper shows that an all‑AI research loop can invent novel model architectures faster than humans, and the authors prove it by uncovering 106 record‑setting linear‑attention designs that outshine human baselines.
Right now, most architecture search tools only fine‑tune blocks that people already proposed, so progress crawls at the pace of human trial‑and‑error.
🧩 Why we needed a fresh approach
Human researchers tire quickly, and their search space is narrow. As model families multiply, deciding which tweak matters becomes guesswork, so whole research agendas stall while hardware idles.
🤖 Meet ASI‑ARCH, the self‑driving lab
The team wired together three LLM‑based roles. A “Researcher” dreams up code, an “Engineer” trains and debugs it, and an “Analyst” mines the results for patterns, feeding insights back to the next round. A memory store keeps every motivation, code diff, and metric so the agents never repeat themselves.
📈 Across 1,773 experiments and 20,000 GPU hours, a straight line emerged between compute spent and new SOTA hits.
Add hardware, and the system keeps finding winners without extra coffee or conferences.
2
📈 Across 1,773 experiments and 20,000 GPU hours, a straight line emerged between compute spent and new SOTA hits.
Add hardware, and the system keeps finding winners without extra coffee or conferences.
Add hardware, and the system keeps finding winners without extra coffee or conferences.
3
Examples like PathGateFusionNet, ContentSharpRouter, and FusionGatedFIRNet beat Mamba2 and Gated DeltaNet on reasoning suites while keeping parameter counts near 400M. Each one solves the “who gets the compute budget” problem in a new way, often by layering simple per‑head gates instead of a single softmax.
4
🔍 Patterns the agents uncovered
The chart compares how often each component shows up in 106 winning architectures versus 1,667 discarded ones.
Gating layers and small convolutions dominate both groups at roughly 14% and 12% usage, while staples like residual links and feature pooling follow close behind. Exotic pieces, such as physics‑inspired or spectral tricks, hardly appear in the successful set.
The pattern is clear, the top models lean on a tight, proven toolkit, whereas the larger pool experiments with a very long list of rare ideas that rarely pay off. In other words, focused refinement of well‑tested components beats wide exploration when the goal is higher benchmark scores and lower loss.
The chart compares how often each component shows up in 106 winning architectures versus 1,667 discarded ones.
Gating layers and small convolutions dominate both groups at roughly 14% and 12% usage, while staples like residual links and feature pooling follow close behind. Exotic pieces, such as physics‑inspired or spectral tricks, hardly appear in the successful set.
The pattern is clear, the top models lean on a tight, proven toolkit, whereas the larger pool experiments with a very long list of rare ideas that rarely pay off. In other words, focused refinement of well‑tested components beats wide exploration when the goal is higher benchmark scores and lower loss.
5
6
As to author credibility, they are mostly GAIR/SJTU (Shanghai Jiao Tong University) folks led by Pengfei Liu, a well-cited NLP professor with 20k+ citations.
7
The authors compare it to AlphaGo’s surprise "Move 37", because these AI‑born ideas push model architecture into territory humans had not explored.
Humans lack
(i) the raw throughput to generate and test the millions‑scale design variants needed to reach exotic corners of the search space and
(ii) the unbiased, memory‑perfect pattern‑mining that turns that torrent of results into new principles.
The AI loop overcomes both limits by trading human cognition for scalable computation, letting model architecture exploration expand into territory that was pragmatically out of reach for human researchers.
Humans lack
(i) the raw throughput to generate and test the millions‑scale design variants needed to reach exotic corners of the search space and
(ii) the unbiased, memory‑perfect pattern‑mining that turns that torrent of results into new principles.
The AI loop overcomes both limits by trading human cognition for scalable computation, letting model architecture exploration expand into territory that was pragmatically out of reach for human researchers.
8
below table shows that top‑performing models rely more on lessons drawn from earlier experiments and less on bold, never‑seen ideas.
In other words, experience‑driven tweaks and clear logical checks guide most breakthroughs, while outright originality plays a minor role.
i.e. the highest‑performing designs draw a larger share of their key components from the system’s own cross‑experiment analysis, not from recycled human papers.
Mining patterns across thousands of runs and synthesising them into new hypotheses is a cognitive workload humans simply cannot shoulder.
Look at the numbers. In the winning set, about 45% of the design choices reflect past experience, 49% come from systematic reasoning, and only 7% are brand‑new tricks. The less successful models lean a bit harder on originality, yet that extra novelty does not translate into better scores.
The takeaway is simple: the automated loop climbs fastest when it mines its growing history of runs, tests small hypotheses, and keeps what works. Purely novel components can still help, but they matter far less than a tight cycle of experiment, analysis, and incremental refinement.
In other words, experience‑driven tweaks and clear logical checks guide most breakthroughs, while outright originality plays a minor role.
i.e. the highest‑performing designs draw a larger share of their key components from the system’s own cross‑experiment analysis, not from recycled human papers.
Mining patterns across thousands of runs and synthesising them into new hypotheses is a cognitive workload humans simply cannot shoulder.
Look at the numbers. In the winning set, about 45% of the design choices reflect past experience, 49% come from systematic reasoning, and only 7% are brand‑new tricks. The less successful models lean a bit harder on originality, yet that extra novelty does not translate into better scores.
The takeaway is simple: the automated loop climbs fastest when it mines its growing history of runs, tests small hypotheses, and keeps what works. Purely novel components can still help, but they matter far less than a tight cycle of experiment, analysis, and incremental refinement.
9
ASI-Arch framework operates as a closed-loop system for autonomous architecture discovery, structured around a modular framework with three core roles. The Researcher, the Engineer and the Analyst module.
Step 1 – Researcher proposes a brand‑new blueprint
An LLM named Researcher reads the memory of past experiments, mixes in ideas mined from human papers, then writes the motivation plus working PyTorch code for a fresh architecture.
Step 2 – Novelty and sanity checks guard the queue
Before training starts, a similarity search confirms the idea is not a rerun, and automated code checks verify sub‑quadratic complexity and proper masking. If anything fails, the agent rewrites the model.
Step 3 – Engineer trains and self‑fixes the model
The Engineer agent launches training inside a real environment, catches crashes or slow runs, inspects the error log, patches its own code, and retries until the run completes.
Step 4 – Composite fitness score picks winners
After training, the system averages three signals—sigmoid‑scaled loss gain, benchmark gain, and an LLM judge’s quality rating—into one number that decides which designs survive.
Step 5 – Analyst extracts lessons and updates memory
An Analyst agent studies the logs, writes a short report on what helped or hurt, and stores both raw data and insights so the next Researcher round avoids old mistakes and steals good tricks.
Step 6 – Explore small, verify big
The loop first tests thousands of tiny models to cover ground fast, then scales the most promising few to full size for a final check, saving GPU hours for likely winners.
Step 1 – Researcher proposes a brand‑new blueprint
An LLM named Researcher reads the memory of past experiments, mixes in ideas mined from human papers, then writes the motivation plus working PyTorch code for a fresh architecture.
Step 2 – Novelty and sanity checks guard the queue
Before training starts, a similarity search confirms the idea is not a rerun, and automated code checks verify sub‑quadratic complexity and proper masking. If anything fails, the agent rewrites the model.
Step 3 – Engineer trains and self‑fixes the model
The Engineer agent launches training inside a real environment, catches crashes or slow runs, inspects the error log, patches its own code, and retries until the run completes.
Step 4 – Composite fitness score picks winners
After training, the system averages three signals—sigmoid‑scaled loss gain, benchmark gain, and an LLM judge’s quality rating—into one number that decides which designs survive.
Step 5 – Analyst extracts lessons and updates memory
An Analyst agent studies the logs, writes a short report on what helped or hurt, and stores both raw data and insights so the next Researcher round avoids old mistakes and steals good tricks.
Step 6 – Explore small, verify big
The loop first tests thousands of tiny models to cover ground fast, then scales the most promising few to full size for a final check, saving GPU hours for likely winners.
10
a comment on this paper on reddit.
and I agree with him. 🙂
and I agree with him. 🙂
11
Rohan Paul@rohanpaul_ai
And @leopoldasch said it in that famous Situational awareness piece.
Self-improving AI is the future. Humans will no more be the constraint.
Compute/GPUs/Electricity is the ONLY constraint.
"II. From AGI to Superintelligence: the Intelligence Explosion
AI progress won’t stop at human-level. Hundreds of millions of AGIs could automate AI research, compressing a decade of algorithmic progress (5+ OOMs) into ≤1 year. We would rapidly go from human-level to vastly superhuman AI systems. The power—and the peril—of superintelligence would be dramatic.
III. The Challenges
IIIa. Racing to the Trillion-Dollar Cluster
The most extraordinary techno-capital acceleration has been set in motion. As AI revenue grows rapidly, many trillions of dollars will go into GPU, datacenter, and power buildout before the end of the decade. The industrial mobilization, including growing US electricity production by 10s of percent, will be intense. "
Self-improving AI is the future. Humans will no more be the constraint.
Compute/GPUs/Electricity is the ONLY constraint.
"II. From AGI to Superintelligence: the Intelligence Explosion
AI progress won’t stop at human-level. Hundreds of millions of AGIs could automate AI research, compressing a decade of algorithmic progress (5+ OOMs) into ≤1 year. We would rapidly go from human-level to vastly superhuman AI systems. The power—and the peril—of superintelligence would be dramatic.
III. The Challenges
IIIa. Racing to the Trillion-Dollar Cluster
The most extraordinary techno-capital acceleration has been set in motion. As AI revenue grows rapidly, many trillions of dollars will go into GPU, datacenter, and power buildout before the end of the decade. The industrial mobilization, including growing US electricity production by 10s of percent, will be intense. "
