What are Looped Transformers? Explained clearly

So there are basically two ways people make a LLM "smarter":
1. give it more parameters (a bigger brain),
2. or, give it more compute/data to train on (a longer education).

Both work, but both are expensive!

The ultimate motive of all types of research is to maximize objectives under resource constraints. In that light, we would rather be chasing a tempting third option: reuse the parameters you already have, instead of buying new ones.

The idea has led the world to discover/invent a Looped Transformer.

A looped transformer takes a section of the model and run the same input through it multiple times in a loop, letting it "think" a bit more each pass, almost like re-reading a paragraph a few times to understand it better. If this works, you could get the intelligence benefits of a much bigger model while only storing the weights of a small one.

Instead of training 100 transformer layers, you might train 25 layers, but loop over them 4 times.

The forward pass latency and the number of FLOPs remain similar, but the number of weights decrease 4x.

Effectively, you have trained your weights to be polymorphic - the same weights are capable of reflecting and iterating over past versions of its own output.

Looped models have done surprisingly well on reasoning-heavy tasks, in-context learning, and abstract puzzle-solving benchmarks like ARC-AGI and Sudoku.

The earliest Looped Transformer

One of the most cited papers is this little known 2018 ICLR paper on "Universal Transformers"

In Universal Transformers we take the standard Transformer and replace its fixed stack of N distinct layers with a single shared transition function applied recurrently across time steps.

Instead of processing depth as "layer 1 → layer 2 → ... → layer N," UT iteratively refines the representation of every position in parallel, applying the same weights at each recurrent step.

Some of these themes are present in classic RNNs (recurrent neural networks) as well, where a single module is reused multiple times to process data sequentially. Whereas RNNs looped over the entries in a sequence across time, UT (and Looped Transformers as well) processes each input token parallelly across time. The loop is only applied at the layer level.

UT got a lot of cool results but its lasting legacy is for introducing this crazy idea. It is now cited as the direct architectural ancestor of essentially every modern looped/recurrent-depth language model.

Why UT "failed" (in 2018)

There are several reasons why UT wasnt perceived as a bigger deal back then.

1. The compute-vs-parameters trade-off wasn't understood yet

Given an N× increase in compute, simply making the model N× bigger tends to beat looping it N times. In 2018-19, scaling laws (Kaplan et al., 2020) hadn't even been published, so the field had no formal framework for asking "is recurrent compute or parameter compute more valuable per FLOP?" UT was evaluated at matched parameter count, not matched training compute, which flatters recurrence (recurrent steps are "free" parameters) while hiding its true compute cost.

2. The timing was wrong

UT (2018) landed right as the field was entering the "scale is all you need" era. Think of all the work that came back then: BERT (2018), GPT-2 (2019), and soon GPT-3 (2020) demonstrated that brute-force parameter and data scaling of vanilla, easily-parallelizable Transformers delivered massive, predictable gains. Recurrent depth is fundamentally sequential - you cannot compute step t+1 until step t finishes. 3. Infrastructure wasn't ready

3. The gains were real but not dramatic enough

UT was more of a research paper than a full blown millions of dollars model. The results were cool but they did not feel huge enough to shift the paradigm back then, because the scale they operated in was small.

4. Advances in tech

UT was tested only as a dense architecture. Modern LTs work with knowledge of 7 more years of dedicated research and funds. We got Mixture of Experts models now, more advanced sparse attention methods (like DSA, and more recently CSA).

UT was also an encoder-decoder architecture. In current times, all LMs are decoder only architectures.

5. Compute money

GPUs in 2026 are way more capable than in 2019. Investors are also more likely to invest in cutting edge AI research right now than back in 2019 (which is kinda surprising given most of the backbone of modern AI was arguably built from 2017-2020).

The Modern Looped Transformer (Latent Thinking)

Universal Transformer's core idea was: apply the same self-attention + transition block repeatedly, refining the representation of every token in parallel, with per-token halting deciding when each symbol is "done."

The modern Looped Transformer keeps the same weight-tying thing alive but reframes almost everything else around what a decoder-only, billion-to-trillion-parameter causal language model needs: efficient training at scale, causal (not encoder-decoder) computation, etc.

Instead of looping the entire network including embedding and output layers, modern looped LLMs split the network into three functional groups:

Because backpropagating through dozens of recurrent iterations is expensive (each step requires storing activations), modern implementations typically use truncated backpropagation through the recurrence. Basically, gradients are only propagated through a random subset of the most recent iterations rather than the full unrolled chain, keeping memory and compute tractable while still training the model to converge over many steps.

You will find more about recurrent depth in a paper like: Huginn/Geiping et al. 2025 (https://arxiv.org/pdf/2502.05171)

How much to loop? (Sampling vs Halting)

This is the most important departure from UT. Universal Transformer used something called ACT (Adaptive Computation Time) — a learned per-token halting probability that decided, symbol by symbol, when to stop refining.

Modern looped LLMs mostly drop this in favour of something simpler and more scalable: the loop count r is randomly sampled during training rather than learned.

This trains the model to be robust to whatever depth it is unrolled to at inference time, so at test time you can simply choose rr as a knob to trade off compute for quality!

More iterations generally means better reasoning, with diminishing returns past some point.

Whats next

If you loop a model N times, your training compute roughly multiplies by N-x too. Recurrence isn't free. Under a fixed compute budget, prior work showed that simply making a vanilla Transformer N× bigger tends to beat looping a smaller model N times. The challenge for researchers in this area is to overcome this constraint!

You can also read all these papers on Paper Breakdown: paperbreakdown.com

Start with the Recurrent Depth paper:
https://paperbreakdown.com/abs/2502.05171

And end with the latest Loopies paper: https://paperbreakdown.com/abs/2607.16051