The Next AI Race Is About Inference Efficiency: Two Ways to Break the Memory Bottleneck
Should we bring weights closer to compute by moving them into SRAM, or improve silicon efficiency by giving up some general-purpose flexibility? Cerebras, Groq, Etched, and Google’s Frozen v2 occupy different positions along these two dimensions.
The AI industry’s center of gravity is shifting from training to inference. Training a large model even once is enormously expensive, but deploying it means continuously generating tokens in response to requests from countless users. Inference costs accumulate as models are used, and as extended reasoning and agentic services become more common, power consumption and response speed become increasingly important competitive advantages.
The problem is that inference performance is not determined by a chip’s compute capacity alone. Particularly during the decode stage of an LLM, memory reaches its limits before the compute units do. To understand the coming competition in inference chips, we need to look beyond FLOPS and first ask: “Where are the weights, and how far must they travel for every token?”
There are two broad approaches to this memory bottleneck.
One is to devote more silicon to SRAM, bringing weights closer to the compute units. The other is to reduce the flexibility the hardware must support, so that a greater share of the same silicon and power budget goes directly toward model computation, while unnecessary memory accesses and control overhead are reduced.
Cerebras dramatically increases SRAM capacity. Groq both expands SRAM and reduces general-purpose flexibility. Etched retains HBM as its large-capacity main memory while choosing a much more specialized datapath than a GPU. Google’s Frozen v2 goes a step further, seeking to embed the architecture of a particular model into silicon itself.
These are less competing camps than different choices along two dimensions: memory proximity and hardware flexibility.
The Decode Bottleneck Is Memory Bandwidth, Not Compute
To generate a single token for one request, a dense LLM must reread most of its model weights. Each parameter requires only about two floating-point operations—a multiplication and an addition. Yet the weight must travel from memory to the compute unit every time those operations are performed.
For example, a 70-billion-parameter model stored in FP8 has roughly 70 GB of weights. The H100 SXM has an HBM bandwidth of 3.35 TB/s, so even if we ignore every other cost, the theoretical upper limit for a single request is:
3.35 TB/s ÷ 70 GB ≈ 48 tokens/s
The H100’s enormous matrix-computing capacity is underutilized in this process. The compute units are capable of doing the work; they are simply waiting for the weights they need.
The traditional way GPUs mitigate this problem is through batching. If B requests are grouped together, the weights can be read once to generate B tokens. This reduces weight traffic per request to roughly 1/B.
But not all memory traffic decreases in the same way. Each request has its own KV cache. As batch size and context length grow, KV cache capacity requirements and read traffic grow with them. For a 70B-class model with a 32K context, the KV cache for a single request alone can reach roughly 5–10 GB, depending on precision and model architecture.
Batching therefore changes the shape of the memory bottleneck rather than eliminating it.
This is where the two paths for inference chips diverge.
The First Approach: Increase SRAM Capacity
SRAM is much faster than HBM and has lower latency. If weights reside in SRAM, there is no need to fetch tens of gigabytes from external memory for every token generated. The weights still have to be read, but the location and speed of those reads change.
The tradeoff is clear. SRAM has low storage density and is expensive. Providing the same capacity requires far more silicon. Ultimately, this approach buys bandwidth by using larger chips or more of them.
Cerebras: Turning a Wafer into Memory
Cerebras takes the most direct approach to this strategy. Instead of cutting a wafer into small chips as conventional semiconductor manufacturing does, it turns the entire wafer into one enormous processor.
The WSE-3 provides 44 GB of on-chip SRAM and 21 PB/s of memory bandwidth—thousands of times the H100’s HBM bandwidth. Large models are distributed across multiple wafers, but the central idea is to keep weights from shuttling between slow external memory and compute units for every token.
This architecture is particularly strong with small batches and long outputs. In Artificial Analysis’s measurements of gpt-oss-120B, Cerebras delivered approximately 1,700 tokens/s per user. Compared with GPU-based services, which generally deliver tens to hundreds of tokens per second, the difference is readily noticeable.
On the other hand, SRAM is an expensive place to store large amounts of KV cache. As the number of concurrent requests increases, inexpensive, high-capacity memory becomes more important, restoring the economic advantage of HBM-based systems when serving large batches. What Cerebras sells is less the cheapest token than a fast token that reduces the user’s waiting time.
Meanwhile, the WSE-4, unveiled today, takes this architecture a step further. It focuses on increasing compute density and memory bandwidth per wafer while reducing communication latency between wafers. Through these improvements, it aims to deliver meaningful gains over the WSE-3 in both single-request latency and large-scale inference throughput.
Groq: Choosing Both SRAM and Determinism
Groq also keeps weights in SRAM close to the compute units, but it scales differently from Cerebras. It connects large numbers of smaller chips and distributes the model across them.
Early Groq chips had only about 230 MB of SRAM. Instead, the compiler determines in advance the exact cycle in which each operation and data transfer will occur. There is no need to reorder instructions at runtime, wait for cache hits, or synchronize multiple cores.
Groq therefore invests silicon in SRAM while also giving up the flexibility of dynamic control found in general-purpose processors. Compared with Cerebras, its design moves further along the second dimension: reducing general-purpose flexibility.
The result is very low latency and predictable execution times for individual requests. In the same gpt-oss-120B measurements, Groq achieved approximately 480 tokens/s. That is slower than Cerebras, but considerably faster than typical GPU services.
The NVIDIA Groq 3 LPX, unveiled in 2026, extends this approach into a more practical configuration. Groq-family LPUs handle repetitive FFN and MoE computations, while Rubin GPUs handle prefill, attention, and large KV caches. Computation shared across all requests and data that grows with each request are assigned to different processors and memory systems.
This can be understood as a design that concentrates SRAM where it offers the most value, rather than one that eliminates the weaknesses of the SRAM approach.
The Second Approach: Reduce Flexibility to Reduce Memory Accesses
GPUs can handle a wide variety of models, graphics workloads, and scientific computations. That versatility is powerful, but it comes at a cost. Instruction schedulers, registers, cache hierarchies, branch handling, and synchronization mechanisms all consume silicon area and power to support flexibility.
The operations that actually recur in LLM inference are relatively limited: combinations of matrix multiplication, attention, normalization, activation functions, and tensor movement. Narrowing the supported operations and dataflows allows silicon previously devoted to general-purpose control to be reallocated to compute units and data movement paths.
This approach does not substantially change HBM’s physical bandwidth. Instead, it improves compute utilization and throughput per watt—factors that become more important once larger batches spread weight traffic across multiple requests.
Etched: Keeping HBM While Reducing GPU Flexibility
Etched uses HBM to store large amounts of model weights and KV cache. It also uses SRAM, but the architecture centers on a hierarchical combination of HBM and SRAM.
Etched’s key bet therefore lies in its datapath, rather than its choice of memory. Instead of supporting every kind of computation as a GPU does, it concentrates circuitry and power on the matrix, attention, and vector operations that recur in large language models.
Etched initially described its product as a chip with “transformers etched into silicon.” It has since broadened its scope to support a range of frontier models and sparse MoE architectures, rather than being fixed to a single model. Even so, its defining characteristic remains: it offers far less general-purpose flexibility than a GPU.
The company claims it can sustain utilization above 80% of peak compute performance on large sparse models. Its earlier Sohu announcement, which cited “500,000 tokens/s on Llama 70B with an eight-chip server,” also referred to aggregate server throughput across many requests, rather than the generation speed experienced by one user.
These figures have not yet been sufficiently validated through standardized, independent benchmarks. But Etched’s target is clear: enabling an entire data center to produce more tokens with the same power budget, rather than delivering tokens to a single user at the fastest possible rate.
Frozen v2: Specializing Down to the Model Architecture
According to reports, Google’s Frozen v2 sits at the most extreme end of this dimension. The project goes beyond supporting a model family such as transformers and seeks to reflect Gemini’s architectural characteristics in hardware.
This does not mean fixing the weights permanently. Newly trained weights can be swapped in, but much of the sequence and structure of the model’s operations is built into the circuitry. Decisions that software makes on a GPU are moved forward to the chip-design stage.
According to reporting cited by Reuters, Google aims to improve tokens per watt by 6–10 times over its latest in-house chip, with deployment discussed for as early as 2028. However, Google has not announced official product specifications, and neither the memory architecture nor actual performance has been disclosed.
Frozen v2’s biggest risk is time, rather than performance. Designing and manufacturing a chip takes years, while model architectures can change within months. If the next generation of Gemini adopts dynamic MoE or a new attention architecture that breaks the assumptions built into the circuitry, the flexibility sacrificed for efficiency becomes a liability.
One Underlying Question: Where Should Limited Resources Go?
These four efforts are easier to understand as different positions along two dimensions. One concerns how much high-speed memory, such as SRAM, is placed close to compute. The other concerns how much general-purpose flexibility is removed to specialize in particular operations.
Cerebras makes the strongest bet on the first dimension. By expanding the silicon footprint itself, it secures SRAM capacity and bandwidth at wafer scale, physically reducing the structural problem of weights being far from the compute units. Groq, meanwhile, expands SRAM across an entire system while fixing execution into a deterministic schedule, substantially reducing the flexibility the hardware must support. It pushes both SRAM expansion and specialization at once.
Etched takes a different approach. It retains HBM as its high-capacity memory foundation while removing much of the general-purpose control structure and unnecessary flexibility of GPUs. It redirects those resources toward the computational paths that recur in LLM inference. Its focus is on reducing waste in the datapath more than changing the memory architecture.
Frozen v2 takes this trend furthest. It is an attempt to embed the operation patterns of a particular model architecture—especially the Gemini family—directly into hardware, rather than build a general-purpose inference chip. It deliberately narrows the range of models it can support in exchange for maximizing efficiency within that range.
Viewed together, these four approaches reveal an important point: they are different answers to the same question of how to allocate a limited amount of silicon.
This perspective also makes combinations such as Groq 3 and Rubin easier to understand. GPUs equipped with HBM handle stages where memory capacity and bandwidth matter, such as prefill and KV cache handling. SRAM-based LPUs handle predictable, fixed execution paths, such as repetitive decode computation. Systems are being reorganized so that each stage of inference is assigned to the most suitable memory and execution architecture, rather than expecting one chip to handle everything.
The Goal Is Fast, Inexpensive Tokens
The ultimate goal of the inference-chip race is to produce tokens that are both fast and inexpensive.
For services where users wait for results—such as real-time voice, coding agents, and extended reasoning—generation speed at small batch sizes matters. Cerebras and Groq target this segment by keeping weights close to compute. They use more SRAM and silicon to reduce the latency users experience. As batch sizes grow, however, GPUs can also spread the cost of reading weights across multiple requests, while the KV cache that grows with each request becomes more significant. The strengths of these architectures are therefore most pronounced in real-time markets where low latency commands a premium.
Conversely, in environments where requests can be grouped into sufficiently large batches—such as search indexing, offline inference, and large-scale data processing—tokens per watt and total cost of ownership matter more than single-user speed. Once batching spreads weight traffic across requests, more specialized architectures such as Etched and Frozen v2 gain an advantage. By removing unnecessary control circuitry and execution overhead, they can improve compute utilization.
Still, fully replacing GPUs is difficult in large-scale serving environments that inherently require dynamic adaptation. The winner will be the system that assigns each stage of a workload to the most suitable memory and compute resources, producing the greatest number of useful tokens for the total cost. In that process, specialized chips will gradually take over portions of the workload that GPUs have handled inefficiently.