Apparently, one way to make LLM inference almost 2x faster is to...
Navaneeth Krishnan@navaneethvb
11 views
Aug 23, 2026
~2 min read
Advertisement
1
Apparently, one way to make LLM inference almost 2x faster is to make the CPU stop talking to the GPU so much.
And this is where CUDA graphs come in, but before getting into the details let us actually understand the problem first.
An LLM forward pass is not one giant GPU operation. Underneath it, you have a bunch of smaller programs called kernels doing matmuls, attention, normalization, routing, sampling etc, and normally the cpu keeps launching them one after another.
think of it like having an insanely fast chef, but after every single step he has to stop and wait for you to tell him the next instruction.
at some point the chef is not the bottleneck anymore. you are and this is basically where CUDA Graphs come in, so instead of the cpu repeatedly going:
run A → now B → now C → now D
you record that sequence once and replay it, so you cut a lot of the kernel launch overhead.
It may sounds perfect for llm inference, except llm inference has one big annoying problem.
The workload keeps changing everytime. For decode it is relatively nice here. if you have 32 active requests, each one usually contributes one new token to the next decoding step, and the next step looks pretty similar again.
That repetition is exactly what CUDA Graphs like.
But prefill is way messier because one batch could look like:
request A → 200 prompt tokens
request B → 4,000
request C → 15,000
And then the next batch can look completely different.
So CUDA Graphs are basically asking:
“can you please give me the same predictable execution again?”
and llm serving is like:
“best i can do is a completely different batch every few milliseconds.”
This is why I found SGLang's Breakable CUDA Graph idea pretty interesting. Instead of forcing the entire forward pass into one static graph, you let the weird dynamic parts run normally and keep the predictable parts inside captured graphs.
So it becomes something like:
recorded gpu work → dynamic operation → recorded gpu work → dynamic operation → recorded gpu work
for prefill they can also bucket the token count. if the live workload has 3,700 tokens and you already captured a graph for 4,096, you can pad it and reuse that graph.
so you are literally doing some extra gpu work just to avoid making the cpu launch everything again and the numbers are pretty nice too.
On SGLang's gpt-oss-120b prefill benchmark:
- eager execution → 1x
- Breakable CUDA Graph → ~1.70x
- full CUDA Graph → ~1.93x
we always talk about more flops, more bandwidth, faster kernels etc, but sometimes there just tiny things like this that, if optimized, can give the best inference gains.
Image Source: NVIDIA blog, 'constructing-cuda-graphs-with-dynamic-parameters' (speed comparison without and with graphs)
And this is where CUDA graphs come in, but before getting into the details let us actually understand the problem first.
An LLM forward pass is not one giant GPU operation. Underneath it, you have a bunch of smaller programs called kernels doing matmuls, attention, normalization, routing, sampling etc, and normally the cpu keeps launching them one after another.
think of it like having an insanely fast chef, but after every single step he has to stop and wait for you to tell him the next instruction.
at some point the chef is not the bottleneck anymore. you are and this is basically where CUDA Graphs come in, so instead of the cpu repeatedly going:
run A → now B → now C → now D
you record that sequence once and replay it, so you cut a lot of the kernel launch overhead.
It may sounds perfect for llm inference, except llm inference has one big annoying problem.
The workload keeps changing everytime. For decode it is relatively nice here. if you have 32 active requests, each one usually contributes one new token to the next decoding step, and the next step looks pretty similar again.
That repetition is exactly what CUDA Graphs like.
But prefill is way messier because one batch could look like:
request A → 200 prompt tokens
request B → 4,000
request C → 15,000
And then the next batch can look completely different.
So CUDA Graphs are basically asking:
“can you please give me the same predictable execution again?”
and llm serving is like:
“best i can do is a completely different batch every few milliseconds.”
This is why I found SGLang's Breakable CUDA Graph idea pretty interesting. Instead of forcing the entire forward pass into one static graph, you let the weird dynamic parts run normally and keep the predictable parts inside captured graphs.
So it becomes something like:
recorded gpu work → dynamic operation → recorded gpu work → dynamic operation → recorded gpu work
for prefill they can also bucket the token count. if the live workload has 3,700 tokens and you already captured a graph for 4,096, you can pad it and reuse that graph.
so you are literally doing some extra gpu work just to avoid making the cpu launch everything again and the numbers are pretty nice too.
On SGLang's gpt-oss-120b prefill benchmark:
- eager execution → 1x
- Breakable CUDA Graph → ~1.70x
- full CUDA Graph → ~1.93x
we always talk about more flops, more bandwidth, faster kernels etc, but sometimes there just tiny things like this that, if optimized, can give the best inference gains.
Image Source: NVIDIA blog, 'constructing-cuda-graphs-with-dynamic-parameters' (speed comparison without and with graphs)
