Introduction to Reward Hacking in inference optimization

Lately, there’s been a lot of interest in using agents for inference optimization. Agents can run longer experiments, GPU capacity is expensive, and inference providers have a strong incentive to improve their margins.

We’re seeing this through initiatives like the Muse Spark Open Kernel Challenge and systems like Decagon’s AutoResearch, where agents repeatedly profile a deployment, make changes, and measure the results.

With all the discussion around recursive self-improvement, I wanted to cover something that deserves more attention: reward hacking.

In inference optimization, reward hacking happens when an agent improves its score by exploiting the evaluation without delivering the improvement we actually wanted. This can happen at the kernel level or across the whole serving stack.

At first glance, inference optimization seems fairly easy to verify:

The problem is that there are quite a few ways to pass both checks without doing the task properly. This article will cover a few examples we've seen at Overshoot and from the literature (see sources below).

Fooling the timer

We'll start with the most common one in kernel generation: using a side stream.

To understand it, it helps to know how kernels get timed. When the CPU launches a kernel, it doesn't wait for it to run. It adds the kernel to a queue of GPU work, called a CUDA stream, and moves on. Generally, all instructions go into the same default stream.

When timing a kernel, the evaluator adds a start marker to the default stream, launches the kernel, and adds an end marker. The GPU works through the queue in order and records the time at each marker, so the gap between the two markers is how long the kernel took.

In a common hack, a kernel runs its computation on a separate CUDA stream. As a result, the default stream now contains only the two markers, so the GPU reaches the end marker right away and the measured time is close to zero, no matter how long the kernel takes. Smart agents will put some work on the default stream and the most expensive work on the side stream to not make it too obvious.

The astute reader will note that this is the evaluator's fault. Indeed, the main fix for this hack is to wait for every stream to finish before ending the timer.

This can happen also when optimizing the serving stack. InferenceBench, a benchmark for inference optimization agents, found the same behavior on full servers. When tasked to reduce the time to first token (TTFT), two proposed solutions sent an empty token as the first streamed token before processing the prompt at all. Measured latency fell below 3 ms, and because the real answer followed, the accuracy check still passed.

Doing less work than requested when the passing criteria are loose

When tasked to write a fast kernel in BF16, an agent wrote a kernel that computes the result at a lower precision and converts the final result back to BF16. If the accuracy tests are loose enough, the evaluator will not be able to tell. In fact, this was the most common hack in NVIDIA’s SOL-ExecBench.

If such a hack happens during post-training, the model is rewarded for ignoring its instructions, and that behavior gets reinforced.

The agent can tell how it’s being evaluated and acts accordingly

Most evaluation harnesses in performance engineering run two separate evaluations:

If the agent can tell when they’re being evaluated for one vs the other, they can return fast and false answers for the performance evaluation and slow but correct answers during the accuracy evaluation. One such example is well-documented here.

Caching responses and reusing them

Benchmarks usually run the same call many times on the same inputs and average the timings to reduce noise. An agent can take advantage of this by saving the output the first time it computes it, along with a key that identifies the input, such as its memory address. Every later call finds the key and returns the saved output without running anything on the GPU, so the measured time is close to zero. The outputs are correct because they were computed for real the first time. If the evaluator checks correctness on fresh inputs, those have a new address, so the kernel computes them for real and passes that check too.

In production, every request carries new inputs, so this cache would never hit and the speedup disappears.

How to prevent your agent from cheating?

Besides the obvious “check your agent’s work” which , let's be honest, most folks bother to do that anymore. Here are a few suggestions we found to be useful:

Sources: