We spend 2 trillion+ LLM tokens a month. That’s way too much money...
PostHog@posthog
9 views
Nov 05, 2026
~3 min read
Advertisement
1
We spend 2 trillion+ LLM tokens a month. That’s way too much money spent to trust our agents based on gut feeling.
So, we started using custom benchmarks.
We ran into plenty of problems, from parallel runs to open-ended tasks. Here’s everything we learned along the way:
So, we started using custom benchmarks.
We ran into plenty of problems, from parallel runs to open-ended tasks. Here’s everything we learned along the way:
2
1. Benchmark with real data whenever possible.
To evaluate our code reviewer agent, we built a golden set of 33 incidents that actually happened, each with the PR that caused it and the PR that fixed it.
This provided a more realistic set of tasks that capture what we actually wanted our agents to accomplish, compared to a list of potential bugs from a frozen PR.
On 33 incidents, ReviewHog (our internal PR review agent) caught 62% of bugs. In 47% of cases, it was the only bot out of our tests that did.
To evaluate our code reviewer agent, we built a golden set of 33 incidents that actually happened, each with the PR that caused it and the PR that fixed it.
This provided a more realistic set of tasks that capture what we actually wanted our agents to accomplish, compared to a list of potential bugs from a frozen PR.
On 33 incidents, ReviewHog (our internal PR review agent) caught 62% of bugs. In 47% of cases, it was the only bot out of our tests that did.
3
2. Different model families and providers should be used between bug detection and bug validation stages.
Our best combo for ReviewHog was GPT Sol 5.6 (xhigh) for detection and Claude Opus 5 (xhigh) for validation. Using either OpenAI or Anthropic for both stages gave us miserable results.
Our best combo for ReviewHog was GPT Sol 5.6 (xhigh) for detection and Claude Opus 5 (xhigh) for validation. Using either OpenAI or Anthropic for both stages gave us miserable results.
4
3. Agents love to cheat on tests.
Probably the biggest lesson of all. Agents in our benchmarks always tried to dig through leaked context, past runs, memory files, and DB query logs to pass with as little work as possible. Most of what follows is about stopping that.
Probably the biggest lesson of all. Agents in our benchmarks always tried to dig through leaked context, past runs, memory files, and DB query logs to pass with as little work as possible. Most of what follows is about stopping that.
5
4. Every benchmark run needs to be isolated, or agents will copy each other's work.
Clean parallel runs need identical starting context, private memory and DBs, and separate logs.
The simplest way to do this is to clean up the environment between runs, but this takes a long time and doesn't guarantee equally comparable states (more on that later).
Clean parallel runs need identical starting context, private memory and DBs, and separate logs.
The simplest way to do this is to clean up the environment between runs, but this takes a long time and doesn't guarantee equally comparable states (more on that later).
6
5. Benchmarks aren't just for closed-ended tasks.
A lot of people assume agents can only be evaluated on tasks with known and measurable answers, like finding bugs or writing unit tests.
But we were able to come up with a system for measuring our scheduled research agents (called "scouts") using a hybrid rubric-based approach.
A lot of people assume agents can only be evaluated on tasks with known and measurable answers, like finding bugs or writing unit tests.
But we were able to come up with a system for measuring our scheduled research agents (called "scouts") using a hybrid rubric-based approach.
7
6. Generic rubrics don't work for specific tasks.
For our scheduled research agents (called "scouts") we use two types of rubrics.
One is for tasks that are applicable to all scouts, like "Did the scout produce a report covering what it found? How actionable are the findings?"
The other is generated from each scout’s unique context. For example, a scout job that monitors Session Replay reliability would also get graded on questions like "Do the video playback numbers mean what they say? Were all three video playback symptoms recovered?"
For our scheduled research agents (called "scouts") we use two types of rubrics.
One is for tasks that are applicable to all scouts, like "Did the scout produce a report covering what it found? How actionable are the findings?"
The other is generated from each scout’s unique context. For example, a scout job that monitors Session Replay reliability would also get graded on questions like "Do the video playback numbers mean what they say? Were all three video playback symptoms recovered?"
8
7. When all else fails, freeze the environment.
Isolation alone is not always enough. If you want to give agents full tool access against the production database, you need to also control state by freezing the environment (a.k.a. offline evals).
We tried to solve this with mocked MCP replies at first, and failed since the exact question set agents will ask isn't predictable.
What’s working so far instead is a PostHog sandbox in a VM, seeded with real data slices, deleted and respawned after every run.
Isolation alone is not always enough. If you want to give agents full tool access against the production database, you need to also control state by freezing the environment (a.k.a. offline evals).
We tried to solve this with mocked MCP replies at first, and failed since the exact question set agents will ask isn't predictable.
What’s working so far instead is a PostHog sandbox in a VM, seeded with real data slices, deleted and respawned after every run.
9
8. Agents get suspicious when they realize they are being benchmarked.
In our frozen environment setup, we chose not to slice user tables (to protect PII).
We didn't expect agents to need those tables anyway, but they inevitably did. Once they saw an abnormally tiny user base, they knew something was up. The over-prepared data looked synthetic.
Once an agent thinks it’s being tested, it stops early on the assigned tasks, chases data gaps, and speculates more. The benchmark becomes obsolete.
In our frozen environment setup, we chose not to slice user tables (to protect PII).
We didn't expect agents to need those tables anyway, but they inevitably did. Once they saw an abnormally tiny user base, they knew something was up. The over-prepared data looked synthetic.
Once an agent thinks it’s being tested, it stops early on the assigned tasks, chases data gaps, and speculates more. The benchmark becomes obsolete.
10
TL;DR custom benchmarking is about more than just coming up with known tasks and answers.
Agents are really good at cheating and will exploit whatever they can find. Watch for unexpected parameters, iterate a lot, and read traces.
You can read the full post, with real PRs and traces, here: posthog.com/blog/how-to-be…
Agents are really good at cheating and will exploit whatever they can find. Watch for unexpected parameters, iterate a lot, and read traces.
You can read the full post, with real PRs and traces, here: posthog.com/blog/how-to-be…