Searching a billion tokens of agent data

Authors: @andrew_li03 @rao_abinav

How our data agent uses evals-as-indexes for better, faster, cheaper search.

Searching through agent traces is a new and hard problem

Most agent failures don’t come with a big flashing error. You usually find them by accident, while digging into how users interact with your agent, when you notice it did something unexpected. Then comes the harder question: how often has this happened? Answering it means searching through billions of tokens of production data.

Large-scale data systems have long turned vast amounts of raw data into useful signals. Meta’s ads platform, for example, used tens of thousands of engineered features (e.g. recently clicked ads and page-visit counts) to personalize recommendations. At its core, the recommendation system turns vast amounts of user activity into better insights about people's preferences.

Long-horizon agents now produce data at a much larger scale. Their traces do contain structured events, such as searches, page visits and tool calls, but much of what we care about lives in free-form text across those events. A single run can produce thousands of model calls, tool invocations and intermediate decisions, most of them unstructured text. Whether a run actually succeeded often depends on context spread across many spans: a request the agent quietly dropped, a correction from the user, or a tool call that “succeeded” but did the wrong thing. However, the infrastructure and best practices for making sense of this data are still developing. Reading traces by hand helps you understand your agent’s behavior and discover what to look for. But finding every instance of that behavior across billions of tokens requires a more scalable approach.

The natural next step is a data agent: an agent that answers questions about your production traffic by writing SQL over your traces, reading the relevant spans, and citing the evidence it finds. But pointed at raw traces alone, it hits the same wall we do. The behaviors we care about are often latent, so keyword search can’t reliably find them, and the data agent has to find and evaluate them from scratch on every question.

Luckily, LLMs have made it much easier to semantically parse the traces we’re storing. Compared to earlier BERT-style models, LLMs are more capable of handling arbitrary, complex rubrics without fine-tuning. That lets us break each run down along multiple semantic dimensions:

Was the user frustrated?

Did the agent follow the specified instructions?

What was the action’s outcome?

Each of these is an eval, and many teams already run evals like these as online monitoring, to track quality and catch regressions in production. But the same results can also serve as an index, a column you can filter and search on. That makes your data much easier to understand, and makes it far more likely you’ll catch what’s going wrong.

Why search needs these evals

Without evals, a data agent typically falls back on one of two approaches:

Read traces one by one. With no index to narrow the search, it has to inspect trace after trace. It’s slow, and for a data agent it fills the context window with irrelevant results. As the context grows, the agent suffers from context rot, and the evidence it found earlier gets truncated or compacted away, so the search gets worse the longer it runs.

Search by keyword. SQL LIKE or grep only finds the words it thinks to try, and some questions have no words to search for. Take “did the agent follow the policy it was given?” It may not know what each agent’s policy says, and breaking it doesn’t leave any particular word behind. To answer, it have to read the policy, read what the agent did, and compare the two.

What about vector search? Vector search can find relevant passages even when their wording differs from the query, and a reranker can help prioritize the results. But answering questions like “did the agent repeat a mistake after being corrected?” still requires evaluating the evidence, sometimes across multiple spans. Evals-as-indexes do that work ahead of time for behaviors you already know you want to track. Subsequent searches can filter the stored results and inspect the supporting evidence, rather than evaluate those behaviors from scratch.

Say you run a sales agent and want to find requests it did not complete. Consider the query: “Find email requests that were not completed.”

For this example, we use three evals, some run as standard LLM-as-a-judge and some with Jev:

requested_action: what did the user ask the agent to do? For a sales agent: send_email, update_pipeline, research_account, schedule_meeting.

action_outcome: what actually happened to it? completed, partially_completed, pending_approval, not_done or failed.

failure_recovery: if something failed, did a later attempt fix it? recovered, partially_recovered or unresolved.

Evals can see an entire trace or an individual span. Their results are stored in telemetry.scores, alongside the traces and spans they describe, with the eval name (judge), its result (label) and an explanation (reason). Each result links to the relevant trace and supporting span. In this example, the evals assess whether the email request was completed at the trace level and cite supporting spans within that trace. We assume at most one result per eval per trace.

The raw query can find the right trace even when the sales agent tells the user “Done, the follow-up is ready.” A save_email_draft receipt with sent_at: null tells us the message was still a draft at that point. The search can miss the evidence that tells us whether the email was ever sent. A later delivery confirmation, for example, might appear in a span that matches neither the tool names nor the phrases in the query. With evals-as-indexes, it can filter for requests labeled not_done, exclude those labeled recovered, and inspect the supporting spans.

Using evals-as-indexes helps the data agent find relevant evidence more accurately, finish faster, and consume fewer input tokens. The search costs below exclude the upfront cost of generating the evals, which can be reused across subsequent searches.

The walkthrough below uses a simplified SQL schema to make the Sonnet 5.5 run easier to follow.

Benchmarking

To measure whether evals-as-indexes actually help, we built a search benchmark from perturbed internal traffic: 4K+ long-horizon agent traces (1B+ tokens), with 50+ search questions. Each question includes a reference answer, human-reviewed evidence trace IDs, and a rubric that checks whether the agent correctly analyzes the evidence.

We built it with an AutoBenchmark-style loop:

Generate questions. LLMs role-played people who use our platform (PMs, agent engineers, CTOs) to write realistic sets of questions grounded in the agent trace structure.

Collect evidence. A research agent searched the traces to assemble each question’s answer, evidence IDs and rubric, which humans then reviewed.

Test with solver agents. We ran agents of varying strength on every question and rewrote or dropped any that were too vague or had valid alternate answers.

Add adversarial distractors. We inserted near-miss traces (for example, a failure the agent later recovered from) and follow-up turns designed to mislead a search agent that doesn’t verify its evidence.

We measure evidence precision and recall against the reference IDs and grade answer quality with the rubric. We ran every model with and without evals-as-indexes.

Results

Our index reuses 18 online monitoring evals, including requested_action, action_outcome and failure_recovery, for a total of 40,482 evals across the dataset. Each trace is first parsed programmatically to extract the evidence each eval needs, then scored with MiniMax M3.1 Flash and Jev, for a total cost of about $60.

Averaging each of the seven models’ relative changes, indexes reduced input tokens by 23.9%, search time by 21.4%, and estimated search costs by 20.8%. Answer quality improved by 22.7%.

In addition to these results, several open-source models saw substantial gains in recall and precision, most of the time because the evals had already surfaced cases the models were less likely to find by searching the raw text. However, stronger retrieval did not always translate into stronger analysis. Some open-source models appeared less likely to challenge the eval results, potentially making them more susceptible to incorrect or incomplete indexes. Across all seven models, recall increased by 54.8%, precision by 43.0%, and SQL calls decreased by 11.8%, averaging each model’s relative change.

When the index isn’t worth it

A data agent can always query raw traces with SQL, and evals-as-indexes are an addition on top of that. In two cases, plain SQL are probably all it needs:

Your data is small. At low scale, the data agent finds shortcuts. If a preview of the whole trace corpus fits in a single model call, it can skim everything, spot the right keywords and search for them directly. For a small dataset, it can start by reading the 100 most recent spans and follow the relevant trace IDs for more context.

Your evals aren’t reliable. Eval results only point the search somewhere, so their quality matters: an eval that over-fires buries real cases in noise, and one that under-fires hides them. Some open-source models take eval results at face value, which likely explains why their recall rose more than their answer quality. Check your evals against hand-labeled examples before relying on them for search.

Conclusion

As agents take on longer and more consequential work, they produce more data than anyone can read manually. Failures like a missed instruction, or a mistake repeated after the user corrected it, span many steps and rarely leave a keyword behind, which makes them hard to find reliably with keyword search at scale.

But most teams already run online monitoring evals to track agent quality, and those results can be reused as indexes. Every search then starts from behavior that’s already been evaluated, and the data agent spends its effort verifying evidence instead of hunting for it.

Our goal is to make agent data as easy to search as any other data: ask anything about what your agents did in production, and get an answer backed by evidence you can check.

Thanks to @22uenos and @PhilipMovery for designing the figures and interactive walkthrough. Follow @JudgmentLabs on X and check out more here.

Searching a billion tokens of agent data