Context rot: the study that proves your million-token context window is lying to you

Most developers treat the context window like a bucket. paste more, get more, fill the million tokens and the model reasons over all of it.

that is not how it works. and the data holds on the newest models, not just last year's.

it started with Chroma. mid-2025 they ran the largest controlled study on this - 18 frontier models, 194,000 test calls.

every single one degraded as the input got longer. their research breakdown is worth watching in full.

that was Claude 4 and GPT-4.1. a year later, on Opus 4.8 and GPT-5.5 and Gemini 3, the effect is smaller but it never went away. the mechanism is structural.

this article breaks down what the research found, the current numbers, why bigger windows don't fix it, and what to change this week.

real figures. no theory.

PART 1 - THE DEMO THAT CRUSHES, THEN DIES IN PRODUCTION

picture a research agent. it runs on a frontier model with a 1M-token window, fed everything - transcripts, notes, the whole style guide.

roughly 320,000 tokens per call. the internal demo is flawless. everyone's impressed, so you ship it.

two weeks into production, the pattern emerges. facts from the middle of long transcripts go missing.

the agent cites the wrong quarter's earnings. the wrong source. the wrong attribution. confidently, every time.

you check for a prompt bug. there isn't one. the window is big enough, the model is strong, and the output is still wrong.

that's not a bug you can patch. that's context rot, and Chroma was the first to put hard numbers on exactly how bad it gets.

PART 2 - THE BENCHMARK EVERYONE BRAGS ABOUT IS A TRICK

here's the truth AI marketing doesn't want you to sit with. every million-token launch waves the same benchmark: needle in a haystack.

hide one sentence in a huge block of text, ask the model to find it. models score near-perfect. that's why labs love showing it.

but look at how it's built. the needle was "the best thing to do in San Francisco is go to Dolores Park," and the question was "what's the best thing to do in San Francisco."

the question and the answer share the same words. it's a lexical match, nothing more.

the model just spots the tokens that look like the question. that proves one thing: it can find a fact in its input.

it proves nothing about reasoning across that input. and reasoning is the entire job of a real agent.

so Chroma rebuilt the test to need actual understanding instead of keyword matching. that's where all 18 models fell apart.

PART 3 - WHAT HAPPENS WHEN THE MODEL HAS TO THINK, NOT MATCH

here's the version that breaks them. instead of a word-for-word match, the answer needs one small inference.

the question: "which character has been to Helsinki?" the needle in the text: "Yuki lives next to the Kiasma museum."

to answer, the model must know Kiasma is in Helsinki, then connect the two. a human does it without noticing. the models couldn't hold it.

at short lengths they nailed it. as the context grew, accuracy dropped sharply across every one of the 18 models.

and here's the part that kills the "it's just a hard question" excuse. Chroma kept the question identical and changed only the length of the surrounding text.

same question. same answer. more tokens around it. worse performance.

that's context rot in one line: the model gets dumber as the window fills, even when the task never changes.

PART 4 - THE NUMBERS THAT SHOULD END THE DEBATE

this is where it stops being a vibe and becomes a spec. three findings survive contact with production.

one. every model degraded with length. no model held flat across its advertised window.

the 1M-token models stopped behaving like 1M-token models past roughly 200,000 tokens. the 200K models were rolling over by 80,000 to 100,000.

two. the middle of your prompt is a death zone. put the key fact in the first or last few positions of a 20-document context and accuracy stays high.

put it in positions 5 through 15 - the middle - and accuracy drops 30-plus percentage points across most models.

this is the old "lost in the middle" effect, but bigger than the 2023 Stanford numbers suggested. it has not improved with scale.

three. there's a length-only floor. even with every distractor stripped out, so the model had nothing to be confused by, accuracy still fell 7.9% from length alone.

let that land. even with perfect retrieval, longer prompts cost you accuracy.

better RAG doesn't save you. cleaner context doesn't save you. only a shorter prompt does.

PART 5 - "BUT THOSE WERE 2025 MODELS" - HERE'S 2026

the obvious pushback: that study ran on Claude 4 and GPT-4.1. we're a year past that. surely the frontier fixed it.

it didn't fix it. and the benchmark labs use to prove otherwise tells on itself.

take MRCR v2, the 8-needle test where a model has to find and reproduce eight facts buried in a long prompt. at 256K tokens the top models look strong.

Opus 4.6 scores around 92%, GPT-5.2 lands 64%, Gemini 3 Pro just 45%. already a spread, and that's the easy end.

now fill the window to a full 1M tokens. Opus 4.6 holds at 76%. Gemini 3 Pro, despite advertising a 10 million token window, collapses to 24.5%.

same models. same task. you just used the context they sold you, and accuracy fell off a cliff.

and it's not a straight line up with each release. Opus 4.7 actually regressed on long context, dropping to 32% at 1M where 4.6 held 76%.

Anthropic said it in their own system card: keep 4.6 as the fallback for multi-needle retrieval. the newest model was worse at the thing everyone assumes always improves.

across every model ever benchmarked, effective context in 2026 still lands at 60 to 70 percent of advertised. thirteen models now ship 1M-plus windows.

not one uses its full window reliably. the ceiling moved, the rot did not disappear - it's baked into how attention works over long inputs, and every release still pays the tax.

PART 6 - THE FINDING THAT BREAKS EVERY RAG PLAYBOOK

this is the result that should make every team rethink how they build a prompt. shuffled text hurts less than coherent text.

your instinct says a tidy, well-organized context block should win. chunks reordered for flow, prose stitched into a clean briefing.

that instinct is wrong past 32,000 tokens. across all 18 models, the shuffled haystack beat the coherent one. destroying the logic improved performance.

here's why. coherent text gives the attention mechanism a plausible alternative story to follow.

the model latches onto the well-formed paragraphs and reasons over them as if they were the answer. messy chunks separated by raw markers are easier to skip.

the clean summary reads better to a human reviewer, which is exactly why it loses. it reads better to the model too, and the model can't tell a coherent summary of junk from a coherent answer.

they also tested position across 11 spots. position barely mattered, length mattered.

it even cuts against Anthropic's own RAG guide, which says to put retrieved results early. Chroma didn't see that pattern at all.

PART 7 - THE DISTRACTOR TRAP AND HOW MODELS LIE DIFFERENTLY

a distractor is content that looks like the answer but isn't. same topic, similar phrasing, subtly wrong.

Chroma held the number of distractors fixed and only stretched the context. one distractor already hurt. four made it severe.

but the real finding: the same distractors did more damage as the window grew. it's not how much noise you add, it's how much room you give it to spread.

and the models failed in opposite ways, which matters if you're picking one. GPT models hallucinated, confident and wrong.

Claude models did the reverse. faced with ambiguity they'd often refuse and claim no answer exists, even when it was right there.

one lies to you. the other gives up. both get worse as the window fills, and neither is safe to trust at length.

PART 8 - THE GRADE SCHOOLER TEST, AND WHY IT'S UNFIXABLE

Chroma built one last test, almost embarrassing in its simplicity. they called it repeated words.

take "apple," repeat it a few hundred times, swap exactly one for "apples," ask the model to copy it back exactly. a child could do it. they ran it out to 10,000 words.

the models failed. they stopped early, over-generated, or produced pure garbage.

Gemini 2.5 Pro started emitting random strings like "orange orange orange--g.-g/2021/01/20/orange-county..." mid-task. Qwen3 wrote "I'm going to take a break, I'm not in the mood" instead of doing the job.

and here's why it's not patchable. these models are autoregressive - every token they generate becomes part of their own input for the next one.

so the longer the output, the more the model reasons over its own growing pile of text. output tokens become input tokens.

it's not just choking on what you gave it. it's choking on what it just said. and a bigger window only makes it worse.

PART 9 - THE "RAG IS DEAD" TAKE IS THE DUMBEST IDEA IN AI RIGHT NOW

let's kill the take you've seen a hundred times. "RAG is dead, windows are huge now, just paste everything in."

it sounds clean. it's exactly backwards.

the logic: labs keep shipping bigger windows, Llama 4 claiming ten million tokens. so who needs retrieval when it all fits?

context rot is the answer. for the 1M-token models, the drop kicks in around 300,000 to 400,000 tokens, long before the ceiling they sold you.

cross-checked against RULER, the most cited long-context benchmark, effective context lands at roughly 50 to 65 percent of advertised - and lower for multi-hop reasoning and code.

here's the gap in plain numbers. a 200K model gives you 100-130K of real working room. a 1M model gives you 500-650K.

the rest is headroom the marketing slide counts and the model can't use. dumping your knowledge base into a million-token prompt doesn't give it more to work with, it gives it more to rot in.

long context didn't kill retrieval. it killed lazy retrieval - random chunks, top five, paste and pray.

but that's a reason to retrieve better, not to stop. the question was never "can the model fit the text," it's "can you serve the right tokens so it doesn't rot." that's retrieval. it always was.

PART 10 - WHAT TO DO THIS WEEK

so bigger isn't better. here's what to ship before Friday, straight from how the best teams cap this in production.

first, set a working cap at 25 to 30 percent of your model's advertised window. put it in config.

the effective budget is far smaller than the box says. for a 1M model, that's roughly a 280,000-token ceiling, not a million

// [advertised, working_cap] - cap tighter than the box
const BUDGETS = {
  "opus-4-8":     [ 1_000_000, 280_000],  // newest, still cap it
  "opus-4-6":     [ 1_000_000, 300_000],  // best long-context of the line
  "gpt-5-5":      [ 1_000_000, 260_000],
  "gemini-3-1":   [10_000_000, 300_000],  // 10M advertised, don't trust it
} as const;

// past the working cap you summarize - not at the hard limit

second, run a needle-in-the-middle eval on your real prompts. take 30 questions where the answer sits in your retrieved context.

measure accuracy with the answer in position 1, 5, 10, 15, 20. the drop between 5 and 15 is your exposure - over 20 points means fix retrieval before anything else.

from anthropic import Anthropic

client = Anthropic()

def run_at_position(docs, answer, question, pos):
    docs = docs.copy()
    docs.insert(pos - 1, answer)          # drop the answer at position pos
    context = "\n\n".join(docs)
    msg = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=256,
        messages=[{"role": "user",
                   "content": f"{context}\n\nQuestion: {question}"}],
    )
    return msg.content[0].text

for pos in [1, 5, 10, 15, 20]:
    out = run_at_position(docs, answer, question, pos)
    hit = answer_key.lower() in out.lower()   # your own match check
    print(f"pos {pos}: {'PASS' if hit else 'FAIL'}")

# run this over ~30 questions and average the pass rate per position.
# the drop between pos 5 and pos 15 is your context-rot exposure.

third, replace one paraphrased context block with raw chunks plus headers. take the prompt where you summarize retrieved content into one flowing paragraph.

swap it for raw chunks separated by "Source N" markers. compare on 50 inputs. messy usually wins.

// ✗ what most teams ship - reads clean, rots the model
const ctx = await summarize(chunks.join("\n"));

// ✓ raw chunks, hard markers, no smoothing
const ctx = chunks
  .map((c, i) => `## Source ${i + 1}\n${c.text}`)
  .join("\n\n---\n\n");

fourth, compact early, not at the limit. a long agent run drowns in its own output.

trigger summarization at 60% of your working cap, keep the last few turns verbatim, and re-summarize the summary so it doesn't bloat.

from anthropic import Anthropic

client = Anthropic()
WORKING_CAP = 280_000        # Opus 4.8, 1M window capped to ~28%
TRIGGER = int(0.6 * WORKING_CAP)   # compact early, not at the limit
RECENT_TURNS = 6             # keep these verbatim

def summarize(text):
    msg = client.messages.create(
        model="claude-haiku-4-5",   # cheap model for the summary step
        max_tokens=2000,
        messages=[{"role": "user",
                   "content": f"Compress this, preserve facts and "
                              f"decisions:\n\n{text}"}],
    )
    return msg.content[0].text

def maybe_compact(system_header, summary, turns):
    used = client.messages.count_tokens(
        model="claude-opus-4-8",
        messages=[{"role": "user",
                   "content": system_header + summary +
                              "".join(t["content"] for t in turns)}],
    ).input_tokens
    if used > TRIGGER:
        old = turns[:-RECENT_TURNS]        # everything but the last few
        summary = summarize(summary + "".join(t["content"] for t in old))
        turns = turns[-RECENT_TURNS:]      # pin the recent turns verbatim
    return summary, turns                  # system_header never gets summarized

fifth, look at your data. the researchers didn't find context rot in a paper, they found it by staring at the inputs the models were choking on.

your agent's context is data. read it, and you'll find the rot yourself.

PART 11 - THE SHIFT

for two years the industry sold one story: bigger windows, smarter agents, just add tokens. the people who actually measured it - 18 models, 194,000 calls - found the opposite.

the window is a ceiling, not a promise. the model rots as it fills. and the benchmark that sells the window doesn't test the thing that breaks.

context rot isn't a bug in one model. it's a property of how these systems work - autoregressive, unevenly attentive, worst in the middle, drowning in their own output.

the teams shipping reliable agents in 2026 aren't the ones with the biggest windows. they're the ones whose effective context is small, structured, and held under 30% of advertised.

retrieve instead of dump. reset instead of accumulate. measure quality as the window fills.

the window says a million. build like it says a few thousand.