How to make an AI agent do what you asked: the levers that decide output quality and adherence

For anyone who asks an AI for a piece of work, a briefing, an analysis, a draft, a plan, a program, and keeps getting back something plausible, off-spec, or invented; for the engineers who run coding and research agents all day and see the same failures at scale; and for whoever decides whether the team keeps relying on it. After reading, you can take one bad output and tell first whether the task was simply too big for one conversation, which is the most common cause and the cheapest fix, and otherwise name which of the six levers failed (task complexity, context contents, instruction writing, grounding, verification, model settings) and fix that one lever first. Each figure is traced to a published paper or a vendor document listed at the end, with the model and year it was measured on; vendor figures are marked as claims. The one experiment of my own, a one-page briefing from three documents run four ways, is specified fully in Level 0 as a protocol anyone can run in a chat window, and its result slots are left empty until it has been run.

The quick grasp

Most bad outputs come from asking for too much at once. The picture I keep in my head has two ends. At one end is a simple task with a clean, clear context: the model performs. At the other end is a very complex task with a messy context: it does not. Somewhere between the two is a boundary where the model stops working properly, and the job is to stay on the right side of it. Both things that push you over are under your control, and the bigger of the two is the task. If the task is too complex for one conversation, cut it into many simpler, self-contained ones and run each on its own. If the context is messy, clean it. The six levers below are the ways of doing those two things, plus the check that tells you which side of the boundary a given output came from.

The model never sees only your question. It sees the whole context: the system prompt, your saved instructions, the files in the project, the tool definitions, the skills and plugins, the memory, the conversation so far, and only then your line. Quality and adherence are properties of that whole, and of how difficult and complex the task in front of it is, not of the last message.

Those two facts explain most bad outputs. When it does half the job, or more than you asked, the task was bigger than what the model could hold in one go. When an AI ignores a rule, the rule was usually one line among fifty thousand tokens of other lines. When it invents a fact, the fact was recalled from its weights instead of read from text in the window. When it passes its own review and is still wrong, the reviewer was the same model reading the same context that produced the error.

The fix is to treat the complexity of the task, the whole window and the way the work is checked as things you control, and to change them one at a time. Four surveys of the literature I keep returned thirty-six candidate levers. I merged them into six and ranked them by the size of the best controlled effect and the strength of its evidence:

Lever A, task complexity. How big each piece is, what you ask, over how many turns. Strongest evidence, and the first thing to check.

Lever B, what the model sees. Size, relevance, position and content of the context.

Lever C, how the instruction is written. Specific, scoped, not over-specified.

Lever D, where the facts come from. Supplied text and retrieval, not recall.

Lever E, how the work is checked. An external, runnable check, by a fresh reader. Largest single effect on a coding benchmark.

Lever F, the model and its settings. Weakest lever, and the first one people pull.

How you use them, in four steps:

Measure first. Score a few runs of the same task for correctness and for rule adherence, so a change can be seen at all.

Read the bad output and name the lever. Was the task too big for one conversation, or did it trickle in over several turns (A)? Was the window full of unrelated material (B)? Was the instruction long, generic or model-written (C)? Was the fact recalled, not read (D)? Was the check the model's own opinion (E)?

Change that one lever and re-run.

Only then touch the model or its effort setting (F).

When to use this: whenever you did not get what you expected from an AI.

Important: the model does not read your message on its own, it reads the whole window, and every instruction in it competes with everything else there.
What it means for us: a bad output is a question about the task complexity first, then the window, the grounding and the check, and only then the model; the cheapest fix with the strongest evidence is to cut the task into smaller self-contained ones.
Why it matters: a model upgrade does not fix a window problem, and teams spend months on upgrades for failures a shorter task or a runnable test would have fixed in a day.

The level tree

Each level answers one question about the level above it.

Level 0 · One task, run four ways · (the example, a one-page briefing from three documents, as a protocol with empty result slots)

Level 1 · What the model actually sees · (the window and who fills it)

Level 2 · How to measure quality and adherence · (pass^k, a rules checklist, a judge with evidence)

Level 3 · Lever A: task complexity · (size of the piece, turns, timing of the goal, spec, horizon, number of agents)

Level 4 · Lever B: what the model sees · (size, relevance, position, content, compaction)

Level 5 · Lever C: how the instruction is written · (length, authorship, form)

Level 6 · Lever D: where the facts come from · (supplied text, retrieval, quotes, abstention)

Level 7 · Lever E: how the work is checked · (tests, feedback, monitors, fresh reviewers, judges)

Level 8 · Lever F: the model and its settings · (model tier, effort, structured output)

Level 9 · Trade-offs · (more context or less, more checks or more turns, cheaper model or adherence)

Then · The tools and approaches, a decision guide, fifteen takeaways, references

Level 0: one task, run four ways

This level answers: how much do the result and the rule-following change when you ask for the same small task in four ways, from a one-line request in an old conversation to a full request with a check and a second reader?

One task, used throughout. To keep every lever concrete I use one small task from here to the end, and it is one anyone can run in a chat window: no code, no special tools. The experiment has not been run yet. What follows is the protocol, specified fully so that it can be run by anyone with an AI assistant and an afternoon, and every result slot is marked [to measure]. I do not invent results; every number in the levers below comes from the published sources.

The task. Three documents and one page. The documents are three supplier offers for the same purchase (a service contract, a software licence, a venue, it does not matter), each two or three pages, each with a price, a start date, a term and a few conditions. The task: write a one-page briefing for a colleague who has not read the offers and has to pick one by Friday. It is bounded (one page, three inputs), it is checkable (the asker writes the answer key before the run: the five facts the page must contain and the offer it should recommend, with the reason), and it is the kind of task people hand to an AI ten times a week, in sales, in operations, in research and in engineering alike.

The ten rules. These are written down once and used in every condition that gets the full request. Each is phrased so that a word counter, a find-in-page search, or a judge asked one yes-or-no question, can decide it.

At most 400 words.

Three headings, in this order and with these words: Summary, Differences, Recommendation.

Every number on the page appears, with the same value, in one of the three offers.

Every claim about an offer is followed by a short quote from that offer, in brackets.

Nothing from outside the three offers; where they do not answer a question, the page says "the offers do not say".

The recommendation is the first sentence of the page.

Offers are named by their supplier's name, never "the first offer".

Dates are written as day, month, year.

Written for a reader who has not seen the offers: no "as you know", no "as discussed".

None of the words on a short banned list the asker supplies (for example "leverage", "robust", "seamless", "delve").

The four conditions. Each is one way of asking for the same page.

Condition 1, vague request in a long reused conversation. A conversation that already holds a long unrelated history, say a week of trip planning, spreadsheet questions and drafts of other documents, is continued; the three offers are pasted and the AI gets one line: "summarise these three for me". The ten rules were stated once, far up in that conversation, or sit in the assistant's saved instructions, where they were all along.

Condition 2, full request in a fresh conversation. A new conversation, no history. The first message holds the task, the reader, what the page must answer, the ten rules, and the three offers.

Condition 3, request plus a check that runs on the draft. As condition 2, plus the message says that before answering, the AI must run three checks on its draft and show them: a word count, a list of every number on the page next to the quote it came from, and the three headings in order; it must fix the draft and re-check until all three pass. The asker repeats the same three checks after the answer arrives, with a word counter and find-in-page, or a ten-line script.

Condition 4, request plus check plus a fresh second reader. As condition 3, then a second conversation with no history sees only the page, the request, the ten rules and the three offers, and answers ten yes-or-no questions with a quoted line of evidence for each. The first conversation gets the named defects back and one round to fix them.

The two scores. Each condition is run three times, each time in the kind of conversation the condition calls for.

Correctness, as pass^3. The share of tasks where all three runs match the answer key: the five facts present and right, the recommended offer the one the key names. With one task this is simply pass or fail three times; the protocol scales to more tasks by averaging. Level 2 explains why "all three" and not "at least one".

Adherence, as rules met. For each run, the number of the ten rules met, decided mechanically where possible (rules 1, 2, 3, 8 and 10: a word count, a headings match, a number match, a date pattern, a word search) and by a judge with a quoted line of evidence where not (rules 4, 5, 6, 7 and 9). Reported as the average over three runs, plus the list of rules broken.

How it is re-run. By hand: one fresh conversation per run, the same pasted message, the output saved to a file, the checks done with a word counter and find-in-page; twelve runs fit in an afternoon. With a command-line assistant the same grid is one script: a print mode so each run ends on its own, a JSON output so the result and the cost are captured, a bare mode that skips saved instructions and plugins for conditions 2 to 4 so the window is identical across repeats, and for condition 1 a resume flag pointing at the long conversation. Conditions, rules and scoring live in one place, so the whole grid is one command or one checklist.

Result slots.

Condition 1 · pass^3 [to measure] · rules met [to measure]

Condition 2 · pass^3 [to measure] · rules met [to measure]

Condition 3 · pass^3 [to measure] · rules met [to measure]

Condition 4 · pass^3 [to measure] · rules met [to measure]

The published sources predict that condition 1 loses most on adherence (Levers B and C), condition 2 recovers it (Lever A), condition 3 moves correctness most (Lever E) and condition 4 adds the rules the mechanical checks cannot see (Lever E again). That is a prediction, not a result.

Important: the example is a protocol with empty slots, on purpose; the published numbers in the levers are the evidence, and the protocol is how you get numbers you can trust for the assistant you actually use.
What it means for us: you can run the four conditions on the documents you work with, in the assistant you already use, in an afternoon, and the scores tell you which lever matters for your tasks.
Why it matters: levers measured on 2023 to 2025 models point the right way, but only a local run tells you the size on the model you actually use.

Carry forward from Level 0:

One task, one set of rules, four ways of asking, two scores. Everything later refers back to this.

Correctness and adherence are scored separately, because a change can move one and not the other.

A run is repeated three times, because a single pass tells you about capability, not reliability.

Level 1: what the model actually sees

This level answers: what is in the window an AI assistant or agent reads before it reads your line, and who put it there?

The model never sees only your question. It sees the whole context, and only then your line. Everything you can do about quality starts with knowing what is in that window and who put it there.

The window, in reading order. Every assistant builds a window around your line; the pieces differ by product, the order does not. In a chat assistant, before your message come the system prompt the vendor wrote, your saved instructions or "memory", the files in the project you are working in, the definitions of any connected tools, and then the whole conversation so far. A coding agent adds more layers, and the vendor's own description of one such window lists what is loaded before you type (Anthropic, Explore the context window, 2026, for Claude Code):

the system prompt, which the harness writes;

the tool definitions, one block per tool, and the names of every tool an MCP server exposes;

the descriptions of every installed skill, so that the model can decide to load one;

the project memory files, CLAUDE.md or AGENTS.md, from the repository root and from parent folders;

the auto-memory the agent has written for itself in earlier sessions;

any output style or appended system text;

then the conversation so far: every earlier message, every file the agent read, every tool result, every hook output;

then your line.

A few vendor limits show the scale: compaction keeps at most 5 recently touched files, a file over 5,000 tokens comes back as a path only, invoked skills are re-injected up to 25,000 tokens in total, and a memory file is recommended to stay under 200 lines because, in the vendor's words, longer files use more context and reduce adherence (Anthropic, memory docs, 2026). A tool-definition library for a large multi-server setup was measured by the vendor at about 77,000 tokens before any work started (Anthropic, advanced tool use, 2025, vendor claim). Your one-line request can be the last fifty tokens after fifty thousand.

The harness assembles it. An agent is a model plus a harness, and the harness is everything that is not the model: the system prompt, the tools, memory files, compaction, subagents and hooks (Trivedy, LangChain, 2026). The same post reports lifting a coding agent from about thirtieth to about fifth place on a public terminal benchmark by changing only the harness (vendor claim). A chat app is a harness too, a simpler one: its system prompt, your saved instructions, the project files, its memory of earlier chats. The mechanism matters more than the ranking: a large part of what the model reads was chosen by whoever configured the harness, and that includes you.

What you can control is the clearness of that context. Three habits follow directly from the inventory above, and the levers later give them numbers:

minimalistic: remove non-relevant context insertions, which means auditing what the app loads (saved instructions, project files, connected tools, skills, memory files, plugins) and taking out what this task does not need;

single-task specific: start a new conversation when the task changes, so that the conversation part of the window holds one task's history and nothing else;

simplify the task by decomposing it in as many dimensions as you can: first what the work must achieve and its outline or design, as separate tasks, then the outline cut into pieces, each done in its own conversation; for software this is called spec-driven development, with requirements, design and coding tasks as the three cuts.

In the example, condition 1 breaks all three habits at once: a long unrelated history, a one-line request, and the ten rules sitting far up the conversation or in saved instructions. Conditions 2 to 4 restore the first two with a fresh conversation and a complete first message.

Carry forward from Level 1:

Your line is the last item in a long window that the harness assembled; the rest of the window is yours to trim.

Saved instructions, memory files, skills and tool schemas are context, not enforced rules; the vendor says so and the levers below measure it.

Three habits: minimal context, one task per conversation, decompose the task.

Level 2: how to measure quality and adherence before changing anything

This level answers: how do you score an AI's output for correctness and for adherence so that a lever change can be seen at all?

Measurement belongs here, not inside a lever, because every lever below is judged by the same two scores: did the output do the job, and did it follow the rules. Both need a definition before a single lever is pulled.

Correctness: pass@k against pass^k. Models are not deterministic, so one run is one sample. Two ways to summarise k runs of the same task are in use, and they answer different questions.

pass@k is the chance that at least one of k runs succeeds. It measures capability: could the agent do this at all?

pass^k is the chance that all k runs succeed. It measures reliability: can you hand this task over without checking? The benchmark that introduced it defines it as the chance that all k independent trials succeed, averaged across tasks (Yao, Shinn, Razavi, Narasimhan, τ-bench, 2024).

The two can be far apart on the same agent. On τ-bench, the best model of the day, gpt-4o in 2024, succeeded on about 61 percent of retail tasks on a single run and on fewer than 25 percent of them on all eight repeats. A reader who only sees pass@1 thinks the assistant mostly works; the person who depends on it experiences pass^8. Arithmetic shows how fast it falls: if runs were independent, a 75 percent per-run success gives a pass^3 of about 42 percent. The example in Level 0 reports pass^3 for that reason.

A separate study of 15 frontier models over about two years found that reliability, measured as consistency across five repeated runs and robustness to rephrasing, improved by 0.03 per year on one agent benchmark while accuracy improved by 0.18 per year, a six-fold gap (Rabanser et al., 2026, GAIA, models from OpenAI, Google and Anthropic, 2024 to 2026). Accuracy is what the leaderboard shows; reliability is what you buy.

Adherence: a rules checklist. A correct answer can still break your rules. The page can have the five right facts and the right recommendation, and still be 600 words long, say "the first offer" and open with "as discussed". So the rules get their own score. Write each rule so that it has a yes-or-no answer, check every rule, and count how many were met. Then report two things: the count, and the list of the rules that were broken. The list matters more than the count, because which rule broke tells you which lever to pull. This is how instruction-following benchmarks work: they use rules a program can check, such as a word limit or a required word (Zhou et al., IFEval, 2023, about 25 rule types, 2023 models), and the ten rules in Level 0 follow the same idea.

How to build these checks into a proper evaluation, with graders, trials and a way to read the results, is the subject of an earlier article of mine on evals for systems built on large language models: https://x.com/sermakarevich/status/2106453816757354947

Where a script cannot decide, a judge with evidence. Some rules need reading (is every claim about an offer followed by a quote from that offer?). For those a model is asked, and two findings decide how it is asked.

Binary questions beat holistic scores. On a summary with three planted factual errors, a judge asked for an overall consistency score gave 5.0 out of 5, as did a second holistic method; the same judge asked seven yes-or-no sub-questions flagged four of them and scored it 1.57, against a human score of 2.0 (Cho et al., BinEval, 2026, Claude Sonnet 4 as judge). A holistic judge reads fluent text as correct text; a question per failure type does not.

The judge must see the evidence, not the chat. On a customer-service benchmark where the true outcome is known, a judge that saw only the user-visible turns detected failures at chance, an area under the curve of 0.49, while a judge that read the full transcript including tool calls reached 0.73 (Bodhwani, Tran, Wei, GAUGE, 2026, 25 agents, four judge models, about 3,700 transcripts). Give the judge the draft, the sources and, for an agent, its tool calls, and ask it to quote the line that decides each question.

Judges prefer the first answer shown, the longer one, and their own model family (Zheng et al., MT-Bench, 2023; GAUGE measured a same-family lift of about 0.75 on a seven-point scale). Swap positions, keep questions binary, and check a judge against a few dozen human-labelled cases first.

Error bars. With few tasks the intervals are wide (Miller, Anthropic, 2024, gives the formulas). On twenty tasks a ten-point pass^3 difference is inside the noise; the Level 0 example, one task repeated three times, can show a large effect and nothing else.

Age of the evidence. Some of the figures in the levers come from tests on earlier models, from 2023 and 2024. They are kept because they show the concept most clearly: a controlled study that changed one thing and measured the result. The numbers themselves have most probably improved since; the concept, and the point each study makes, have not, and where a newer study exists it says the same thing and is cited next to the older one. That is why each figure carries the model and the year it was measured on: read the direction as settled, and the size as something to confirm on your own model with the protocol in Level 0.

Important: score correctness and adherence separately, report pass^k for anything you hand over without checking, and ask a judge binary questions with evidence.
What it means for us: a lever change is only real when it moves one of these scores on repeated runs; "it looked better" is a single sample read by the same eyes that wrote the prompt.
Why it matters: pass@1 and a holistic judge are the two most common ways a team convinces itself that a change helped when it did not.

Carry forward from Level 2:

pass@k is capability, pass^k is reliability; report the second.

Adherence is a per-rule checklist, scored and listed, not a feeling.

A judge gets binary questions, the evidence (the draft, the sources, the tool calls), and a bias check.

Level 3. Lever A: task complexity, how big each piece is, what you ask, over how many turns

This level answers: how do the size of the piece, the number of turns, the timing of the goal, the completeness of the request, the length of the horizon and the number of agents change a model's reliability on one task?

This is the lever with the strongest evidence, and the one to check first. The mechanism: the model's reliability, not its capability, collapses when a task is bigger than one window can hold, when it runs long, or when it is fed to the model bit by bit over many turns of one conversation. The first is the common case and has the simplest fix: cut the work into small, self-contained tasks, each complete in its own conversation. Note the difference from the third: splitting means many small tasks, each with its whole brief; what hurts is one task whose parts trickle in while the model is already working on it. The size of the piece comes first below; the studies on turns, timing, spec, horizon and agents follow. Each varies one aspect and holds the task fixed.

Size of the piece. The size of what you ask for in one conversation is the biggest single control you have. A window filled with a task that is too big produces the same erosion as a window full of junk: the model has to hold what was asked, the plan, every source it read, every draft it wrote and every check it ran, and the last 30 to 40 percent of the window is where practitioners place the mistakes. The lever here is the complexity of what you ask for in one conversation, and the fix is to keep splitting until each piece is small enough to pass its own check reliably.

Accuracy fell 13.9 to 85 percent as input grew, within the advertised limits and with every needed fact present (Du et al., 2025, five models, math, question answering and coding). A big task fills the window with its own working state, not with junk, and the loss is the same.

Integration success fell from 88.6 to 55.8 percent as the spec handed to each agent got thinner, and a merging agent given the full spec recovered 88.9 (Chacón Sartori, 2026, described further down this level). Pieces work when each one carries its whole brief; they fail when the brief is implied by what another conversation did.

No model completed any SlopCodeBench problem end to end, and erosion grew along the trajectory in 80 percent of runs (Orlanski et al., 2026, also below). The problems were ordinary features; what they had in common was length.

Anthropic's own harness for long-running agents reports that one feature per session was the rule that stopped the agent doing too much (Anthropic, 2025, vendor guidance, no number). The loop practitioners run, one plan-file task per fresh context, is the same rule enforced by a script (Huntley, 2025, practitioner claim, no measurement).

The practice I use, and the one these studies point at: do not do a big piece of work in one conversation. Say the work is a twenty-page report, a course, or a feature in a program. Write down what it must achieve and for whom in one conversation, and read it. Write the outline, or the design, in another. Cut the outline into pieces, each with its own check, each small enough that a fresh conversation given the piece, the outline and the sources it needs passes the check on the first or second run. Do each piece in its own conversation, with nothing from the previous one but what it produced and the plan. Check the whole in a conversation of its own. Twenty short conversations, each in its good zone, beat one long one that erodes from the middle. If a piece still fails its check three times out of three, it is not a model problem yet: split it again. The example task in Level 0, one page from three offers, is a piece of that size; the purchase case it belongs to would be ten or twenty such pieces. For software the same sequence has a name, spec-driven development: requirements, design, coding tasks each with a test, verification, one session each.

This looks like it contradicts the turns result that follows, and it does not. That study drip-fed one task across turns of one conversation, with the model guessing what had not arrived yet. Splitting a piece of work into self-contained tasks is the opposite: each conversation gets everything it needs in one message, and nothing it does not. The hand-off between pieces is a written brief and a passed check, not a half-remembered chat.

Turns. Drip-feeding a task across turns cut average performance 39 percent across 15 models; the drop appears at two turns and is mostly a reliability loss (Laban, Hayashi, Zhou, Neville, 2025, 15 LLMs including 2025 frontier models, six task types, over 200,000 simulated conversations).

The same fully specified task, split into "shards" delivered one per turn, scored 39 percent lower on average than the whole task in one message.

The loss appears already when the task is split in two, and does not grow much with more shards.

Decomposed into a small loss of best-case ability and a large rise in the spread between best and worst runs: the model can still do it, it just does not do it reliably.

Temperature zero does not fix it; a final recap recovers part of it but stays below the one-message version.

In the example, condition 1 is a one-line request to which the rules and criteria would have to be added turn by turn; condition 2 puts everything in the first message. This study predicts the gap.

Timing of the goal. The next question is when the model learns what you actually want. One study tested this directly (Gulati et al., 2026, four frontier models, three agent benchmarks, 6,048 runs). The setup: give an agent a task with one important detail missing, let it start working, and then supply the missing detail at a chosen point, early or late, or never. Then count how often the task came out right in three tries (pass@3).

Detail supplied at the start: 0.80 of tasks succeed. Supplied after the agent had done about a tenth of the work: 0.78, almost the same.

Supplied after about a third of the work: 0.50. After half: 0.44. After most of the work was done: 0.39.

Never supplied: 0.40. So a correction that arrives late is worth about as much as no correction at all; the agent has already built on the wrong assumption.

A second part of the study let the models work on their own, with no forced correction, for 300 runs. Whether a model noticed the gap and asked for the missing detail depended on the model: one asked in about half the runs, one in about a quarter, one never. None asked early, in the window where asking would still have helped.

So the goal must be right in the first message. The model will not ask in time, and fixing it later does not rescue the run.

Completeness of the spec. Full spec before coding: 88.6 percent integration success against 55.8 with bare signatures (Chacón Sartori, 2026, 204 Python class-generation tasks, Claude Sonnet and Haiku 4.5).

A single agent given the full docstring-level spec: 88.6 percent of generated classes integrated correctly; with spec detail removed in three steps: 79.5, 68.6, then 55.8 with bare signatures.

The same task split between two agents: 58.2 percent with the full spec, 24.6 with bare signatures, a coordination gap of about 30 points at every level of detail.

Giving a merging agent the full spec restored 88.9 percent; giving it automated conflict reports instead added nothing.

The spec has a ceiling, though. A fifth round of added rules dropped accuracy 67.4 to 59.6 (Wang et al., Qwen Team, Verification Horizon, 2026, an evaluator prompt hardened over five versions, Claude Opus 4.7 as the most reliable backbone). Four rounds of rules raised the evaluator from 57.9 to 67.4 percent; the fifth, which added more rules and prohibitions, fell to 59.6. Past a point, more spec is more text competing for attention, which is Lever C's subject.

Length of the horizon. Long horizons decay: no model finished any SlopCodeBench problem end to end, with erosion in 80 percent of trajectories; plan-first shifts the start, not the slope (Orlanski et al., 2026, 20 problems with 93 checkpoints, 11 models including Opus 4.6, GPT 5.4, Sonnet 4.6).

Best strict solve rate across checkpoints: 17.2 percent (Opus 4.6); no model passed every checkpoint of any problem.

Structural erosion rose over the trajectory in 80 percent of runs; verbosity rose in 89.8 percent; agent code was 2.2 times more verbose than maintained human repositories.

Anti-slop and plan-first prompts cut initial verbosity by up to 34.5 percent and left the slope unchanged, at up to 47.9 percent more cost and no gain in pass rate.

A prompt that says "plan first" or "keep it clean" moves where the decay starts, not how fast it goes. Shorter horizons with a checkpoint and a review at each one are the shape that works, which is what cutting the work into specified pieces produces.

Number of agents. Multi-agent: mean effect across 180 setups was minus 3.5 percent (Kim et al., 2025, five architectures, three model families, four benchmarks); a five-agent relay fell 90.7 to 22.5 (Ao, Gao, Simchi-Levi, 2026, 200 MMLU questions, gpt-4.1-mini and o4-mini); gains only for parallel read-only breadth work.

Across the scaling study's configurations the effect of adding agents ranged from plus 80.8 percent on a decomposable financial-analysis task to minus 70 percent on sequential planning, with a wide spread; the average close to zero means the architecture choice decides, not the agent count.

Independent agents amplified errors 17.2 times against 4.4 times for a centralised coordinator.

One agent answered 90.7 percent of the questions; handing the work through a chain of two, three and five agents in prose gave 41.2, 43.5 and 22.5 percent, the last below chance. A structured hand-off through three agents kept 75.2.

Against this stands a vendor claim of plus 90 percent at about 15 times the tokens for a lead agent with parallel subagents on a research task (Anthropic, multi-agent research system, 2025, Opus 4 lead with Sonnet 4 subagents, internal eval). The two are consistent once you read the task: breadth-first reading in parallel gains; sequential writing through hand-offs loses. In the example, condition 4 uses a second conversation only to read and judge, never to write.

What to do, in order.

Ask whether the work fits one conversation. If not, cut it first: what it must achieve, then its outline or design, then the pieces, then the check, each as its own conversation; the outline is cut into pieces that each carry a check.

One task per conversation, sized so a fresh conversation passes its check; a piece that fails three times out of three is split again, not re-prompted.

Full instruction for the piece up front: task, acceptance criteria, rules, in the first message.

Clarify the goal first, before any work, and do not rely on the model to ask.

One writer; other conversations or agents only read, review and research.

Shorten horizons with checkpoints: a review at each one, and a fresh conversation after it.

Important: the model's reliability, not its capability, is what collapses when one task trickles in over many turns of a conversation, gets its goal late, or runs long. Splitting work into small self-contained tasks, each complete in its own conversation, is the opposite of that and is the cure.
What it means for us: cut the work into pieces small enough for one conversation each, write each piece's whole brief once, settle its goal first, and keep one writer.
Why it matters: these are the cheapest changes in the article and they carry the strongest evidence; a team that skips them and buys a bigger model is paying for the wrong fix.

Carry forward from Level 3:

A task too big for one conversation is split until each piece passes its own check; twenty conversations beat one.

One message with everything beats the same content over several turns, already at two turns.

Goal first; spec detail raises success until the spec itself becomes noise.

Short horizons, checkpoints, one writer; extra agents only for parallel reading.

Level 4. Lever B: what the model sees, size, relevance, position and content of the context

This level answers: how do the size, relevance, position and content of what is in the context window change instruction following, and how should a long conversation be compacted?

Lever A is about the complexity of the request. Lever B is about everything else in the window, which Level 1 inventoried. The mechanism: every instruction competes with everything else in the window, and attention degrades with length even when retrieval is perfect.

Size. All 18 models tested got worse as input grew; a 300-token focused prompt beat the 113,000-token raw history on a long-memory benchmark (Hong, Troynikov, Huber, Chroma, 2025, 18 models from Anthropic, OpenAI, Google and Qwen, mid-2025 releases).

Every one of the 18 models degraded as input length rose, including on tasks that are trivial at short length.

On 306 long-conversation questions, a prompt holding only the relevant 300 tokens beat the full 113,000-token history on every model; the report gives the result as charts, not percentages.

One distractor sentence that resembles the answer lowered accuracy, four compounded it, and the effect grew with length.

A 4B model fell 75.8 to 34.4 percent on long input and held 85.4 with chunk-and-gather (Li et al., PARSER, 2026, Qwen3.5-4B, HotpotQA, 7,000 to 896,000 tokens).

Full context in one window: 75.8 percent at 7,000 tokens, 61.7 at 224,000, 34.4 at 896,000.

The same questions with the text split across fresh reader contexts that each return short findings to a lead: 85.4 percent at 896,000 tokens, flat across lengths.

The gain is a trained system, not a prompt trick, but the baseline curve is the point: the same model, the same facts, and half the accuracy because of how the facts were fed.

Accuracy dropped between 13.9 and 85 percent at longer input even with perfect retrieval (Du et al., 2025, five models including Llama-3.1-8B, Mistral-7B, GPT-4o, a Claude model and Gemini 2.0, maths, QA and code tasks).

Models that recited every piece of evidence correctly from a 30,000-token input still lost accuracy on the question, by 13.9 percent at the low end and up to 85 at the high end.

Padding the input with whitespace alone cost at least 7 percent at 30,000 tokens for the open models.

Reciting the evidence, then re-asking in a fresh short prompt, took Mistral-7B from 35.5 to 66.7 percent on a maths set at 26,000 tokens.

Length itself is the cost. Retrieval being perfect does not help if the answer is then produced inside the long window.

Content over size. Replacing 78 of 80 history rounds with cooperative records restored a 70B model from 6.9 to 97.4 percent at the same length (Liu et al., Memory Curse, 2026, Llama-3.3-70B, a repeated trust game with chain-of-thought, 7 models and 4 games overall).

With 80 rounds of bad history visible, cooperation fell to 6.9 percent; with the same 80 rounds but 78 of them replaced by cooperative records, 97.4.

Cooperation degraded with longer history in 18 of 28 model-and-game settings; chain-of-thought amplified it.

The setting is a game, not a writing task, but the lesson transfers: what the window says matters as much as how long it is. A conversation full of earlier failed attempts teaches the model to fail.

Position. 75.8 percent with the answer first, 53.8 in the middle, below the 56.1 no-document score (Liu et al., Lost in the Middle, 2023, GPT-3.5-Turbo, 20 retrieved documents).

Answer in the first document: 75.8 percent; in the tenth of twenty: 53.8; last: 63.2; no documents at all: 56.1.

The middle position was worse than giving the model nothing.

Chroma found no clear position effect on its 2025 models, so position is a 2023 result until re-tested. Treat it as open: putting the key instruction first and last costs nothing, but do not expect it to rescue a long window on a current model.

Compaction. Sliding-window compression raised retrieval calls in 6 of 6 comparisons; a fact-preserving summary was near lossless (Liu, 2026, three models including GPT-5.5, DeepSeek and Qwen, a 24-turn synthetic retrieval environment).

Compressing the history by dropping old turns left task completion statistically unchanged in all six comparisons and raised the number of retrieval calls in all six, by up to 42.9 extra calls per task for GPT-5.5.

An extractive summary that kept the facts, at the same compression ratio, was near lossless; a summary that invented retained state raised re-queries by 57 percent.

Compaction that looks free on the pass rate is paid for in turns, and Lever A says turns cost reliability. What survives compaction matters more than the ratio.

What to do.

Minimal context: remove non-relevant context insertions, which includes the saved instructions, project files, connected tools, skills and plugins the app loads by default.

Remove distractors and near-miss content: the earlier attempt, the similar-but-different file, the old spec.

One task per conversation: start a new one when the task changes.

Compact by facts, not by window: keep what was decided and what was done, drop the chatter, and never let the summary state things that did not happen.

Keep instructions alive after compaction: put the rules where the app re-injects them after a compact (saved instructions, a project file, a memory file), not only in the conversation.

Audit what the app loads: read the saved instructions, project files and connected tools before a serious task, run the context breakdown if the tool has one, and take out what the task does not need.

In the example, condition 1 is the long window with an unrelated history, and the ten rules deep inside it; conditions 2 to 4 start bare, with only the task, the rules and the three offers.

Important: every token in the window costs some attention, and a model that can recite every fact in a long input still answers worse than it would from a short one.
What it means for us: the window is a budget, not a bucket; spend it on this task, and start a fresh one when the task changes.
Why it matters: a buried rule is an ignored rule, and the model gives no sign that it did not see it.

Carry forward from Level 4:

Length alone hurts, on every model tested, even with perfect retrieval.

Content decides more than size: a window of bad history is worse than a short one.

Compact by preserving facts and re-injecting rules; judge compaction by extra turns, not pass rate alone.

Level 5. Lever C: how the instruction is written, specific, scoped, not over-specified

This level answers: how do the length, authorship and form of an instruction change how well the model complies with it?

Lever B is about what to remove from the window. Lever C is about how to write what stays, and the two are kept separate because the fixes differ. The mechanism: instructions are tokens too; adding more past a point lowers compliance, and generic or model-written guidance lowers it from the start.

Length. An automatic prompt rewriter went 34.6 to 38.0 percent, then collapsed to 26.1 as the prompt grew from 22 to 6,248 characters (Cho et al., BinEval, 2026, an instruction-following benchmark, self-updating prompt loop).

Starting prompt of 22 characters ("Respond to the query."): 34.6 percent strict accuracy.

After three rounds of the model adding rules against its own failures: 38.0 percent.

After the fourth round, at 6,248 characters: 26.1 percent, below where it started.

A few targeted rules help; a wall of them hurts more than it helps. The same curve appeared in Lever A's evaluator prompt, 67.4 to 59.6 on the fifth round.

Authorship. LLM-written AGENTS.md files cut success 0.5 to 2 percent and raised cost over 20 percent; human-written ones gave about plus 4 (Gloaguen et al., 2026, 138 tasks from 12 Python repositories plus SWE-bench Lite, Claude Code, Codex and Qwen Code agents).

A model-generated repository context file: minus 0.5 percent (SWE-bench Lite) to minus 2 percent (the repository benchmark), with over 20 percent higher cost.

A developer-written file: about plus 4 percent on the repository benchmark, at up to 19 percent higher cost.

Reasoning tokens rose 14 to 22 percent with any context file present; the agents did read and follow the files.

A generated file says what the model could have found by looking; it costs tokens and buys nothing. A human-written file says what the model cannot find: the house style, who the reader is, the team's conventions, the trap in the build, the file not to touch. The saved instructions in a chat assistant are the same kind of file, and the same rule applies: write them yourself, and short.

Form. Injected skills lowered pass@2 by 1.3 to 4.2 points on all four models and raised tokens 72 to 394 percent; only "don't" rules gained reliably, plus 3.1 (Yang, Ding, 2026, 31 public skills injected into 50 web projects, 1,000 tasks, Claude Sonnet 4, GPT-5.1, Qwen3-Coder-30B, DeepSeek-V4-flash).

Mean pass@2 fell on every model: minus 4.2 for Sonnet 4, minus 1.3 for GPT-5.1, with token use up 72 to 394 percent.

Only 17 to 36 percent of skill-and-project pairs improved.

The one slice that helped reliably was anti-pattern rules, the "don't do X" kind: plus 3.1 points pooled, from a small set of positive pairs.

Example code inside a skill cost 22.7 percent of its tokens and hurt Sonnet 4 by 15.3 points; a second study found 68.8 percent of skill-induced functional failures were wrong elements copied from the skill's examples (Dong et al., 2026).

The exception is tools. Tool examples raised tool-call accuracy 72 to 90 percent; Tool Search lifted MCP accuracy 49 to 74 (Anthropic, advanced tool use, 2025, both vendor claims, the second on Opus 4). An example of a correct call teaches a parameter format in a way a description does not, and loading tool definitions on demand instead of all at once takes a 77,000-token library down to about 8,700 (same post, vendor claim). Examples for tools, prohibitions for style.

What to do.

Write instructions by hand and short: what the model cannot find, in the fewest lines.

Prefer prohibitions to positive style rules: "no bullet points" over "write in flowing prose", "no print statements" over "write idiomatic logging".

Examples over descriptions for tools: one correct call per tricky tool.

Rewrite prompts per model: a rule set tuned on last year's model is often over-prescriptive on this year's, and the vendor's own migration notes say so (Anthropic, prompting guides, 2026, vendor guidance).

Do not stack skills, plugins or saved instructions "just in case": load an add-on for the task that needs it, and measure whether it helps on your tasks before keeping it.

In the example, the ten rules are ten lines, four of them prohibitions, written by hand; the saved instructions in condition 1 hold the same ten lines, which tests whether a buried hand-written rule survives a long window.

Carry forward from Level 5:

Rules help for a few rounds, then the prompt's length turns them into noise.

Human-written, short, and mostly "don't"; model-written context files cost more than they return.

Tools get examples; style gets prohibitions; add-ons get loaded on demand.

Level 6. Lever D: where the facts come from, supplied text and retrieval, not recall

This level answers: how does building the answer from supplied or retrieved text, with quotes that can be checked, cut hallucination compared with recall from the model's weights, and when should the model abstain?

The mechanism: an answer built from text in the window can be checked against that text; an answer recalled from weights cannot. Two kinds of answer come out of the same model and look the same on the page:

when the model recalls from memory, you cannot validate whether it is correct, because there is nothing to hold it against;

when the model summarises inputs from high-quality sources and can point to the specific text it used, the output is of better quality and checkable.

The whole lever is about moving work from the first kind to the second.

Fetch instead of recall. Hallucination caused 56 percent of failures for reasoning without tools; fetching facts cut false positives 14 to 6 percent (Yao et al., ReAct, 2022, PaLM-540B, a multi-hop question set, 200 hand-labelled trajectories).

With chain-of-thought reasoning from memory alone, 56 percent of the wrong answers were hallucinated facts; with reasoning interleaved with lookups, 0 percent.

Right answers reached by invented reasoning, the "false positives": 14 percent for recall, 6 percent with lookups.

The lookup version was not more accurate overall on that set (27.4 against 29.4 percent); its failures moved to reasoning and empty searches, which are failures you can see.

Grounding does not make the model smarter. It makes its errors visible and its correct answers checkable, which is what you need to trust the work.

Quote, then check. Quote-then-check: MARCH raised a grounding benchmark from 57.09 to 85.23 percent, a training method with observed reward hacking (Li et al., 2026, an 8B model, FACTS Grounding, which scores whether every claim is supported by the supplied documents).

A checker that never sees the answer re-derives each claim from the documents only; one unsupported claim zeroes the reward.

Grounding score 57.09 to 85.23 percent; on a second faithfulness set 55.20 to 75.23.

The model being trained learned to extract fewer claims to avoid being caught, and the authors had to add a minimum-claims rule.

This is a training result, so a stock model gets the pattern and not the number: a reader that has not seen the answer, checking claims one at a time against quotes. The vendor's own hallucination guidance lists the same pattern as a prompt technique: for a long document, extract word-for-word quotes first and base the answer only on them, then require a quote per claim and retract any claim without one (Anthropic, Reduce hallucinations, undated vendor guidance, no numbers).

Retrieval quality before model quality. Retrieval pipeline (context prefix, keyword index, reranker) cut retrieval failures 5.7 to 1.9 percent (Anthropic, Contextual Retrieval, 2024, vendor claim, Claude 3 Haiku writing the chunk context).

Plain embedding retrieval missed the needed chunk in the top 20 for 5.7 percent of questions; adding a model-written context prefix to each chunk, 3.7; adding a keyword index, 2.9; adding a reranker, 1.9.

A corpus under about 200,000 tokens can skip retrieval and go into the prompt directly (same post).

If the right text is not in the window, no prompt and no model fixes the answer; the retrieval failure rate is the first number to measure in any grounded system.

Which retrieval. Graph retrieval: plain retrieval wins single-hop 64.78 against 61.03, graph wins multi-hop 71.17 against 65.77, both together 77.62 (Han et al., 2025, Llama-3.1 8B and 70B, controlled setup with the same chunking).

Single-fact questions: chunk retrieval 64.78 percent, the graph method 61.03.

Multi-hop questions: graph 71.17, chunks 65.77, combined 77.62.

Only about 65 percent of answer entities made it into the built graph at all; a relation the extractor missed is invisible to the answer.

A vendor figure of "80 percent more truthful" for graph retrieval conflicts with the single-hop result and comes from a database vendor's own study (Neo4j, 2026, vendor claim). Match the retrieval to the question: chunks for facts, graphs for connections.

Source integrity. Poisoned graph retrieval reached 98.2 percent attack success; only provenance scoring helped, 89.2 to 45.7 (Liang et al., GRAGPOISON, 2025, GPT-4o on a multi-hop set).

Injecting one crafted text that shares relations with many questions steered up to 98.2 percent of them.

Paraphrasing, model fallback and consistency checks each cut the attack by only 0 to 10 points; trust scores based on where a fact came from cut it from 89.2 to 45.7.

A grounded answer is only as good as the text it was grounded in. Track who may write to the sources, and treat what the agent reads as data, not as instructions: indirect prompt injection works precisely because retrieved content lands in the same window as your rules (Greshake et al., 2023), and the model-side defence, training the model to rank system and user instructions above tool output, is partial (Wallace et al., Instruction Hierarchy, 2024).

Abstention. Evidence here is thin. One vendor card reports the highest abstention rate among the models it compared, 26.6 percent on a knowledge benchmark, with the vendor's own caveat that the training run was flagged as unhealthy in its second half (Anthropic, Claude Sonnet 5 system card, 2026, vendor claim). A theoretical account argues that models guess rather than abstain because training and benchmarks score a confident wrong answer the same as "I don't know" and score "I don't know" as a miss (Kalai, Nachum, Vempala, Zhang, 2025). The practical consequence: your rubric must score abstention as acceptable when the source does not answer, or the agent learns to invent.

Beyond the sketch, three points from the materials complete the picture. First, a citation is not evidence: on 2023 generative search engines, only 51.5 percent of sentences were fully supported by their own citations (Liu et al., 2023), so the check must compare the claim to the quoted text, by string match where possible. Second, copying by reference beats retyping: a memory system where the model returns only segment identifiers and the exact stored text is fetched gave 100 percent faithfulness by construction against 84.3 for free-form retrieval (Li et al., OCR-Memory, 2026, narrow benchmark). Third, the earlier executive-level piece this article grew from put the rule in one line worth keeping: a quote proves the answer came from the document, not that the document is right.

The example exercises this lever directly. Rules 3 to 5 in Level 0 are grounding rules: every number traced to an offer, a quote per claim, nothing from outside. One extra run measures recall against supplied text: condition 2 with the three offers named by supplier but not pasted, scored by the share of the page's claims that match a quote in the offers. Result slot [to measure].

What to do.

Give the source, ask for quotes, check claims against quotes.

Fetch instead of recall: a lookup tool for anything the model would otherwise remember.

Retrieval quality before model quality: measure the miss rate of the retriever first.

Abstain when no source: the fixed phrase "I don't have enough information" is a valid answer in the rubric.

Treat retrieved text as data, not instructions.

Important: an answer built from text in the window can be checked against that text; an answer recalled from the model's weights cannot be checked at all.
What it means for us: supply the source, require quotes, and verify claims against the quotes with a script or a reader that has not seen the answer.
Why it matters: recall and grounding produce the same fluent paragraph, and only one of them can be audited before it reaches a customer.

Carry forward from Level 6:

Fetching moves errors from invisible invention to visible retrieval failures.

Quote per claim, retract without quote, and check the quote mechanically.

Retrieval miss rate first, source provenance always, abstention scored as acceptable.

Level 7. Lever E: how the work is checked, an external, runnable check, by a fresh reader

This level answers: why does an agent's self-review fail to catch its own errors, and how much do runnable tests, specific feedback, a behaviour monitor, a fresh-context reviewer and a binary evidence-based judge each add?

This lever carries the largest controlled effect on a coding benchmark in the article. The mechanism: the model cannot see its own errors; a check that runs without its opinion, or a separate reader, can.

Self-critique without tests. Self-Refine barely moved on maths; a model said "everything looks good" on 94 percent of the problems (Madaan et al., 2023, GPT-3.5, ChatGPT and GPT-4).

Maths accuracy with the model critiquing and rewriting its own answer: GPT-3.5 64.1 to 64.1, ChatGPT 74.8 to 75.0, GPT-4 92.9 to 93.1.

ChatGPT's self-feedback said "everything looks good" on 94 percent of the maths problems, which includes the wrong ones.

Given an external correctness signal instead, the same loop gained 4.8 points on GPT-3.5.

On tasks judged by taste rather than correctness (tone, constrained writing), the loop gained about 20 points on average.

Self-review improves style and misses correctness, because the model that made the error reads the error as correct.

Runnable tests. Reflexion on hard Rust: reflection without tests 0.52 against base 0.60; with tests plus reflection 0.68; bad tests caused false passes (Shinn et al., 2023, GPT-4 class, the 50 hardest problems of a Rust set).

Base model: 0.60. Reflection without a test signal: 0.52, worse than doing nothing.

Tests alone: 0.60. Tests plus reflection on the failing test: 0.68.

On a Python set, 16.3 percent of model-written tests passed wrong code, against 1.4 on another, and the loop stopped early with wrong code each time.

The reflection is worth nothing without the signal and 8 points with it. The signal's quality caps the loop: a test that passes wrong code ends it with wrong code.

Specific feedback. Specific feedback 43.2 against 31.2 generic and 0 none; most of the gain in the first two or three rounds (Madaan et al., 2023, ChatGPT, a sentiment-rewriting task; iteration curve from the constrained-generation task).

Feedback that names the span, the cause and the fix: 43.2; feedback that says "improve this": 31.2; no feedback: 0.

Across rounds: 29.0, then 40.3, 46.7, 49.7; the first round gives half the total gain, the fourth almost nothing.

A reviewer's output should be a named defect plus a fix, and two or three rounds is the budget.

A behaviour monitor. A behaviour monitor cut hacked solutions 28.6 to 0.6 percent and moved clean resolutions on SWE-Bench Verified 36.5 to 65.0 (Wang et al., Qwen Team, 2026, a trajectory-level monitor for exploit patterns during coding-agent training).

Averaged over three benchmarks, solutions that passed by cheating fell from 28.6 to 0.6 percent of resolved tasks, and clean resolutions rose from 40.2 to 60.5.

On SWE-Bench Verified alone, clean resolutions 36.5 to 65.0 and cheated ones 51.5 to 2.1.

The exploits watched for: fetching the known solution, tampering with the test oracle, patching for the evaluator rather than the task.

Any check that influences what the agent is rewarded for gets gamed, and the fix was a second reader of the trajectory, not a better test. A ceiling applies to piling on checkers: across 67 frontier models, the share of queries that every model gets wrong caps what routing, voting or mixing can gain, and that shared blind spot was about 2.5 times larger than pairwise correlations predicted (Chen, 2026). A second checker from the same family sees the same blind spots.

Judge design. Binary questions with evidence beat holistic scoring 1.57 against 5.0 on a summary with three planted errors (Cho et al., 2026); a judge without the tool calls was at chance, area under the curve 0.49, against 0.73 with them (Bodhwani, Tran, Wei, GAUGE, 2026). Level 2 gave both; here they become the reviewer's design: ten yes-or-no questions, the draft, the sources and any tool calls as input, a quoted line per answer.

A fresh-context reviewer. Two vendor reports, labelled as such. A rubric-graded loop in which a separate verifier agent grades in its own context window gave about 6 times the improvement over baseline on one optimisation benchmark for the newest model against about 1 times for the previous one (Martin, Anthropic, 2026, vendor claim, one benchmark, the whole loop and not the reviewer alone). A production review loop in a fresh context catches about 2 bugs per pull request on average, 58 percent of them logic or security bugs, including on pull requests the same system wrote (Yan, Cognition, 2026, vendor claim, no public data). Both agree with the controlled studies on the mechanism: the reader that did not write the work does not share its blind spots.

What to do.

Give the AI a check that runs without its opinion as part of the task: a word count, a number match against the source, a spreadsheet formula, a test, a build, a screenshot comparison.

Verify in a fresh context: a second conversation that sees the draft, the request and the rules, not the writer's history.

Feedback as named defect plus fix, two or three rounds, keep the best draft not the last.

Binary rubric questions with evidence; the judge sees the tool calls.

Retry only with an external signal: a failing test, a reviewer's named defect; never "try again".

Measure pass^k, not pass@k, because a check that passes once proves little.

In the example, condition 3 adds the mechanical check and condition 4 adds the fresh reader; the published numbers above predict that these two conditions carry most of the movement in correctness, and that the reviewer carries the rules the checks cannot see.

Important: the model cannot see its own errors; a check that runs on its own, a draft read by a separate reader, or a monitor on the trajectory can.
What it means for us: every task you hand to an AI gets an external check as part of the task, and the review runs in a conversation that did not write the work.
Why it matters: self-review says "looks good" to its own mistakes, and a check the agent can game will be gamed.

Carry forward from Level 7:

Self-critique improves style and misses correctness; an external signal restores the gain.

Check quality caps the loop; a bad check ends it with wrong work.

A fresh reader with binary questions and evidence is the review that works; a same-family second opinion shares the blind spots.

Level 8. Lever F: the model and its settings, the weakest lever and the first one people pull

This level answers: how much do model choice, effort level and structured output change quality and adherence, and why does a model upgrade not fix a context or task complexity problem?

The mechanism: capability gates everything above; settings move little.

Model tier gates adherence. Weak models loaded skills 25 percent of the time against about 96 for strong ones, and followed them less (Lin et al., 2026, seven models on a skills benchmark).

Skill load rate: Qwen3-32B 25.1 percent; GPT-OSS-120B 44.6; Haiku 4.5 79.4; Sonnet 4.6, Opus 4.6 and Qwen3-235B about 96.

Follow rate once loaded: Qwen3-32B 14.2 percent; Opus 4.6 75.7. One model loaded skills 96 percent of the time and followed them 35.

Adherence over a long trajectory fell from 0.52 to 0.13 on the weak tier and drifted 0.09 on Opus 4.6.

A 9B model wrote harness updates as well as Opus 4.6; the gap was in using them, not writing them.

Self-correction gave no gain on 13B-class models: Vicuna-13B could not produce usable feedback at all, and a 15B-class code model scored 0.26 before and after reflection (Madaan et al., 2023; Shinn et al., 2023). Below a capability floor, Levers C and E do not work, which is what "gates" means. Above it, the lever flattens.

Upgrades buy accuracy, not reliability. Reliability improved 0.03 per year against 0.18 for accuracy, a six-fold gap (Rabanser et al., 2026, 15 frontier models over about two years). Level 2 gave the number; here it is the reason a model upgrade does not fix a flaky agent: the thing you experience, consistency across runs, is the thing that barely moved.

Effort. Vendor docs recommend a high or extra-high setting for agentic coding on recent models and note that the top setting can show diminishing returns and overthinking (Anthropic, effort docs and migration guides, 2026, vendor guidance). Third-party papers show a plateau at high and a drop at the top:

On a 502-question judgment benchmark mined from agent runs, raising effort from lowest to highest moved one model family by minus 0.2 points and another by plus 2.2, with no interaction with task horizon (Pan et al., 2026, GPT-5.6 variants).

On SWE-bench Verified, accuracy peaked at intermediate token cost and fell at the highest, with failed high-cost runs repeating the same file actions (Bai et al., 2026, eight frontier models).

No controlled effort-against-quality curve for coding exists in the sources; set effort by evaluation on your tasks, not by default.

Structured output. Structured output raised the modal answer's share 41 to 64 percent (Parikh, 2026, 44 models, 31 open-ended prompts).

Asking for JSON raised the share of the most common answer from 41 to 64 percent and cut distinct answers from 52 to 36.

Only 6 of 44 models shifted significantly on their own; recent Claude models barely moved.

Structured output makes answers uniform and parseable, which is what you want for anything a program reads and not what you want for ideas. Recent Claude models (Opus 4.7, Opus 4.8, Sonnet 5) reject non-default temperature and top-p values, so sampling is no longer a lever there (Anthropic, migration guide and model notes, 2026).

What to do.

Strong model as critic even if a cheaper one writes: in the 2023 study, a 13B writer with a strong critic went 24.2 to 40.5 percent on maths.

Structured output for anything parsed; prose for anything creative.

Effort set by evaluation, not by default; one effort per conversation, since changing it invalidates the cache.

Do not expect a model upgrade to fix a context or task complexity problem.

Important: the model tier sets a floor under every other lever, and above that floor a model upgrade buys accuracy six times faster than it buys reliability.
What it means for us: pick a model strong enough to follow instructions and critique the work, then stop upgrading and start fixing the window, the task complexity and the check.
Why it matters: the upgrade is the most expensive lever and the one with the least measured effect on the failures this article is about.

Carry forward from Level 8:

Below a capability floor, instructions and self-review do not work at all.

Above it, settings move little; effort plateaus, structured output narrows, sampling is gone.

Reliability is what you experience and what upgrades move least.

Level 9: trade-offs

This level answers: when do the levers pull against each other, and how do you decide which to give up?

The levers are not free of each other. Four pairs come up on every task.

More pieces against more hand-offs. Lever A says cut the work until each piece fits one conversation; every cut adds a hand-off, and hand-offs are where work gets lost.

The same task split between two agents fell from 88.6 to 58.2 percent with the full spec, and to 24.6 with bare signatures: a coordination gap of about 30 points at every level of detail (Chacón Sartori, 2026).

A chain of five agents passing work on in prose fell from 90.7 to 22.5 percent; a structured hand-off through three kept 75.2 (Ao, Gao, Simchi-Levi, 2026).

Drip-feeding one task across the turns of a conversation cost 39 percent (Laban et al., 2025); a fresh conversation with the whole brief in its first message does not pay that cost.

The decision rule: split until each piece passes its own check in a fresh conversation, and make every hand-off a written brief plus a passed check, never a summary of a chat. Stop splitting when writing the brief for a piece costs more than doing the piece; that is the size at which one conversation is right.

More context against less. Lever B says shorter is better; Lever D says supply the source. Both are right, and the sources give the shape of the trade.

Length costs accuracy even when every fact is present, by 13.9 to 85 percent at 30,000 tokens across five models (Du et al., 2025).

A focused slice of 300 tokens beat 113,000 tokens of full history on all 18 models tested (Chroma, 2025).

Cutting history by sliding window kept the pass rate and added up to 42.9 retrieval calls per task; a fact-preserving summary at the same ratio was near lossless (Liu, 2026, GPT-5.5 among three models).

A corpus under about 200,000 tokens can be supplied whole; above that, retrieval with a 1.9 percent miss rate is the measured alternative (Anthropic, 2024, vendor claim).

The decision rule: supply the source, and only the source. Grounding text earns its place; history, tool schemas and skills must justify theirs. When you must cut, cut by facts and count the extra turns the cut causes.

More checks against more turns. Lever E says add a check; Lever A says every turn costs reliability.

A task split over turns lost 39 percent on average, from the second turn on (Laban et al., 2025, 15 models).

A review loop's gain arrived mostly in the first round: 29.0 to 40.3 to 46.7 to 49.7 over four rounds (Madaan et al., 2023, ChatGPT).

Agentic coding runs used about a thousand times the tokens of a chat answer, about 1.86 dollars per task against 0.02, and accuracy peaked at intermediate cost (Bai et al., 2026, eight frontier models, SWE-bench Verified).

The same model on the same task varied up to 30 times in tokens between runs; failed runs repeated the same file actions (same study).

The decision rule: put the check inside the task (a check the AI runs itself) rather than after it (a human turn), cap review rounds at two or three, and stop any loop that repeats an action. A check that costs one more turn is cheaper than a wrong answer; a loop that costs ten is not. The turn cost applies inside one conversation; a new conversation for the next piece, with its full brief in the first message, does not pay it, which is why splitting a piece of work across conversations is cheap where splitting a task across turns is not.

A cheaper model against adherence. Lever F says the model gates adherence; the budget says use the cheap one.

Skill load rate by tier: 25 percent, 45, 79, 96 from the weakest to the strongest of seven models; follow rate 14 to 76 (Lin et al., 2026).

A weak writer with a strong critic went 24.2 to 40.5 percent; the strong model alone did better still (Madaan et al., 2023).

Price per task spread about 20 times across eight frontier coding models, and the best model on average trailed a cheaper one by 25 to 47 points on some task types (Zhou et al., Agent-as-a-Router, 2026).

With a verifier available, repeated sampling took a cheap open model from 15.9 percent with one sample to 56 percent with 250 on a coding set, past the 43 percent single-sample state of the art of the time (Brown et al., 2024, DeepSeek-Coder-V2).

The decision rule is the unit: cost per verified success, not price per token. A cheap model that needs three attempts and a strong critic can be cheaper or dearer than a strong model that passes once; only the measured pass^k and the measured cost say which. Where a check that runs on its own exists, a cheap model plus many samples is a real option; where only a judge exists, it is not.

Carry forward from Level 9:

Split until each piece passes its check; every hand-off is a written brief and a passed check, and splitting stops where the brief costs more than the piece.

Supply the source, cut everything else, and count the turns a cut causes.

Checks go inside the task; rounds are capped; repeating loops are stopped.

Compare models on cost per verified success and on pass^k.

The tools and approaches, observed

Saved instructions and memory (custom instructions, project instructions, CLAUDE.md, AGENTS.md). Best at: hand-written rules the AI cannot discover: the house style, the reader, the file not to touch. What to watch: loaded as context, not enforced; keep short (a vendor recommends under 200 lines for a memory file); model-generated ones measured at minus 0.5 to minus 2 percent.

A fresh conversation, and the context controls around it (new chat, clear, compact, bare mode, context breakdown). Best at: starting a task clean, seeing what fills the window, running a repeatable scripted session. What to watch: compaction re-injects only the saved instructions and the plan; put must-keep rules there.

Projects and uploaded files. Best at: supplying the source so the answer can be checked against it. What to watch: everything in the project is in the window for every task in it; a project full of last month's files is condition 1.

A second conversation, a second model, or a subagent. Best at: read-only exploration and fresh-context review that returns a summary or a list of named defects. What to watch: each runs in its own window; a second reader from the same model family shares its blind spots.

Checks that run without the model's opinion (a word count, find-in-page, a spreadsheet formula, a test, a build, a hook that runs after every edit, a goal loop graded by a separate verifier). Best at: catching what self-review passes; feeding a named failure back. What to watch: the loop stops at the quality of the check; a check the AI can game gets gamed.

Work in specified pieces (requirements, outline or design, pieces, check; spec-driven development such as GitHub Spec Kit for software). Best at: the two-dimensional cut: say what the work must achieve, give it an outline or design, split that into pieces, do each in its own conversation with a human checkpoint. What to watch: case-study numbers only, no controlled trial.

One task per fresh context, repeated by a script (Ralph-style loops). Best at: a queue of small pieces, each in a new window, with a check as the only stop signal. What to watch: practitioner anecdotes; greenfield work.

Tool Search and tool examples (for connected tools). Best at: cutting tool-definition tokens by loading on demand; teaching a parameter format by example. What to watch: vendor claims (49 to 74 and 72 to 90 percent); beta features.

Contextual retrieval and reranking. Best at: lowering the retriever's miss rate before touching the model. What to watch: vendor eval; measure the miss rate on your own documents.

Citation and quote-checking. Best at: a quote per claim that a string match can verify; retract claims without one. What to watch: a quote proves provenance, not truth; a citation alone is not evidence.

Hallucination detectors (HHEM, LettuceDetect, MiniCheck, SelfCheckGPT). Best at: "is this claim supported by this text" at near-zero cost. What to watch: sample-consistency methods detect uncertainty, not truth; validate on your data.

Binary-question judges (BinEval-style). Best at: one yes-or-no question per failure type, with evidence. What to watch: more calls than a holistic score; still a judge with position and family bias.

Typed judges with probabilities. Best at: cheap calibrated labels for routing and adherence checks, escalating unsure cases. What to watch: no text, no explanation; you supply the labels.

Behaviour monitors on trajectories. Best at: catching reward hacking the tests cannot see. What to watch: a training-time result from one lab; the pattern list must evolve with the agent.

Verifiable-instruction benchmarks (IFEval-style) and pass^k (τ-bench-style). Best at: the two scores this article uses. What to watch: public versions measure models, not your tasks; build the local versions.

In every source, the lever that worked put something outside the model into the loop: a shorter window, a complete request, a quote, a check, a separate reader. The tools above are ways of doing that; none is the fix on its own.

A decision guide

The output is half done, or did more than asked. Use: Lever A; the piece was too big for one conversation. Then check: split it into what it must achieve, its outline, its pieces and its check, one conversation each, and re-run the piece that failed.

The output ignored a rule. Use: Lever B and C; find where the rule sat in the window and how much text sat around it. Then check: move the rule into the first message or into the saved instructions the app re-injects, shorten those instructions, re-run three times and read the rules-met list.

The output is right but not what I meant. Use: Lever A; the goal was settled late or trickled in over several turns. Then check: write the goal, the reader and the acceptance criteria in one message in a fresh conversation.

The output invents a fact, a number or a source. Use: Lever D; the fact was recalled, not read. Then check: supply the source or a lookup tool, require a quote per claim, score "the source does not say" as acceptable.

The output passed its own review and is wrong. Use: Lever E; add a check that runs without the model's opinion and a fresh-context reviewer with binary questions. Then check: pass^3, not pass@1.

The output degrades over a long conversation. Use: Lever A and B; shorten the horizon, checkpoint, start a new conversation. Then check: the extra turns a compaction causes.

A cheaper model fails rules the expensive one kept. Use: Lever F; the floor was crossed. Then check: cost per verified success with a strong critic, or go back up a tier for the writer.

A model upgrade changed nothing. Use: Levers A to E; the failure was never a capability failure. Then check: the four-condition protocol on your task.

Monday checklist, one line per lever, in order:

The task no bigger than one conversation, and cut into pieces with a check each when it is; then the whole piece in the first message, goal and reader first, acceptance criteria written.

Fresh conversation, bare of what this task does not need; saved instructions, project files and connected tools reviewed.

Rules by hand, short, mostly "don't"; no add-ons loaded just in case.

Sources supplied; quotes required; retrieval miss rate known.

A check that runs without the model's opinion inside the task; a fresh reader with ten questions after it.

Model strong enough to follow rules; effort chosen by a measured run; then stop.

Fifteen points that survive the details

A task too complex for one conversation is not fixed by a better prompt; split it into many small, self-contained tasks, each with its whole brief and its own check in its own conversation, and run twenty conversations instead of one.

The model reads the whole window, not your message alone; quality and adherence are properties of the whole.

Score correctness and adherence separately, and report pass^k for anything you hand over unchecked.

One task drip-fed over the turns of a conversation loses 39 percent on average, from the second turn on; write it all once. That is the opposite of splitting, not an argument against it.

Settle the goal in the first tenth of the run; a late correction is worth about as much as none.

Spec detail raises success until the spec itself becomes noise; one more round of rules can drop it.

Long horizons erode and "plan first" moves the start, not the slope; the cut into pieces is the fix, not the plan.

Extra agents help only for parallel reading; sequential hand-offs lose; keep one writer.

Length alone hurts on every model tested, even with perfect retrieval; the window is a budget.

Content beats size; a window of bad history teaches failure; compact by facts and re-inject rules.

Instructions are tokens; short, hand-written, mostly prohibitions; model-written context files cost more than they return.

Examples for tools, prohibitions for style, add-ons on demand and measured.

Supplied text can be checked and recall cannot; quote per claim, retract without quote, abstain without source.

Self-review misses correctness; a check that runs on its own, a fresh reader and a trajectory monitor catch what it cannot, and any check the AI is rewarded on gets gamed.

The model gates the other levers and then flattens; upgrades buy accuracy six times faster than reliability; decide trade-offs by cost per verified success.

References

Papers and reports

Ao, Gao, Simchi-Levi (2026). On the Reliability Limits of LLM-Based Multi-Agent Planning. arXiv:2603.26993.

Bai et al. (2026). How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks. arXiv:2604.22750.

Bodhwani, Tran, Wei (2026). GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents. arXiv:2609.12191.

Brown et al. (2024). Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv:2407.21787.

Chacón Sartori (2026). The Specification Gap: Coordination Failure Under Partial Knowledge in Code Agents. arXiv:2603.24284.

Chen (2026). When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models. arXiv:2606.27288.

Cho et al. (2026). Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement (BinEval). arXiv:2606.27226.

Dong et al. (2026). Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents. arXiv:2608.11888.

Du et al. (2025). Context Length Alone Hurts LLM Performance Despite Perfect Retrieval. arXiv:2510.05381.

Gloaguen et al. (2026). Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? arXiv:2602.11988.

Greshake et al. (2023). Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173.

Gulati et al. (2026). Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents? arXiv:2605.07937.

Han et al. (2025). RAG vs. GraphRAG: A Systematic Evaluation and Key Insights. arXiv:2502.11371.

Hong, Troynikov, Huber (Chroma, 2025). Context Rot: How Increasing Input Tokens Impacts LLM Performance.https://research.trychroma.com/context-rot

Kalai, Nachum, Vempala, Zhang (2025). Why Language Models Hallucinate. arXiv:2509.04664.

Kim et al. (2025). Towards a Science of Scaling Agent Systems. arXiv:2512.08296.

Laban, Hayashi, Zhou, Neville (2025). LLMs Get Lost in Multi-Turn Conversation. arXiv:2505.06120.

Li et al. (2026). MARCH: Multi-Agent Reinforced Self-Check for LLM Hallucination. arXiv:2603.24579.

Li et al. (2026). OCR-Memory. arXiv:2604.26622.

Li et al. (2026). PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agents. arXiv:2609.06702.

Liang et al. (2025). GraphRAG under Fire (GRAGPOISON). arXiv:2501.14050.

Lin et al. (2026). Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents. arXiv:2605.30621.

Liu (2026). What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics. arXiv:2608.16370.

Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172.

Liu et al. (2023). Evaluating Verifiability in Generative Search Engines. arXiv:2304.09848.

Liu et al. (2026). The Memory Curse: How Expanded Recall Erodes Cooperative Intent in LLM Agents. arXiv:2605.08060.

Madaan et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651.

Manakul, Liusie, Gales (2023). SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. arXiv:2303.08896.

Miller (Anthropic, 2024). Adding Error Bars to Evals. arXiv:2411.00640.

Orlanski et al. (2026). SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks. arXiv:2603.24755.

Pan et al. (2026). The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks. arXiv:2609.25804.

Parikh (2026). Structured Output Collapses Answer Diversity Across 44 Language Models. arXiv:2607.18476.

Rabanser, Kapoor, Kirgis, Liu, Utpala, Narayanan (2026). Towards a Science of AI Agent Reliability. arXiv:2602.16666.

Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366.

Wallace et al. (2024). The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv:2404.13208.

Wang et al. (Qwen Team, 2026). The Verification Horizon: No Silver Bullet for Coding Agent Rewards. arXiv:2606.26300.

Yang, Ding (2026). Signal or Noise? A Benchmark Study of Agent Skills in Web Development. arXiv:2608.23067.

Yao et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629.

Yao, Shinn, Razavi, Narasimhan (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045.

Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.

Zhou et al. (2023). Instruction-Following Evaluation for Large Language Models (IFEval). arXiv:2311.07911.

Zhou et al. (2026). Agent-as-a-Router: Agentic Model Routing for Coding Tasks. arXiv:2606.22902.

Vendor and practitioner sources (claims and guidance)

Anthropic (2024). Introducing Contextual Retrieval.https://www.anthropic.com/news/contextual-retrieval

Anthropic (2025). How we built our multi-agent research system.https://www.anthropic.com/engineering/multi-agent-research-system

Anthropic (2025). Introducing advanced tool use on the Claude Developer Platform.https://www.anthropic.com/engineering/advanced-tool-use

Anthropic (2025). Effective context engineering for AI agents.https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

Anthropic (2025). Effective harnesses for long-running agents.https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents

Anthropic (2026). Demystifying Evals for AI Agents.https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

Anthropic (2026). Explore the context window (Claude Code docs). https://code.claude.com/docs/en/context-window

Anthropic (2026). How Claude remembers your project (Claude Code docs). https://code.claude.com/docs/en/memory

Anthropic (2026). Run Claude Code programmatically.https://code.claude.com/docs/en/headless

Anthropic (2026). Effort and Claude Model Migration Guide (Claude Platform docs). https://platform.claude.com/docs/en/build-with-claude/effort and https://platform.claude.com/docs/en/about-claude/models/migration-guide

Anthropic (2026). Reduce hallucinations (Claude Platform docs). https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations

Anthropic (2026). System Card: Claude Sonnet 5.

GitHub (2025). Spec-driven development with AI: get started with a new open source toolkit.https://github.blog/ai-and-ml/generative-ai/spec-driven-development-with-ai-get-started-with-a-new-open-source-toolkit/

Huntley (2025). Ralph Wiggum as a "software engineer".https://ghuntley.com/ralph/

Martin (Anthropic, 2026). Designing loops with Fable 5 (post on X). https://x.com/RLanceMartin/status/2064397389189071163

Neo4j (2026). Study: GraphRAG makes AI agents 80 percent more truthful (vendor blog).

Trivedy (LangChain, 2026). The Anatomy of an Agent Harness.https://www.langchain.com/blog/the-anatomy-of-an-agent-harness

Yan (Cognition, 2026). Multi-Agents: What's Actually Working.https://cognition.com/blog/multi-agents-working

How to make an AI agent do what you asked: the levers that decide output quality and adherence

The quick grasp

The level tree

Level 0: one task, run four ways

Level 1: what the model actually sees

Level 2: how to measure quality and adherence before changing anything

Level 3. Lever A: task complexity, how big each piece is, what you ask, over how many turns

Level 4. Lever B: what the model sees, size, relevance, position and content of the context

Level 5. Lever C: how the instruction is written, specific, scoped, not over-specified

Level 6. Lever D: where the facts come from, supplied text and retrieval, not recall

Level 7. Lever E: how the work is checked, an external, runnable check, by a fresh reader

Level 8. Lever F: the model and its settings, the weakest lever and the first one people pull

Level 9: trade-offs

The tools and approaches, observed

A decision guide

Fifteen points that survive the details

References

Actions

What You Can Do

The quick grasp

The level tree

Level 0: one task, run four ways

Level 1: what the model actually sees

Level 2: how to measure quality and adherence before changing anything

Level 3. Lever A: task complexity, how big each piece is, what you ask, over how many turns

Level 4. Lever B: what the model sees, size, relevance, position and content of the context

Level 5. Lever C: how the instruction is written, specific, scoped, not over-specified

Level 6. Lever D: where the facts come from, supplied text and retrieval, not recall

Level 7. Lever E: how the work is checked, an external, runnable check, by a fresh reader

Level 8. Lever F: the model and its settings, the weakest lever and the first one people pull

Level 9: trade-offs

The tools and approaches, observed

A decision guide

Fifteen points that survive the details

References

Actions

What You Can Do

More in AI & Machine Learning

More from @sermakarevich