Build your own harness in 17 steps
This article gathers the concepts from the first wave of the build-your-own-harness tutorials into one story. Each tutorial added one job. This text keeps the jobs and drops the build steps.
The rule behind the whole series: an agent is a model plus a harness. The model is a large language model (LLM for short), a program that reads text and writes the next piece of text. The harness is everything around it: the loop that calls the model, the tools the model can use, the context the model sees, and the rules it must follow.
The harness decides most of the result. One team moved from Top 30 to Top 5 on a terminal benchmark by changing only the harness around the same model. Another study measured a 6x swing from the harness alone. The working rule is: put rules in code, not wishes in prompts. The cause is plain. The model sees only what the harness puts in its context. It acts only through what the harness runs for it.
Tutorial repo: https://github.com/sermakarevich/harness
## 1. A model call is plain HTTP, and it forgets everything
A model call is one request and one reply. You send a model name and a prompt. You get back a list of typed blocks. In our setup the reply holds reasoning blocks first, then a message block with text parts. The harness decides which blocks you see. Ours shows the text and keeps the reasoning inside.
Your secret key travels in a header. The model never sees it. A session id travels in another header. It routes the call. It does not store anything.
The request is stateless. Nothing is kept between calls. The tutorials prove it with two prompts. The first says: remember the word pelican. The model answers that it will remember. The second asks for the word. The second request carries only the second question. The model answers that no word was given. Agreement is not memory. Every call replays only what you send in that call. Treat the model as a stateless compute unit. Keep all state outside it, in the harness.
## 2. The message list is the memory
Tutorial 2 fixes the forgetting by keeping a list and resending the whole list every turn. The first question and its answer stay in the list. The second question travels with them. The model answers pelican. The list is the memory.
This list grows without bound. Each turn adds messages. Later chapters put a price on that growth and then cut it back.
The first entry is the system prompt. It is the standing order. It sits first and shapes every answer. It is the highest-leverage point in the harness. Ours is a text file, not code. It is filled with today's date and the working directory. That keeps the order readable and changeable without touching code.
One adapter hides the vendor. It is the single seam where a model object is built. Nothing above it sees a header or an address. The model stays swappable because only one place knows vendor details. Type `/new` and the harness keeps only the system message. The next question travels alone. The model forgets again, on purpose this time.
## 3. The turn becomes a graph with a keeper and a stream
Tutorial 3 moves the turn into a graph. A graph here means a small workflow library (we use LangGraph) where one node does one step and edges say what runs next. One node prefixes the system prompt and answers. That leaves room for a tool node later. The library owns the repetition. Hand-written loop code goes away.
The checkpointer replaces the variable that held the list. You name a thread and it loads the list stored under that id. The id is the session name. Every real harness keeps this list outside the model: one in a local database, one in line-delimited files, one in an append-only log. Ours saves after every turn. Nothing survives unless written.
Tokens arrive one by one, so you print them as they arrive. That is streaming. A failed call prints one line and keeps the chat. The terminal has three commands: `/new` for a fresh thread, `/help` for the list, `/exit` to leave. A restart still loses the list at this stage. Persistence fixes that later.
## 4. How the model sees tools
This is the idea worth slowing down for.
A tool is two things. The schema travels to the model with every call. The function stays in your harness and runs only when asked. The schema holds the name, the description, and the argument shape. The function body never leaves your machine.
When we ask our code what the `read_file` tool looks like to the model, the answer is plain data:
```json
{
"type": "function",
"function": {
"name": "read_file",
"description": "Read a text file under the working directory. Paths are relative to it.",
"parameters": {
"properties": {
"path": { "type": "string" }
},
"required": ["path"],
"type": "object"
}
}
}The model never executes anything. It returns a message naming a tool and its arguments. The harness decides whether and how to run it. Then it returns the result as a tool message tied to the call by id. The model reads that message and answers with text.
The loop is graph structure. The model node routes to the tool node on a tool call and to the end otherwise. The tool node always routes back. Reason, act, observe, repeat. Real loops carry a step budget so a confused model cannot loop forever.
Three details matter in practice. Schemas cost context every turn even when unused, so you add tools with care. Tool calls arrive as streamed pieces, name first and arguments after, and the terminal adds the pieces up before it prints one dim line per call. Tool errors are text back to the model, not exceptions. A wrong path costs one more turn instead of the chat.
Our first tool only reads. Reading has no side effect so it needs no gate. Only tools that change things need asking. The file system won as the agent workspace because models were trained on it.
## 5. Files and the shell: the model sees no process, only text
Tutorial 5 adds write, edit, and shell. The graph does not change. Only the tool list grows.
Edit is exact find-and-replace. The model sends exact old and new text. The harness replaces only on exactly one match. Anything else returns error text the model can use. Models quote text they just read better than they count lines, so text beats line numbers. A failed edit must explain itself so the model can recover.
Shell runs one command with a timeout in the working directory. A command that never ends would hang the chat, so the harness kills it after a fixed number of seconds and says so. The timeout is a setting, not a literal, because a test suite that needs minutes is normal.
A process becomes text. The model never sees your process. It sees output, error output, and the exit code as one message. That is why the exit code comes back every time. A silent failure would look like success. The shell is the general-purpose tool because it can do anything text can describe.
Every path goes through a shared resolver that rejects escapes. The shell inherits the environment so names like `pytest` resolve. Big outputs are capped in a later chapter. The demo tells the story: the model writes a file with a deliberate bug, runs the test, reads `1 failed`, fixes one line, runs the test again, reads `1 passed`.
## 6. The permission gate: no side effect without a yes
Reading is safe. Writing and running are not. Tutorial 6 wraps every side-effect tool in a human yes-or-no gate.
The policy names which tools need no question. Only `read_file` passes at first. A tool nobody listed as safe needs a question. Forgetting to classify a new tool costs one question instead of a deleted file.
The tool node checks before it runs, then freezes the turn. Your answer resumes it. A resumed node starts again from its first line, so nothing may run before the pause. Otherwise the tool would run twice.
The terminal asks with yes, always, and no. Always covers that tool for the session. A no is text the model reads, not a crash. The refused call still gets a tool message saying it was denied. The model answers from there. The loop never breaks on a refused call.
## 7. Persistence: the thread id becomes the session
Memory lived in the process until tutorial 7. Closing the terminal lost the conversation and the permission answers with it. Checkpoints move from memory to disk. The change is small because the graph never knew where its state was kept.
The store is a SQLite file (a small database in one file) opened once for the whole run. The thread id becomes the session name. The banner prints it. `/resume` takes it. You list recent conversations newest first, numbered by the first thing you said, and continue one by id. `/new` keeps its meaning and no longer destroys anything. The old conversation stays on disk.
Each step saves, so a mid-turn kill still resumes. A waiting permission question is state too, so it survives a restart. Answering it finishes the old tool call.
## 8. Cost: every reply carries its own bill
You cannot count tokens yourself. Only the server knows how its tokenizer split your text. Every reply carries a usage record. The harness reads the bill from saved messages. It keeps no counter, so nothing can drift and a resumed conversation brings its own bill back.
A bill is three numbers against three rates. Full-rate input, cached input (text the server has already seen), and output. On our model output costs about twice input, and cached input costs about one fiftieth of fresh input. Thinking tokens you never see are billed as output. Most of what you pay for is thinking you never read.
Every turn pays for the whole conversation again, because every turn resends the whole list. Caching keeps this survivable. One study put the same task at about 1.1 million tokens under one harness and 2.2 million under another. The harness decides most of the bill. Raw spend explained about 1 percent of success. Paying more is not doing better.
The terminal prints one dim line after each turn with this turn and the total so far. The rates live in settings because a provider changes prices without asking you.
## 9. Big tool outputs stay out of the conversation
One tool call can flood the context. A command prints what it wants. Every character stays in the conversation. Then you pay for it on every later turn.
The fix: save it, then point at it. Over a character limit the whole result lands in a file. The model keeps the first part plus the path and the full size. Nothing is lost. The model reads the file on demand with the read tool.
The cap sits where every result passes, inside the tools node, so a tool written next month gets it for free. The spill file is named after its text, not its call, so the same output never lands twice and nothing the model chose ends up in a file name. The exit code moves to the front of a shell result so the cap can never cut it.
Caching buys money back, never space. A cached second turn costs almost nothing yet still fills the window. A cap is a separate job from a price. The demo shows it: `seq 1 2000` prints 8893 characters. Without the cap the next unrelated question resends thousands of tokens. With the cap the turn shrinks by more than half and the full text sits in the spill folder.
## 10. Compaction: summarise the old, keep the recent
Tutorial 9 bounds one result. Tutorial 10 bounds the whole list. The list only grows and the window has an end.
Over budget, older turns become one short recap while the newest stay untouched. The cut happens only at a turn start, never in the middle of a tool round. That shape guarantee is why the check runs at the entry to a turn. The size measure is the input count the model itself reported on its last reply. It is exact, free, and always agrees with the bill.
What survives matters more than how much you cut. Keep names, paths, numbers, and decisions, so the model never re-asks for them. A fact-preserving summary stays near-lossless. Plain deletion triples re-querying. Compression that throws detail away raises retrieval calls while completion never moves. Checking only that the agent finished hides the bill it paid re-asking.
The summary and the dropped turns stay inside the running total. The summary call costs its reply and the dropped cost is carried beside the messages. A total that forgets what you spent is worse than no total. In the demo three big shell runs push the conversation past the budget. The compacting turn pays for the summary once. The next question reads about 3000 input tokens instead of 5000.
## 11. Memory files: standing rules outside the messages
A fresh session knows nothing about your project. Model weights never change. Each talk starts from the same beginner. Rules live in plain files pasted into the prompt before your first word.
A project note is shared with everyone through the checked-in files. A home note is yours alone. The harness walks up from the working directory, takes the first matching name in each folder, and pastes farthest first so the nearest is read last and wins. Ours looks for `AGENTS.md` then `CLAUDE.md`, prints what it loaded, and carries it on every call of that session.
Notes are pasted, not learned, and you pay per call. The demo asks without tools what to name a helper file. Without notes the model answers `utils.py`, the one name the project forbids. With notes it follows the project rule. The price shows in the token line: about twice the input tokens on every call. Notes load once per session so the cached prefix stays stable. An edited note takes effect on `/new`. After compaction this matters twice as much: the summary can drop a rule, but notes outside the messages survive it. Keep notes short. Let skills carry the rest.
## 12. Skills: one line now, the rest on demand
Some instructions are needed sometimes, not always. A note charges you every turn. A skill keeps one line up front and loads the rest only when the model asks. Name and description live in the prompt. The body arrives through one tool call.
A skill keeps the model on the rails. It rarely teaches a fact. Across hundreds of matched runs, skills beat a memory note, and judges credited procedure far more than knowledge. Our house commit format shows it: without the skill the model writes `Add skills loader`. With the skill loaded on demand it writes `docs: add the skills loader`, scope, colon, lower-case summary. The scope itself can still be wrong. A skill shapes an answer more than it supplies a fact.
More skills is not more capability. As the pool grew from 5 to 100, the right-skill share fell from about 30 percent to about 3 percent while success stayed flat. Hundreds of confirmed cases showed skill-caused damage: wrong answers or passing runs at double the cost. A longer list makes the model's aim worse. Index in the prompt, body behind a tool, and keep the list short. A turn that needs no skill pays only for the index.
## 13. Reliability and parallel tools: sort, retry, batch
Until tutorial 13 one failed call ended the turn and every tool call ran one after another. Real calls fail. A rate limit, an overloaded server, a dropped connection, and a refused key all arrive as one exception. Only some pass on retry.
Sort the error first, wait second, and let only one layer do it. A wrong key retried three times is three waits and the same error. Ours retries server and connection errors up to three tries with a growing wait and a small random spread, and turns the retrying hidden inside the vendor client off. One policy you chose beats two policies fighting.
Independent calls run together. Calls that write stay alone. All four reference harnesses run siblings at once and return results in the model's order. Permission questions are asked first, in order, because a question cannot be asked from inside a thread. The win is bounded by the serial share: three five-second sleeps take 23 seconds in sequence and about 11 overlapped. Fanning out to 100 calls can collapse tool use to zero, so batch with care. A tool call may already have changed a file, so the tools node has no retry of its own.
## 14. Planning: the list lives in state, not in prose
Ask a model to plan and it plans in prose, in the past tense, once the work is over. A plan that lives only in the reply is not a plan. Nothing outside the words holds it. The summary can drop it. Long-running agents stop early and overestimate how much is done.
Our planning is one short list the harness holds. A few steps, each waiting, in progress, or done. The model decides the steps and rewrites them as it works. The harness keeps the list, prints it, and puts it back under the system prompt on every call. Because the list is state and not a message, the summary node never sees it.
One writer, and the whole list every time. The model replaces the list whole on each update. That costs a few more tokens than patching one item and removes every way for an index to drift. The harness keeps one rule: at most one item in progress. Extras are demoted. The terminal prints the list instead of a tool arrow, so you watch the shape of the job while it runs: empty boxes, a half-filled doing marker, checked done boxes.
A short list steers a model that has forgotten everything else. A cheap model handed an exploration trail plus a todo list reached most of a frontier model's score at much lower cost. The price is more model calls: each list update ends one call and starts another. The same small job costs about twice the tokens once it carries a visible plan.
## 15. Sub-agents buy context, not cleverness
A sub-agent hands one job over and gets one answer back. Ours is the same agent started again: the same graph, its own empty message list, its own turns and tool calls. The parent's single tool call is the whole life of the child. The call starts it, it runs until it has an answer, its last message becomes the tool result, and then it is gone. That is why we call the tool a helper.
A helper buys context, not cleverness. With equal thinking budget a single agent matched or beat every multi-agent arrangement on multi-hop reasoning. The multi-agent versions pulled ahead only when the single agent's context was damaged on purpose. A helper is worth it when context is the scarce thing, not when the thinking is hard. It also knows only what the job says. When agents each held part of a specification, integration success fell by 30 to 35 points. Write the job text well. It is all the helper knows.
One rule gives the whole policy. The tools that need no permission question are exactly the tools a helper may use. That keeps a helper read-only. Asking for a helper needs a question, so a helper cannot ask for one. The depth limit needs no counter. Two traps remain in the code: the helper's own model calls are counted nowhere, so the cost line under a delegated turn reads lower than the turn really cost, and delegation never makes the parent turn cheaper. What shrinks is what the conversation carries afterwards. Two files the helper read never enter your history. The follow-up turn reads hundreds fewer fresh tokens.
## 16. Outside tool servers: borrowed tools, same loop
Tutorial 15 shared jobs but every tool was written here. Tutorial 16 runs tools nobody here wrote. An outside tool server is a second program that offers tools. The harness starts it, talks to it through the pipes of a child process, and asks one question first: what tools do you have? Each answer becomes a tool in the same list the loop already runs. The Model Context Protocol (MCP for short) is the agreement both sides follow, so no custom code per server is needed.
You name the servers in the settings file. The harness connects at start-up, renames each tool after its server so two servers never collide, and wraps each one so the existing loop runs it with no change. A borrowed tool is a normal tool by the time the loop sees it. Everything interesting happens in the renaming and wrapping.
Three costs come with it. You pay for a borrowed tool on every turn whether you use it or not, because its description travels with every model call. In one measured setup tool definitions alone took tens of thousands of tokens per turn. A borrowed tool is someone else's code holding your permissions, so every borrowed tool stays off the no-question list and the harness asks before each call. Each server gets its own connection and its own try, so a dead server costs only its own tools. The demo asks the time in Tokyo. Before, the model reached for the shell and you judged a command line. After, it calls one narrow tool by name and you approve that tool.
## 17. Evaluation: one run proves nothing
The last ability is measurement. You write tasks, repeat them, and print one table. A few small tasks with expected answers, each run several times unwatched, one row per task with pass count, tokens, money, and seconds.
One run proves nothing. Repeats of one task spread several points, as much as the effect you hoped to measure. The table needs money and time columns next to pass and fail because the same tasks through two harnesses moved cost by about 40 percent and wall time by about 40 percent while quality stayed flat. Keeping quality while halving the bill is a win, and only the table shows it.
Two traps shape the runner. Nobody answers the permission question, so the runner says yes to everything, safe only because each task runs in a throw-away folder. A shared session would leak the first answer into the next task, so each run starts fresh with its conversation in memory only. Scoring is exact match, so only tasks with one right answer qualify. Free-text judging waits. The command exits non-zero when any run failed, so a machine can run it as a gate.
## The thread through it all
Every chapter patches one model limit with harness code. The model forgets, so you keep the list. The list grows, so you price it, spill big results, and summarise the old turns. Standing rules must survive all of that, so they live in notes outside the messages and in skills loaded on demand. Calls fail and work has shape, so you sort errors, batch independent calls, and hold a visible plan in state. Context is scarce, so you delegate to a helper that leaves your history clean and borrow narrow tools instead of reaching for the shell. Claims need proof, so you run a fixed suite more than once and read pass next to money and time.
Build in that order and each step stays small. Skip one and its absence shows up as a bill, a lost session, a deleted file, or a plan that existed only in prose.