A short article for people who decide on AI projects. It walks through the layers every AI agent is made of, from the model itself to fully autonomous systems, one short layer at a time. No technical background is needed. Every number comes from a published paper or a vendor document listed at the end; vendor numbers are marked as claims.
This article is a picture of what an AI agent is made of, for people who decide on AI projects. Each layer below is a short summary on its own. Read the first three and you know what an "AI agent" is. Read all thirteen and you can hold your own in any planning meeting about one.
Layer 1: core primitives
Question: what is an AI system actually made of?
Three things, and only three: a model, a context, and tools. Everything else in this article is built from them.
The model. The model is a kind of neural network. It was trained on a very large share of the text publicly available on the internet, with one goal: predict the next word. That training is where all of its general intelligence comes from. We send it text and it produces a reply by picking one word at a time. It was trained on public text only. It has never seen your policies, your prices or your customers.
The model is stateless. It does not remember who you are or what you asked a minute ago. It does not learn from your requests, and its intelligence does not change as you use it. Each request you send may even be served by a different copy of the model in a different data centre. If you say in your first message that your name is Bob and the model still calls you Bob in the fifth, that is not memory. The software sent the first message again, inside the fifth request. Drop it from the request and the model has never heard of Bob.
The context. Everything we send to the model, as one long document the model reads from the top, is called the context. It is much more than your message. Before the first message is even typed, the software has already put in:
Then the conversation history is added as it grows: your messages, the model's replies, and the output of every tool the model asked for. When the model reads a file or lists a folder, the result becomes one more entry in the context.
The context is measured and billed in tokens, pieces of words; a token is about three quarters of an English word. Everything in the list above counts: the system prompt, the tool definitions, every tool result, and the model's own hidden reasoning when it is switched on.
The tools. Tools are helpers the model can ask to have run: a calculator, a database lookup, a web search, a request for a human to step in. The model cannot run anything itself. All it can do is produce the next word. So tool use works like this. The calculator's description sits in the context. When the model decides it needs two plus three, it writes a reply in a fixed format, called a tool call, that says which tool to run and with which input. Our software reads that reply, runs the calculator on our side, and sends the result back as the next entry in the context. The tool runs on our computer, never inside the model.
So, in one sentence: the model is a neural network that writes a reply based on what it learned in training and on the context we are responsible for assembling, and it can ask for tools that we run on our side and whose output we send back.
Important: the model has no memory and cannot act. Everything it "remembers" is text sent to it again, and everything it "does" is a request our software chose to carry out.
What it means for us: the model is the one part we do not control. The context and the tools are the parts we do.
Why it matters: every improvement in the layers below works by changing the context or the tools. None of them changes the model.
Layer 2: what you can change to get better output
Question: for each of the three parts, what can we change, and how hard is it?
The context. Five things to manage. The first three are the ones that fail most often; the last two are the ones that fail most expensively.
Hygiene means paying attention to what sits in the context. Tools that are never used, skills that do not apply, instructions about another topic: all of it is read on every request, and a messy context confuses the model. The basic recommendation is to start a new session for every new question and to limit the number of skills and plugins installed.
Proprietary information means remembering that the model was trained on public text only. It knows nothing about your business. If the assistant should know the 15-day return window, somebody has to put that sentence into the context, one way or another. Layer 5 is about the ways.
Size means that the model performs worse as the context grows, and costs more. There is no clean limit after which it breaks; the quality slides down as the context fills, long before the advertised maximum is reached. Anthropic's guidance treats the context as a scarce budget to be spent on what the task needs, not as free space to fill (Anthropic, 2025, context engineering).
Position means that where a fact sits in the context changes whether it is used. The model reads the beginning and the end of a long context better than the middle, so a policy pasted between two long documents may be ignored even though it is there.
Trust means that everything in the context is read as if it came from you. A web page, an email or a document fetched by a tool enters the same context as your instructions, and text hidden in it can be obeyed. The industry name is prompt injection.
Cost follows the same curve. Every request re-sends the whole context, so a conversation that has grown to 100 pages costs 100 pages per message, not per conversation. Short contexts are both more accurate and cheaper.
One more thing follows from "every request re-sends the whole context": the whole context leaves your company on every request. Whatever is in it, policy, customer record, credentials passed to a tool, is sent in full to the vendor. Deciding what may enter the context is a data decision, not only a quality one.
The tools: hygiene and capabilities. Do not install tools that will not be used. Better still, connect only the tools a session needs, so the model is not misled about what is expected of it.
The other side of tools is capability. The model can only produce words; tools are how it gets things done the right way. Tools are how it does arithmetic, counts the letters in a word, fetches a document, reads your internal data, or asks a human for help. A capability the assistant lacks cannot be prompted into existence. If the assistant cannot look up orders, no instruction will make it know where an order is.
The model: fine-tuning. To make the model itself know your business, you can fine-tune it: continue its training on your own examples. This is the hardest of the three to change. You need a dataset, computing power, and then your own deployment of the resulting model, and not every model allows it. If you can afford it, you get a model whose default knowledge differs from your competitors'.
Before fine-tuning anything, there is the cheaper model lever: which model to buy. Judge it by measured cost per task, not by list price per token, because models that reason longer consume far more tokens for the same question.
Important: the model is the hardest part to change. The context and the tools can be changed in a day, and the studies above show that changing them moves accuracy by tens of points.
What it means for us: before any budget for a "better model", audit what the current one is being sent.
Why it matters: fine-tuning needs a dataset, computing and a deployment. Trimming a context needs none of them.
Layer 3: what an agent is
Question: people say "agent" as if it were a new kind of AI. Is it?
No. By now we know the model can only produce a text reply given a context, and that it can ask for tools which we must run. An agent is the piece of software that does the running. It is written as a loop, or a graph, or a workflow, and its job is:
Take a support assistant asked "where is my refund?". The model asks for an order lookup; the software runs it and sends the context again. The model asks for a search of the returns policy; the software runs it and sends the context again. On the third reading the model has the order status and the policy text, and writes the reply. The model never touched the database or the policy. It wrote two requests; the software decided to act on them.
The ability to run this loop and call tools to collect what the answer needs is what makes a model an agent, in its simplest form.
The loop also has a cost that a fixed program does not: it is unpredictable. The same task can take two tool calls or twenty, and the result can differ between runs. So the first question is not "which agent" but "do we need one".
Layer 4: hallucination
Question: when the model is wrong, how would we know?
Models try to produce the best reply they can, but they make mistakes, and the mistakes are fluent. There are two kinds, and they differ in whether you can check them.
One limit of grounding: a quote proves the answer came from the document, not that the document is right. A stale page, or one that somebody planted, is quoted just as faithfully.
Important: a model that must quote its source can be audited. A model that answers from memory cannot.
What it means for us: design the assistant so every factual claim comes from a document in the context, and show the document.
Why it matters: it turns "is the AI right?" into "is this line in our policy?", which anyone can answer.
Layer 5: knowledge bases
Question: if the context must stay short, how does the right page get into it?
Managing the context is the most important thing we can do to improve an agent and reduce hallucination. A knowledge base is how the information the model needs is exposed to it: internal instructions, policies, product descriptions. Say you have a policy to process returns within 15 days if the product was unpacked. The model will never know that from training. The rule is useless until it is in the context at the moment the question is asked.
If it is small, paste it. When there are few policies and they are short, put them all in from the start of the conversation.
A short policy document fits. A product catalogue, years of support tickets and supplier contracts do not. For collections past that size, the community's approach is to split the text into smaller pieces and find a way for the model to retrieve the right piece.
Way 1: a wiki the model walks. Proposed by Andrej Karpathy under the name LLM wiki. Keep the original documents untouched and have the model write and maintain a second layer: a page per topic, a short summary per page, and an index of summaries at the top. Higher levels summarise the pages below them. To answer, the model starts at the index, reads the summaries, and goes down the tree to the page or pages it needs (Karpathy, 2026). I organise my own knowledge base this way, and I use it instead of skills, rules or instruction files. Everything the model should know lives in one wiki, which keeps the setup generic and the same for every provider: Claude, OpenAI, Google, open-source tools.
Way 2: search by meaning. This is what the industry calls RAG, retrieval-augmented generation. It needs a second kind of model, a sentence or embedding model. Where the language model takes text and produces a reply, the sentence model takes text and produces a list of numbers, a vector. Two pieces of text with similar meaning get similar vectors. That lets a database find text by meaning rather than by exact words. The practical trick: for each chunk of your documents, ask the model what question that chunk answers, turn the question into a vector, and store it as the key. When the assistant has a question, turn it into a vector with the same sentence model and fetch the chunk whose stored question is closest. A customer's "where is my refund" lands on the refund-timing section even though the policy never uses the word "where".
Way 3: a map of entities. A graph RAG stores entities and their relationships instead of chunks: the product, the supplier, the warranty, the policy, and the links between them. Questions that need several facts joined ("which supplier covers the warranty on the product in this order?") walk the links.
Whichever way: somebody owns it. A knowledge base goes out of date silently. A policy that changed and was not updated is served as true, with a quote. Each part of it needs a named owner, a way for corrections to flow back, and a date on every answer that says how fresh the source is.
Important: the model does not know your business and never will from training. It knows exactly what is in the context, and a knowledge base is the machinery that puts the right page there.
What it means for us: the quality of the assistant is bounded by the quality and organisation of the documents behind it, not by the model.
Why it matters: cleaning and structuring your documents is a task your team already knows how to do.
Layer 6: memory
Question: if the model forgets everything between requests, how does an assistant "remember" a customer?
It does not learn. At the end of a conversation the software writes a short note to disk: "this customer disputed a fee, resolved by waiving it". At the start of the next conversation it puts that note into the context. That is all memory is: a way to write into the context before the first message. Everything from Layer 2 applies to it. Notes make the context longer, and a stale or wrong note is read on every later session as if it were true.
Remembering that a customer already received a goodwill waiver is useful. Letting the assistant write its own notes without review is how a one-off waiver becomes a policy.
Important: memory is not learning. It is a note the software writes today and puts into the context tomorrow.
What it means for us: decide who may write memory and what may be written before deciding what to remember.
Why it matters: a write path into the context is a write path into every future answer.
Layer 7: harness
Question: if the model is fixed, where does the rest of the quality come from?
The harness is everything that is not the model. The loop from Layer 3 is part of it. So is the code that builds the context, the permission check before a tool runs, the routine that trims the context when it gets long, the memory notes from Layer 6, and the logging that lets you see what happened. Different companies wrap the same model in very different harnesses and get very different results.
One harness part deserves its own paragraph: permissions. Every tool request passes through the harness before it runs. That is where "the assistant may look up orders but may not issue refunds above 200 euros without a human" is enforced. The model never sees the rule as a wall; it sees a tool request that was declined. A hand-off-to-human tool is the same rule made visible: anything the assistant may not decide goes to a person.
Asking a person to approve each request does not scale, because people stop reading the prompts. The stronger control is to fence what the agent can reach at all, with a sandbox, and keep the approvals for what is left.
The last harness part is the log. When an agent fails, the cause may be the model, a tool, or the harness itself, and only a step-by-step record of the run lets you tell which.
Important: the same model gave 6.7% and 68.3% depending on the harness. The harness is where the difference was made.
What it means for us: "we use the same model as the market leader" says nothing about the results. Ask about the harness.
Why it matters: harness choices are yours to make: what the agent may reach, what it remembers, what it must hand off, and what gets logged.
Layer 8: subagents
Question: what if one task is bigger than one short context?
Layer 2 said the context should stay short. Some tasks do not fit: "read all 3,000 tickets from last quarter and list the five most common complaints" cannot be done in one context of 100,000 tokens. The answer is not a longer context. It is more contexts. The harness starts a second, fresh context with a narrow brief ("read tickets 1 to 300, return the complaint categories you see"), lets it work, and takes back only a short summary. It does that ten times and gives the ten summaries to the main context. Each helper is called a subagent. Each stays short and focused, and the main context only ever sees the summaries.
The cost is real and the control problem is new: a helper that returns a wrong summary cannot be caught by the main context, which never saw the source. The eval in Layer 10 is the check. Two more limits show up as soon as helpers are used for work rather than reading.
Layer 9: loops
Question: what if the work takes longer than one conversation?
A single session degrades as its context fills. The simplest pattern for long work is to not have long sessions. Start a brand-new session, give it one task from a list, let it work, check the result, and start another. Progress lives in files on disk, not in the context. The community name is the Ralph loop.
Labelling a large ticket archive works this way: one run per hundred tickets, each from a clean start, with a counter and a stop condition.
Layer 10: evals
Question: we changed the context, the tools, or the model. Did it help?
You cannot tell by chatting with it. The industry's answer is an eval: a written test with an answer key. Collect twenty to fifty real questions from your users, write down what a correct answer must contain, run the assistant on all of them after every change, and grade. Rules grade what is definite (did it state the right number of days? did it quote a section?). A second model grades what is a matter of judgement (was the tone right?), and that grader is itself checked against a sample of human grades before it is trusted.
A full article on evals, with a worked example and its own numbers, is here: [link to the evals article].
Important: a model that is not measured is not improving. It is changing.
What it means for us: any AI project should have its test set before its second week.
Why it matters: evals are the only way to say "the new version is better" and mean it.
Layer 11: interactive versus headless
Question: what changes when there is no person reading the replies?
Everything so far assumed a person at the other end, typing and reading. Interactive use means the assistant stops after each reply and waits for a human. Headless use means it is started by a program, works through a task, and ends without anyone watching. The same model, the same loop, and three things change.
Two numbers say how much of this is really new.
Headless is the step between an assistant and an autonomous system. An assistant you talk to; a headless agent you brief, launch, and inspect afterwards.
Layer 12: assistant versus autonomous AI
Question: when is it safe to let the agent run on its own?
An assistant does one task when asked. An autonomous system takes tasks from a queue, works through them without a human in the loop, and reports back. Getting from the first to the second is not a bigger model. It is five harness properties.
The fourth property, verification, carries the most weight, because the agent's own report cannot be trusted.
A full article on the road from an interactive assistant to an autonomous fleet, with the failures along the way, is here: [link to the autonomous AI article].
Layer 13: are "skills" useful?
Question: vendors and marketplaces sell "skills" for agents. Is that proficiency?
No. A skill in this industry is a text file: a description of expectations, rules and steps the model should follow for a kind of task. It does not make the model better at anything. It adds to the context, like any other instruction. Marketplaces now list skills in bulk, many of them generated by a model, and the studies below measured what they do to results.
A short, hand-written procedure for handling refund disputes helps. Forty installed skills about customer service do not.
Important: a skill is a text file in the context, not a capability. Curated ones help; generated ones in bulk hurt.
What it means for us: write few skills, by hand, by people who do the job, and treat marketplace skills as untrusted software.
Why it matters: the harm is twofold: worse answers and a new door for attackers.
Questions to ask your team
What is in the context on the first message? Ask for the list: system prompt, skill headers, tool definitions, memory notes. If nobody can produce it, nobody is managing it.How long does a typical session get? Use: a hard cap near 100,000 tokens. Then check: whether anyone has seen a session go past it.How many tools are connected, and are any two similar? Use: fewer than ten, distinct names. Then check: what the agent can reach when it runs them, and whether a sandbox fences it.What leaves the company in the context? Use: a written list of allowed data, direct connection to the vendor. Then check: whether tool outputs are treated as untrusted text.Where does each fact in a reply come from? Use: quotes from the context. Then check: a sample of replies against the quoted sources.Are the policies pasted, indexed, or searched? Use: paste under 500 pages. Then check: how the index or the search is maintained.Who owns each part of the knowledge base? Use: a named owner and a freshness date per part. Then check: when a changed policy last reached the agent.Who can write into memory? Use: only the harness, from approved steps. Then check: whether an update or a deletion really takes effect.What does the eval set look like? Use: twenty to fifty real cases with expected answers, each run several times. Then check: the "passes every time" number, not "passed once".What is the budget per run? Use: a hard spend limit the harness enforces. Then check: how much the same task varied between runs.What happens when the agent runs without a human? Use: brief, permissions and check written down before launch. Then check: whether "done" is verified against the system or taken from the agent's report.Which skills are installed and who wrote them? Use: few, by hand. Then check: anything from a marketplace.Twelve things to remember
An AI system is a model, a context, and tools. Nothing else.The model is stateless and cannot act. Memory is re-sent text; action is a request the software honours.The context is everything sent to the model, including the instructions, the tool list and every tool output. It is assembled before the first message.Shorter context, better and cheaper answers. Keep it under about 100,000 tokens and start a new session per task.Few, distinct, well-described tools. Everything a tool brings back is untrusted text, and the whole context leaves the company on every request.Fine-tuning is the hardest change and the last to try. It teaches behaviour, not facts.An agent is a loop around the model that honours tool requests. The loop is where control lives.A reply grounded in quoted text can be checked. A reply from the model's memory cannot. Require quotes, and control who writes the sources.The model never knows your business. A knowledge base is the machinery that puts the right page into the context.The harness, not the model, made the measured difference. Fence what the agent can reach; people approve almost every prompt. Log every step.Long work is many short sessions with state on disk and a spend cap per run. An eval, run several times per case, is the only way to know a change helped.Autonomy is queue, claim, isolate, verify, escalate, with "done" checked against the system, never the agent's report. Skills are text files; test each one against a run without it.References
Agensh (2026). Scaling Organizational Intelligence to 1,024 Agents. arXiv:2609.26781.Anthropic (2024). Introducing Contextual Retrieval. anthropic.com/news/contextual-retrieval.Anthropic (2025). Effective context engineering for AI agents. anthropic.com/engineering/effective-context-engineering-for-ai-agents.Anthropic (2025). Introducing advanced tool use on the Claude Developer Platform. anthropic.com/engineering/advanced-tool-use.Anthropic (2026). Demystifying Evals for AI Agents. anthropic.com/engineering/demystifying-evals-for-ai-agents.Anthropic (2026). How Anthropic enables self-service data analytics with Claude. claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude.Anthropic (2026). How we contain Claude. anthropic.com/engineering/how-we-contain-claude.Anthropic. Claude Code: How Claude remembers your project. code.claude.com/docs/en/memory.Anthropic. Context windows. platform.claude.com/docs/en/build-with-claude/context-windows.Anthropic. Reduce hallucinations. docs.claude.com/en/docs/test-and-evaluate/strengthen-guardrails/reduce-hallucinations.Anthropic. Working with the Messages API. docs.claude.com/en/docs/build-with-claude/working-with-messages.Automating Skill Acquisition through Large-Scale Mining of Open-Source Agentic Repositories (2026). arXiv:2603.11808.Bai et al. (2026). How AI Agents Spend Your Money. arXiv:2604.22750.Barnett et al. (2024). Seven Failure Points When Engineering a Retrieval Augmented Generation System. arXiv:2401.05856.Bodhwani et al. (2026). GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents. arXiv:2609.12191.Brown et al. (2020). Language Models are Few-Shot Learners. arXiv:2005.14165.Chacon Sartori (2025). The Specification Gap: Coordination Failure Under Partial Knowledge in Code Agents. arXiv:2603.24284.Chen et al. (2026). The Price Reversal Phenomenon. arXiv:2603.23971.Chroma Research (2025). Context Rot: How Increasing Input Tokens Impacts LLM Performance. research.trychroma.com/context-rot.Community analysis of the leaked Claude Code source (2026, unverified). x.com/iamfakeguru/status/2038965567269249484.Demystifying Agent Skills: Why They Work, Until They Don't (2026). arXiv:2608.14036.Dong et al. (2026). Agent Skills Can Be Harmful. arXiv:2608.11888.Gulati et al. (2026). Ask Early, Ask Late, Ask Right. arXiv:2605.07937.Han et al. (2025). RAG vs. GraphRAG: A Systematic Evaluation and Key Insights. arXiv:2502.11371.Hubotter et al. (2026). Compiling Agentic Workflows into LLM Weights. arXiv:2605.22502.HumanLayer (2026). A Brief History of Ralph. humanlayer.dev/blog/brief-history-of-ralph.Huntley, G. (2025). Ralph Wiggum as a "software engineer". ghuntley.com/ralph.Husain, H. (2024). Your AI Product Needs Evals. hamel.dev/blog/posts/evals.Kalai et al. (2025). Why Language Models Hallucinate. arXiv:2509.04664.Karpathy, A. (2026). LLM Wiki. gist.github.com/karpathy/442a6bf555914893e9891c11519de94f.Laban et al. (2025). LLMs Get Lost in Multi-Turn Conversation. arXiv:2505.06120.Liang et al. (2025). GraphRAG under Fire. arXiv:2501.14050.Lin, Li and Chen (2026). Mnemonic Sovereignty: LLM Memory Security. arXiv:2604.16548.Liu et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172.Lu et al. (2026). MARCH: Multi-Agent Reinforced Self-Check for LLM Hallucination. arXiv:2603.24579.McKinsey (2026). State of AI trust in 2026: shifting to the agentic era. mckinsey.com/capabilities/tech-and-ai/our-insights/tech-forward/state-of-ai-trust-in-2026-shifting-to-the-agentic-era.Meta-Harness: End-to-End Optimization of Model Harnesses (2026). arXiv:2603.28052.Mieczkowski et al. (2026). LLM Teams as Distributed Systems. arXiv:2603.12229.OpenAI (2025). A Practical Guide to Building Agents. cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf.Ovadia et al. (2023). Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. arXiv:2312.05934.Rabanser et al. (2026). Towards a Science of AI Agent Reliability. arXiv:2602.16666.Raj et al. (2026). Model or Harness? A Failure Taxonomy for Agent Systems. arXiv:2607.28802.Schick et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761.Shapira et al. (2026). Agents of Chaos. arXiv:2602.20021.Signal or Noise? A Benchmark Study of Agent Skills in Web Development (2026). arXiv:2608.23067.SoK: Agentic Skills Beyond Tool Use in LLM Agents (2026). arXiv:2602.20867.Stencil (2026). We improved 15 LLMs at coding in one afternoon. Only the harness changed. stencil.so/blog/the-harness-problem.Wang et al. (2026). The Verification Horizon. arXiv:2606.26300.Yao et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629.Zhang et al. (2026). Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems. arXiv:2604.14228.Zhang et al. (2026). Is Progressive Disclosure All You Need for Long-Context Agents? arXiv:2607.17598.Zhou et al. (2026). Are We Ready for an Agent-Native Memory System? arXiv:2606.24775.Zhu et al. (2026). Contextual Agentic Memory is a Memo, Not True Memory. arXiv:2604.27707.
