AI Agent Engineer in 2026: 12 steps roadmap - Loops, Graphs, Evals, Context, Harness (Full-Course)

This is the complete 12-step roadmap through the seven pillars that actually decide whether an agent works: context, tools, memory, loops, graphs, harness, evals - each with the Claude workflow that proves it.

Frontier models went from 30% to over 80% on SWE-bench Verified in a single year. Coding agents got dramatically, measurably better.

Meanwhile only 17% of executives say they have fully adopted AI agents across their company.

Follow my Substack to get fresh AI alpha:movez.substack.com

Agents fail because context bloats. Because loops never converge. Because nobody can measure whether last week’s change made anything better. The demo works on your machine. Production is a different animal.

This is the 12-step roadmap through the seven pillars that separate the two - each with its own failure modes, its own Claude workflow, and its own way of quietly killing your agent if you skip it.

Here’s the reframe that makes the rest coherent. These are not seven separate skills. They are one skill in seven projections.

Which means there’s no useful way to learn them in isolation - but there is a dependency order, and it runs foundation first, execution second, reliability last. That’s how these twelve steps are arranged.

01. Context - read what actually loads

Karpathy’s framing is the one to hold: the model is the CPU, the context window is the RAM. Context engineering is the art of filling that RAM with exactly what the next step needs - no more.

The number that reframes everything: in Claude Code, roughly 7,850 tokens load before you type a single character - system prompt, auto memory, skill descriptions, CLAUDE.md, environment info, MCP tool names.

Your actual prompt is around 45. Everyone optimizes the 45 and never opens the 7,850.

/context prints your real breakdown by category, tells you which memory files loaded, and flags which tool call ate the most tokens.

Two numbers to watch: memory files (heavy means an overweight CLAUDE.md) and free space. Pair it with /memory to see the exact files in play.

02. Context - cut it, then layer it

Anthropic deleted over 80% of Claude Code’s system prompt for the Claude 5 generation and measured no loss on their coding evals.

Most context isn’t wrong - it’s guidance written for a weaker model that now just costs tokens and forces Claude to reconcile contradictions before it can start.

Two rules make the cutting safe:

Everything else becomes a skill (description loads at startup, body on invoke) or a path-scoped rule (loads only when a matching file is read).

# payments-api
Subscription billing and invoicing for the web app.

## Gotchas
// Non-obvious. Claude cannot infer these by reading the repo.
- All shared types live in `src/types.ts` — one monolithic file.
- `Money` is integer cents, never a float.
- Webhook retries must stay idempotent — provider replays for 72h.
- `db/legacy/` is frozen. Read it, never edit it.

## Deeper guides
- Verification → `.claude/skills/verify/`
- Releases    → `.claude/skills/deploy/`

// NOT here: directory tree, framework, test runner, language
// version. Claude reads those from the repo itself.

03. Tools & MCP - what the agent can reach

Tools are how an agent touches the world, and their descriptions are context - which makes this the hinge between what the agent can do and what it can afford to know about.

The old advice was to teach tools with examples.

That inverted: with current models, examples constrain Claude to the exploration space they describe. Design expressive parameters instead.

A status enum of pending | in_progress | completed teaches an entire lifecycle without a single example - the type is the documentation.

Anthropic reached state of the art on SWE-bench Verified partly through precise refinements to tool descriptions, not model changes.

And deferring beats dumping. Claude Code loads MCP tool names only - around 120 tokens - fetching schemas on demand through tool search.

Applying retrieval to tool descriptions rather than loading all of them improves selection accuracy roughly threefold.

Which makes the description the discovery surface: say when the tool applies, not just what it does. A tool Claude can’t find is a tool you didn’t ship.

@mcp.tool()
def find_stalled_shipments(hours: int = 24) -> list[dict]:
    """Shipments with no scan event in N hours.
    Use when ops asks what is stuck, or before an escalation review."""
    return query(STALLED_SQL, hours)

@mcp.tool()
def reroute(shipment_id: str, hub: str, reason: str) -> dict:
    """Reroute a shipment. Writes an audit row — compliance
    requires a reason on every manual intervention."""
    return post_with_audit(shipment_id, hub, reason)

# "Use when..." is the discovery surface for tool search.
# `reason: str` is required, so the audit trail cannot be skipped.
# The type enforces what a paragraph of instructions would only ask for.

##

04. Memory - what survives the window

Every long task eventually exceeds one window.

What happens at that boundary is a design decision most people never make - they let automatic compaction guess, then wonder why the agent forgot a constraint stated an hour ago.

The mechanics are specific and worth memorizing. Project-root CLAUDE.md and auto memory are re-injected from disk.

But rules scoped with paths: and nested CLAUDE.md files live in message history - they get summarized away and don’t return until a matching file is read again.

Invoked skill bodies come back, capped at 5k per skill and 25k total, oldest dropped first and truncated from the end - so anything critical belongs at the top of a SKILL.md.

The reliable move is the oldest one in computing: write it to a file.

A plan file, a progress log, notes the agent rewrites as it goes. It persists without occupying the window and survives compaction because it lives on disk.

Compress deliberately too - /compact focus on the auth bug keeps what you chose, and /clear is badly underused when the next task doesn’t depend on the last twenty messages.

05. Loops - when to stop

An agent loop is act → observe → decide → repeat. The entire engineering problem lives in that last step: how does it know it’s finished?

Left to itself, a model answers badly in two directions - declaring victory on half-built work, or grinding forever on something that was done three iterations ago.

Neither is fixed by a better prompt, because both are structural. The stopping condition belongs in code that sits outside the model’s judgment. For bounded work that’s a test gate or a schema check.

For discovery of unknown size, the converging pattern is loop-until-dry: keep going until K consecutive rounds surface nothing new.

One detail makes or breaks it, and almost everyone gets it wrong first time: dedupe against everything seen, not against confirmed results.

Otherwise rejected findings reappear every round, the loop never runs dry, and you’ve built a machine that pays forever to rediscover the same dead ends.

const seen = new Set(); const confirmed = []; let dry = 0;

while (dry < 2) {                      // stop after 2 empty rounds
  const found = await runFinders();
  const fresh = found.filter((b) => !seen.has(key(b)));

  if (!fresh.length) { dry++; continue; }
  dry = 0;
  fresh.forEach((b) => seen.add(key(b)));
  // ^ dedupe against SEEN, not against confirmed.
  //   Backwards, and this loop never terminates.

  confirmed.push(...(await verify(fresh)));
}

06. Loops - who checks the answer

A converging loop still converges on whatever the model believed. The fix is a verifier - something outside the model’s own judgment whose only job is to try to kill the finding.

If it survives, it passes. If not, it never reaches the answer.

Three patterns worth having in hand:

› For each finding, spawn three verifiers - correctness, security, reproducibility. Accept only what survives two of three. 

● 6 findings → 18 verifier agents, running in parallel 

✓ missing auth on /invoices/:id 3/3 — accepted ✓ race in webhook retry 2/3 — accepted 

● "unsafe regex in validator" 0/3 — killed 

● "N+1 query in dashboard" 1/3 — killed 2 of 6 findings were confident and wrong. The loop caught them, not your reviewer, and not your users.

Note where this points. A verifier is an eval running inline - the same discipline as Layer III, applied per-result instead of per-release. Teams that build good verifiers find step 11 much easier, because they’ve already written down what “good” means.

07. Graphs - what runs in parallel

Most people write agents as a straight line - step one, step two, step three, each politely waiting for the last. Then they notice half those steps never needed to wait at all.

A node is a unit of work an edge means this output feeds that input.If no data crosses, there is no edge - and the wait is pure waste.

The workhorse shape is the diamond: fan out to gather breadth, reduce with plain code, synthesize with one agent.

The reduce step deserves emphasis because it’s where money leaks - flattening and deduping is flatMap and a Set, not an agent.

Edges are free. Spend agents on judgment, not plumbing.

The ceiling is genuinely high. Claude Code’s dynamic workflows coordinate up to 1,000 parallel subagents in one run, and the orchestration costs zero model tokens because it’s a script, not a conversation.

› Run a workflow to audit every route under src/routes/ for missing 
auth. One agent per route file, verify each finding before reporting. ● 
Claude wrote an orchestration script · launching… ✓ Scope 1/1 ✓ Fan-
out 18/18 one agent per file ◯ Verify 11/18 3-vote skeptics per finding 
○ Synthesize 0/1 your session stays responsive — the fleet runs in the 
background

That architecture is how a team ported ~960,000 lines of the Bun runtime from Zig to Rust in six days, with 99.8% of the test suite still passing.

08. Graphs - what the shape costs you

Topology isn’t cosmetic - it’s the biggest lever you have on both latency and spend, and two choices carry most of it.

First, parallel() versus pipeline(). A parallel() barrier makes everything wait for the slowest node before the next stage starts.

A pipeline() streams each item through all stages independently - item A can be in stage 3 while B is still in stage 1.

Default to pipeline. Reach for a barrier only when a stage genuinely needs every prior result at once, like a cross-set dedupe. “It felt cleaner” is not a reason; barrier latency is real, measurable, wasted time.

Second, model tiering per node. Every subagent inherits your session model unless the script overrides it, so a big run bills entirely at your top tier by default.

Bounded, repetitive nodes - extract this field, classify this ticket - belong on a cheaper model the merge node where judgment actually happens stays high.

A hundred cheap fan-out nodes feeding one top-tier synthesis costs a fraction of the same job run flat, at the same final quality.

09. Harness - surviving session death

Anthropic frames the problem perfectly: a long task is a software project staffed by engineers working in shifts, where each new engineer arrives with no memory of the previous shift.

Compaction alone doesn’t fix it. Even a frontier model looping across context windows on “build a clone of claude.ai” falls short - and it fails in two specific, repeatable ways.

First, the agent tries to one-shot the app, runs out of context mid-implementation, and leaves the next session a half-built undocumented feature to reverse-engineer.

Second, later in a project, an agent looks around, sees real progress, and declares the job done.

Both are fixed before any coding starts, by an initializer agent that runs once and builds the environment: an init.sh that starts the dev server, a progress log, an initial git commit, and a feature list in JSON - over 200 entries for the claude.ai clone, every one marked failing.

{
  "category": "functional",
  "description": "New chat button creates a fresh conversation",
  "steps": [
    "Navigate to main interface",
    "Click the 'New Chat' button",
    "Verify a new conversation is created",
    "Check that chat area shows welcome state"
  ],
  "passes": false
}

// 200+ of these, all starting false. Agents may ONLY flip `passes`.
// "It is unacceptable to remove or edit tests" — deliberately strong
// wording, because a model that CAN edit the spec WILL edit the spec.

One detail worth stealing outright: they chose JSON over Markdown for that list, because models are measurably less likely to inappropriately rewrite JSON.

10. Harness - one increment per session

With the environment scaffolded, each coding session gets a contract: get oriented, pick exactly one feature, verify it as a user would, leave the repo clean.

“Clean” means what it means on a real team - no major bugs, orderly code, and a next engineer who can start work without cleaning up someone else’s mess.

The orientation ritual is mechanical and saves tokens every time: pwd, read the progress file, read the git log, read the feature list, run init.sh, and test that the basics still work before touching anything.

That last check matters more than it sounds - without it, an agent starting a new feature on top of a broken app just makes the breakage deeper.

And on marking things done: Claude’s tendency is to complete a change, run some unit tests, and call it finished without ever checking the feature end to end.

[Assistant] Getting my bearings on the current state. [Tool] bash — 
pwd [Tool] read — claude-progress.txt [Tool] read — feature_list.json 
[Tool] bash — git log --oneline -20 [Tool] bash — ./init.sh start the dev 
server [Assistant] Verifying fundamentals before touching anything… ✓ 
core chat, theme switching, conversation loading all pass [Assistant] 
Picking the highest-priority failing feature. no guessing. no 
archaeology. the previous shift left notes.

Give it real testing tools and require it to verify as a human user would - browser automation, actual clicks.

That single requirement dramatically improved performance in Anthropic’s experiments, catching bugs invisible from the code alone.

11. Evals - a number, not a vibe

The breaking point is always the same sentence: users report the agent feels worse after changes, and the team has no way to verify except guess-and-check.

Without evals, debugging is reactive - wait for complaints, reproduce manually, fix, hope nothing else regressed. You can’t tell a real regression from noise.

Start smaller than you think. Teams delay because they imagine needing hundreds of tasks; 20–50 drawn from real failures is a great start, because early changes have large effect sizes.

Pull them from what you already test manually, your bug tracker, your support queue. Write them so two domain experts would independently reach the same verdict - ambiguity in a task becomes noise in the metric.

And build balanced sets: test where a behavior should fire and where it shouldn’t, or you’ll optimize an agent that searches for everything.

Combine three grader types deliberately.

And grade what the agent produced, not the path it took: checking for an exact sequence of tool calls is brittle, because agents regularly find valid approaches you didn’t anticipate.

task:
  id: "fix-auth-bypass_1"
  desc: "Fix authentication bypass when password field is empty"
  graders:
    - type: deterministic_tests     # fast, objective, reproducible
      required: [test_empty_pw_rejected.py]
    - type: llm_rubric               # nuance tests can't capture
      rubric: prompts/code_quality.md
    - type: static_analysis
      commands: [ruff, mypy, bandit]
    - type: state_check              # the OUTCOME, not the claim
      expect: { security_logs: { event_type: "auth_blocked" } }
  tracked_metrics:
    - type: transcript
      metrics: [n_turns, n_toolcalls, n_total_tokens]

# The agent says "fixed". The state_check decides whether it was.
# Grade outcomes, not announcements.

12. Evals - keeping the number honest

A suite nobody reads is a number nobody should trust. You won’t know your graders work until you read transcripts from many trials - when a task fails, the transcript tells you whether the agent made a genuine mistake or your grader rejected a valid solution.

Failures should feel fair: obvious what went wrong and why.

Two traps make good agents look bad.

Then hold two metrics apart. pass@k is the odds of at least one success in k tries - it rises with k. pass^k is the odds that all k succeed - it falls, fast. At 75% per trial, three trials all passing is only ~42%.

Use pass@k where one success is enough; use pass^k for anything customer-facing, where users expect it to work every time.

› Run the suite against the new model and compare to baseline. ● 47 
tasks × 3 trials · 141 runs · parallel capability suite pass@1 61% → 74% 
+13 regression suite pass@1 99% → 99% held consistency pass^3 
52% → 68% +16 ● 2 regressions: refund_partial, escalation_tone → 
transcripts written to ./eval-results/failures/ the upgrade decision 
took an afternoon, not three weeks

Finally, make it routine. Wire the suite into CI so it runs on every change and every model upgrade.

This is what turns a new model release from weeks of manual testing into a day of running your suite and reading the diff - and it’s the compounding advantage teams without evals never catch up on.

##

7 jobs to run with Claude - one per pillar

Conclusion:

Anyone can get a demo working. The job is everything after that.

The models keep getting better on their own schedule, and that improvement is free - it arrives whether or not you did anything.

What doesn’t arrive for free is the system around the model.

Most people will keep waiting for a model good enough to not need any of this.

The ones who build the twelve steps will ship agents that work on days the model is having a bad one - which, in production, is the only reliability that has ever mattered.