Why Software Factories Fail

or: the harness is not enough

Update - the talk version of this post is live on youtube: https://www.youtube.com/watch?v=Ib5GBkD555M

i guess we doin loops now

We're all racing to put AI coding into production. A lot has been said about loop engineering, and the prevailing wisdom is that we should probably write more loops.

StrongDM wrote about their lights-off software factory where no human reads code and no human writes code.

The narrative goes something like this:

Ryan Lopopolo of OpenAI wrote about this in February and gave a talk in April about OpenAI's software factory, Symphony.

These people are all really dang smart and I have a ton of respect for them. But the most cynical take here would be to call this yet another excuse to pump more VC money into the slop cannon.

it's uh...it's going

Our friend Mario got up at AI Engineer Europe and begged us to slow down -- because companies that have no business having outages due to coding-agent mishaps, are, well... having outages due to coding-agent mishaps.

As Matt Pocock put it, codebases are falling apart faster than they ever have before.

I haven't been able to dig up any definitive data/findings from StrongDM on how that whole dark factory went. The weather-report has a few sparse updates between February and June of this year. edit - there is some conversation with the team on hacker news on July 23 - sounds like we might get a more formal update soon!

The folks at Faros AI put out a report: since we2 all picked up these AI coding tools back in January and February, pull-request review quality is way down.

This report is more of a correlation signal than a verifiable smoking gun (yes i chose that word on purpose, don't get me started on claude prose), and the whole point of this post is to be wary of slop data, but it feels directionally valid based on what I've seen.

"You're holding it wrong" (you're not)

A lot of people will tell you that this is a skill issue -- that if you're not getting good results, that's your fault.

But however you're choosing to...erhm...hold it, I guarantee you're being told that if token-maxxing isn't working for you, it's a skill issue. You just need to spend more tokens. Let go of reading the code. And if you're just getting there, I promise it's part of the progression. I thought this way last summer too.

Unfortunately for my ego, some dumb stuff I decided to say about "how to hold it better" got recorded and now has about a million cumulative views on YouTube. I am not trying to brag here, I share this only to establish that I've been going deep on the best ways to use coding agents for a long time now, and have discovered some things that many others have found genuinely useful.

Anyhow, The promise of all this online "just token harder" yapping we've been forced to endure is, succinctly: with enough harness engineering, we can get the best of both worlds:

All we have to do is configure more linters and sprinkle some magic words like "adversarial review" onto enough PR review bots, and our software will happily build itself without incident.

This is not a skill issue

What I'm gonna try to convince you is that no amount of harness engineering or loopsmaxxing can solve what is fundamentally a model-training issue.

To grapple with this, I had to dig into how coding models are actually trained and evaluated - with respect to both the RLVR and the benchmark side of things.

In this post I'm gonna run through:

I'm gonna try to cut through the hype of every daily-emerging skills plugin and the ai-psychosis-tokenmaxxing advice pandemic, and talk in general terms about the types of things that work without referencing any particular skill or framework.

Video Version: this post is based on (and expands upon) my keynote at AI Engineer World's Fair 2026.

Thanks to @addyosmani, @CyrusNewDay, @HamelHusain, @zeeg, @dillon_mulroy, @nayshins, and @jeffreyhuber for feedback on this post.

An aside: this has nothing to do with vibe coding

Addy Osmani detangled this thing that is worth highlighting:

A developer vibe-coding a side project a dozen people will ever run, and a team keeping a ten-year-old enterprise system alive for another quarter, share almost no constraints worth naming, and most of the advice in circulation is really one of those two people telling the other how to live.

If you love vibe coding, please, go on vibing. I still vibe code lots of things, I just also maintain lots of production software (and through HumanLayer, help 1000s of other engineers do the same), so the rest of this is aimed at folks solving hard problems in complex codebases.

I hear the word brownfield a lot to talk about this split. Historically that meant some ten-year-old Java thing, but at the pace we can ship now, it feels like an agent-built codebase starts to struggle after maybe three to six months -- you start to slow down, and the way you approach adding new things has to change.

A brief history of the software factory

I've been building and studying software factories my whole career, but I only learned this recently: the term traces all the way back to a NATO conference in 1968 -- the same one that gave us "software engineering."

The only other bit I find super interesting since then is that the US Department of Defense wrote a 31-page pdf about how the DoD needs to start using jenkins better or something.

The 2022 software factory

Let's ground our "software factory" definition around 2022, right before AI. In a typical software factory:

And on and on. We haven't even hit AI yet, and there are already several loops in this picture.

front-loading alignment

There's a thing teams figured out decades ago: building takes hours or days, and so does review.

So we front-load the work -- planning, architecture proposals, sprint planning -- together, as a team. That means:

We'll come back to this later - let's look at what happens when you bring agentic coding into the picture.

The agentic software factory

Now every company and their mother --

has spent the better part of this year explaining how they built an agent factory that ships on the order of 75% of their code.

The agentic factory looks mostly like swapping "someone builds the thing" → "an agent builds the thing" -- there's some stuff here like orchestration, a harness, a sandbox, a model, computer use, etc. I won't go in depth on those details because quite frankly I'm sick of reading about it and I'm sure you are too.

When the agent builds the thing:

So you speed review up too:

Review is faster now, but it's also probably still the bottleneck. But we can do more loops.

Next you might route incidents into the factory. Instead of paging someone at 3am, they wake up to a PR that maybe already fixes it.

We can also route user feedback into the factory. People ask for stuff, it gets built.

At which point the job is two questions: how much can you stuff into the queue, and how fast can you review and test what comes out?

Which brings us to the lights-off software factory.

The lights-off software factory

Dan Shapiro coined this term and Simon Willison wrote about StrongDM's implementation of it -- where we no longer read the code.

You look at your beautiful software factory. It's ruined by that annoying little code review step and you say: you know what, that thing where a human reads every change? No thanks.

So you drop it, and you put the effort somewhere else:

And now the job really is just one question: how much stuff can we ask the agent to build? How much of the ocean do we want to boil?

This is going to go great (its not)

I'm going to posit something potentially controversial: the lights off factory does not work.

Let's get into why software factories fail.

We tried this

In July 2025 we went full lights-off. Just read the specs and the tickets, background agents for all the small/medium stuff, the whole thing.

If you've tried this seriously for a few months, you already know how it ends. You find at least one issue gnarly enough that the agent can't solve it -- even with your most advanced prompting and workflows.

Eventually you have to suck it up and go dig into the codebase you stopped reading three months ago, trying to figure out what's broken.

And in the meantime:

The first time this happened to us, I shook it off. Even though I'd just spent the better part of two weeks digging through claude spaghetti, "the downside risk was worth the velocity". By the ~third time in november, we decided it would be easier to rewrite from scratch, and my cofounder spent two whole weeks in VS Code (not even cursor) plumbing out all the patterns by hand.

models degrade codebase quality over time

What I want to get to is this: models have a shortcoming. They can't maintain and improve codebase quality over time -- not without a decent amount of human steering.4

When I say maintainability, I mean the specific thing where it becomes really, really hard to change one part of the codebase without breaking another part. This is Martin Fowler's shotgun surgery.

I'm not going to say much more about maintainability. There are a bunch of books you can go read about it

So, why can't models do software maintainability?

"But surely the models have gotten better since then"

At this point you might be dying to say: but Dex, surely the models have gotten much better since July

They have -- in some ways. In others they're about the same.

I can't prove this. You can't prove it either. There are no good benchmarks for a model's ability to maintain codebase quality. (More on where that's going later.)

THERE ARE NO GOOD BENCHMARKS for a model's ability to maintain codebase quality

But if you've worked with coding agents for a while -- and a lot of people are posting about exactly this -- you probably have the vibe already: they tend to make things worse over time, and make the codebase harder to work in.

So to figure out why this happens, I want to zoom out to the first great coding agent.

Claude Code won because of Reinforcement Learning inside the harness

Claude Code went from nothing to ~$4B -- now something like ~$9B -- in revenue in under a year.

Which is a little wild, because there were already great CLI agents. aider, cline, codebuff -- all predated Claude Code, all with genuinely great context engineering built in, all with the same tool set you might attribute to claude code: read, write, edit, grep, bash. I used them. They were good. But also, tool use would just... fail sometimes -- you'd watch it flail at the same edit three times and open your editor back up to do it yourself.

The SWE-Agent paper from 2024 outlines how small changes in tool shape make noticeable differences, e.g. including line numbers in ReadFile results, or changing an Edit tool from find/replace to line-range edits.

Then Claude Code launched and went vertical pretty quickly. You can hand-wave this as distribution, but the canonically-accepted explanation is that claude code won because it was better, and that it was better because Anthropic RL'd the model inside the harness -- the first time a lab trained a model against the exact tools they were going to ship it with. And it got really, really good at calling those tools in an agentic loop.

It's one thing to fiddle with tool definitions and evals until you find the shape the model likes best -- I've burned weeks doing this for various use cases. It's a different game when you own the weights and can modify the model itself to be better at a particular set of tools.

The OpenAI team gave a talk in November that put this pretty well: if you build a harness but you don't own the weights and can't RL the model inside it, you'll always be at a disadvantage to a team that owns both.

Coding Agent RL in 60 seconds

I did a bunch of research on this topic and cooked up a bunch of visualizations to try to explain the parts that matter, but I found that Calvin French-Owen (MTS on the codex team, founder of Segment) did a talk at AI Council that did a much better and cleaner job, so I'm just gonna drop this animation here inspired by his slides:

To make a model better at coding, you're gonna:

And then you do this millions of times over the course of weeks or months.

The "scoring" part of these things can tend to be whimsically one-dimensional though.

There's no penalty for bad design

Take SWE-bench Multilingual. The tasks are small -- about fifteen minutes of work apiece -- scraped out of open-source repos like Redis, jq, and Django. The reward is one or zero based on:

Here's a real one, fastlane__fastlane-19304, from fastlane -- a Ruby project. Its zip action grabs two optional params and calls .empty? on them straight away, so the moment you leave include and exclude off, it falls over:

The human fix that closed this particular issue is two lines (default nils to empty arrays):

During the evaluation, the model

The agent goes off and writes some code based on the issue. It doesn't see the golden patch or the test patch that serves as the grader:

Then:

Aside - Benchmarks are not verifiers - in fact they have to be held out from each other (don't train on test, yada yada) - I primarily mean this to convey the shape of "judging the quality of a coding agent trace" and its limitations.

How the model got to a correct answer doesn't matter. If the tests pass, we win, but there is no penalty for eroding codebase maintainability.

there is no penalty for eroding codebase maintainability

That's how you get try catches around everything:

Verifying quality is orders of magnitude harder than "did the tests pass"

Running the tests gets you a clean pass or fail in ~seconds. That's why RL can run millions of loops to optimize each model generation.

But the cost function of bad architecture is measured in weeks, months, maybe even years. It happens the first time someone opens that file for a one-line change and realizes they can't make it in one line -- that someone vibed this a little too hard, and now we have to make the same edit in eleven places and hope nothing quietly breaks three files over.

Tests give you feedback in seconds, but the cost function of bad architecture is measured in weeks, months, maybe even years

Bad design is the one thing today's benchmarks can't evaluate. And I know, I know, RL != Benchmarks, but if this was solved in RL, I'm pretty sure it would start to show up in how our benchmarks are designed too.

In any case, I personally don't trust any improvements on today's benchmarks as an indicator that the models are suddenly good at not slopping up your codebase.

The frontier is getting better, slowly

Of course lots of smart folks are working on this. My point is not that it can't be done, it's that the hype is outrunning the discipline.

A few efforts I think are pointed the right way:

But a model judging quality can only go so far.

In fact, it's not hard to imagine that if a model could reliably tell good code from bad, it might have written the good version to begin with. RL needs a fast+reliable oracle, and we don't yet have one for maintainability

if a model could reliably tell good code from bad, it might have written the good version to begin with, but maintainability has no fast oracle, so we can't reward for it during RL

Of course, more review agents and more tokens do help -- they raise the floor, catching the dumb stuff.

But they don't move the ceiling, because the ceiling is whatever we managed to teach the model in RL, and good design is the thing we still don't know how to teach it.

So I still wouldn't bet my codebase on any of these. But they're the first evals I've seen even trying to score maintainability instead of stopping at pass/fail.

Aside Maybe a future model just gets this and we can stop. If you want to yolo prompts until GPT-7 ships and find out, be my guest -- but bitter lesson be damned, we've got problems to solve now, and I'm gonna walk through how we do that.

Turning the lights back on

Today I learned that Twitter Articles have a "media limit" which means the rest of this is going into a part II post - stay tuned