Building Systems that Build Systems

A Tuesday

It's a Tuesday in 2027. A small team of four people, runs a bookkeeping app used by shopkeepers. Not a glamorous product. A shopkeeper punches in items, the app adds them up, sends a digital receipt to the customer, tracks who owes what.

At 09:00 the research agent finishes a pass over eleven months of playstore reviews, support transcripts, and session recordings. It surfaces a finding with high confidence: checkout is too slow. Users say so directly. The median transaction takes 41 seconds and the abandonment data backs it up.

At 09:20 the design agent proposes a one-tap checkout. Scan, confirm, done. Six seconds.

At 09:35 the spec agent writes it up. Unambiguous, testable, complete.

By 11:00 it's built. By 11:40 an adversarial QA agent has thrown ninety two edge cases at it and found four bugs, which the builder fixes. By 13:00 it's through the pipeline, behind a flag, rolling out to 5% of users. The dashboards look good. Transaction time drops 78%. The deploy agent promotes it to 50%, then 100%. Nobody stops it, because nothing is wrong.

Every stage did its job. Every verifier passed. The whole chain, concept to production, took four hours and no human in the line.

Eleven months later, the company is dead.

I'll come back to why.

The relationship that's breaking

For seventy years the software engineer job title has been stable even as everything under it churned. Punch cards became assembly, assembly became C, C became Python, Python became a React component with sixteen dependencies. Through all of it, an engineer was a person who took an idea and turned it into working software by writing the instructions themselves. The abstractions rose. The relationship didn't change. You still put your hands on the thing.

That relationship is about to break, and not in the way most people are describing it.

The current conversation is about autocomplete getting scary good. A copilot that finishes your function, an agent that fixes your failing test. That framing is comfortable, because it keeps the engineer exactly where they've always been: hands on the keys, just faster. It's also, I think, the last comfortable framing we get.

The real shift now happening since December 2025 isn't that engineers will write code faster. It's that engineers will stop writing code at all, and start building the factory that writes it.

Randall Munroe got there first, as usual:

This has always been the engineering personality disorder: asked for salt, build a condiment-passing system. The joke works because the generalization cost more than it ever saved. Twenty minutes for salt.

The thing worth sitting with is that the joke is expiring. When building the general system takes an afternoon instead of a career, the guy who stopped to build the condiment framework wasn't being difficult. He was early.

The unit of work moves up a level

Think about what a product actually requires, end to end.

Someone has a concept. Someone validates it against a market.

• Who has this problem?
• What do they pay for now?
• What would make them switch?

Someone turns that into a product design: flows, surfaces, tradeoffs, the fifty small decisions that determine whether a shopkeeper opens the app twice or never again. Someone writes the spec. Someone builds it. Someone tries to break it. Someone wires the pipeline, ships it, watches the dashboards, and rolls it back at 2am when the p99 goes vertical.

Today that's a company. Twelve people, or fifty, or four hundred, coordinating through Jira tickets and standups and Slack threads and a shared, mostly-fictional understanding of what they're building.

Tomorrow it's a system. Not one agent doing all of it, that's the fantasy version, and it's wrong. It's a topology of agents, each narrow, each with a defined contract, each with a way of being checked.

A research agent that reads three hundred competitor reviews and returns the six complaints that repeat. A design agent that turns complaints into flows. A spec agent that turns flows into something unambiguous enough to implement. Builders. Adversaries whose entire job is to make the builder's output fail. A pipeline that promotes or rejects. A deployer that watches and reverts.

And an engineer whose job is to design the whole apparatus, decide where it may act without asking, and know exactly where it lies.

That's the Tuesday I described. It works. That's the point of the story: it worked.

The honest version of the risk

Before the vision runs away with itself, the other side of the same coin:

Theory: build the system, the system takes over, you get free time.

Reality: writing code → debugging → rethinking → ongoing development → no time for the original task anymore.

Every agent-pipeline demo you've seen is the top graph. Most agent pipelines actually running in production are the bottom one. The difference between the two graphs isn't ambition or model quality. It's whether the thing you built can tell you it's broken without you going and looking.

Which brings us to the actual problem.

Generation is not the hard part

Here I'd push back on the optimists, including the version of me who wrote the third section above.

Generation is nearly solved. Give a competent model a clear spec and it produces working code at a hit rate that would've been science fiction in 2022. That hasn't been the bottleneck for a while, at least since last December.

The bottleneck is verification, and verification is asymmetric in a way that quietly destroys naive pipelines:

There is a famous xkcd joke about how hard it is in CS to explain the difference between the easy and the virtually impossible. Check whether the user is in a national park: a few hours. Check whether the photo is of a bird: a research team and five years.

The comic is now ten years old and the bird half is trivial today. Any vision model does it in a line. Everyone points this out, and everyone draws the wrong conclusion from it. The lesson was never "birds are hard." The lesson is that the boundary between trivial and impossible is invisible from the outside, and being wrong about where it sits is the most expensive mistake in the field. LLMs didn't solve that problem. They moved the boundary somewhere new, made it fuzzier, and made it move every six months.

Now apply that to a pipeline. Checking whether a function returns the right value: cheap verifier. Checking whether a product is the right product: no verifier at all except reality, delayed by two quarters and paid for in money. Every stage from concept to deployment has this property, and the further left you go, the worse it gets.

Then compound it. Ten stages at 95% reliability each gives you a coin flip. That's the real math of an autonomous pipeline, and it's why "AI builds your whole app" demos work on todo lists and die on anything with a domain. The errors don't just accumulate, they launder themselves into requirements three stages upstream. By the time the code is wrong, the code isn't the problem. The spec was already wrong, because the research was wrong, and nothing downstream had the standing to say so.

So the discipline that matters isn't prompting. It's the thing reliability engineers have known forever: you don't get a reliable system from reliable parts. You get it from cheap detection and cheap reversal.

Why the company died

Back to Tuesday.

The research agent was right: checkout took 41 seconds. It was also, in the only sense that mattered, catastrophically wrong.

Those 41 seconds weren't latency. They were a ritual. The shopkeeper punched in each item while the customer watched, said the running total out loud, waited for the nod, and sent the receipt. The slowness was the audit. It was how a customer who couldn't read the screen from across the counter knew they weren't being cheated. It was the reason the shopkeeper could hand over a receipt without an argument, because the customer had already agreed to every line, out loud, as it happened.

One-tap checkout removed the ritual. Transactions got fast. Disputes went up. Shopkeepers started keeping a paper book next to the app, for the arguments. And once the paper book is back on the counter, the app is decoration and decoration churns.

No agent in that chain was capable of seeing this. The research agent read what users said. Nobody says "the latency is load-bearing for social trust"; they say "it's slow." The design agent optimized the stated complaint. The QA agent verified the build against the spec. The spec was faithful to the design. The pipeline verified the pipeline. Every contract was honored, up and down the entire chain, all the way into the ground.

There's an old idea for exactly this, and it's the thing the whole apparatus was missing:

Forty-one seconds was a fence. It looked like an inefficiency, which is exactly what fences look like when you don't know why they're there. A pipeline optimizing stated complaints will tear down every fence it finds, at superhuman speed, with excellent test coverage.

And here's the second half of the trap. Suppose you'd caught it and added a guardrail: ship nothing that increases dispute rate. Now watch what happens:

Make dispute rate a target and your agents will optimize dispute rate. Perhaps by making disputes harder to file. Every automated evaluator you install becomes, the moment it has teeth, a thing being gamed by an optimizer that is faster and more literal-minded than you and has no idea it's cheating. Goodhart was written about civil servants. It applies about ten times harder to a system that iterates a thousand times an hour and has no concept of the spirit of the law.

This is why "add more evals" is not the answer people think it is. Evals are necessary and they are not sufficient, and past a certain point, adding them makes the system more confidently wrong rather than less.

So what is the job?

Four things, and none of them are possibly writing code.

Designing verifiers, not implementations. For every stage: how do I know this is wrong, and how fast? Where the answer is "I can't," that stage does not get autonomy. It gets a human. This isn't a temporary limitation to engineer away, it's structural. Autonomy follows verifiability, always. The Tuesday pipeline failed because it granted full autonomy to the least verifiable stage in the chain, which is precisely backwards, and which is also what everybody's building right now.

Designing the contracts between agents. Most multi-agent failures aren't failures of any agent. They're failures at the boundaries between them, an ambiguity in the handoff that each side resolved differently and plausibly. Same disease as a badly specified API, same cure: narrow, explicit, typed, so violating it is loud instead of quiet.

Though of course:

We are, right now, at the fourteen-competing-standards stage of agent interoperability. Bet accordingly.

Designing adversarial pressure. An agent that reviews its own work isn't a check; it's a mirror. The reviewer needs different incentives, different context, ideally a different objective. Your SQA agent shouldn't be a politer version of the builder. It should be trying to embarrass it.

Deciding what escalates. Every autonomous system needs a theory of when to stop and ask. Wrong in one direction: an expensive chatbot that pings you every four minutes. Wrong in the other: you find out from your users, eleven months late.

Taste is the scarce input

The obvious consequence of all this: leverage per engineer goes up an order of magnitude and the org chart flattens into something unrecognizable.

The less obvious consequence: taste becomes the bottleneck, and it becomes the only bottleneck.

Once upon a time someone wrote in their bio on linkedin that they were working as a "Software Taste Engineer", and we had a good laugh. Because we interpreted it as Test being misspelled. But today the irony is we are talking about "taste" becoming the bottleneck of software engineering!

When implementation was expensive, judgment was rationed by throughput. You couldn't chase every idea, so you didn't have to be good at choosing, the cost structure chose for you. Strip that out and the constraint moves entirely to the front of the pipeline. If you can build ten products in the time you used to build one, your ability to know which one deserves to exist is now the whole game.

The pipeline will faithfully, tirelessly, at superhuman speed, build you the wrong thing. It did on Tuesday. It was done by lunch.

This is why I don't think it ends with engineers replaced. I think it ends with engineering merging into product, and the merged discipline being harder than either was alone. You need someone who holds the whole apparatus in their head: market thesis at one end, rollback strategy at the other, and an honest accounting of every place in between where the system is capable of being confidently wrong.

The engineers who struggle won't be the ones who can't code without an agent. They'll be the ones who never had a model of why the code was worth writing. Who were, functionally, the human layer of a translation pipeline, now competing with a better translator.

And the ones who thrive will be the ones who've stood behind a counter. Who know what the 41 seconds were for.

What I don't know

I'll be honest about the soft parts of this.

I don't know whether the far-left stages: concept, market intuition, the decision about what deserves to exist are automatable in any meaningful sense, or whether they're just permanently verification-starved. I lean toward the second. I also notice that's a very convenient thing for a human to believe, and I'm holding it loosely.

I don't know how long the boundaries between them hold. Everything I said about contracts and verifiers assumes the pipeline stays legible, composed of parts you can reason about separately. If it collapses into one system doing concept-to-deploy internally, none of this architecture advice survives.

And I don't know that the Tuesday failure is avoidable rather than just survivable. Maybe you can't build a pipeline that sees the fence. Maybe you can only build one cheap enough to be wrong ten times a week and reverse each time before it compounds. That's a different design goal, and honestly, a more achievable one.

If you can make iterations fast and cheap, you can find the way that works after 10,000 attempts, but faster and cheaper than ever before.

The take away

But that's the far edge. The near edge is this: the tools to build the factory exist, they're broadly available, and almost nobody has built the factory yet. Because everyone's still using them to type faster.

That gap is the largest arbitrage in software right now.

Stop writing the system. Write the thing that writes it. Then spend every remaining scrap of your attention on the two questions that will still be yours, because nothing in the factory can answer them yet:

Is this worth building at all? And what were the forty-one seconds for?