Uncle Bob called it Distributed AI and I can't do better

I recently shared Squeaky Clean with Robert C. Martin and he was kind enough to tweet about it to his followers. He hasn’t run it, and I am careful not to confuse it for an endorsement. However, the way he described the project’s central idea stuck with me.

I haven't tried this, but it looks interesting. Fascinating concept of distributed AI using high parameter models for architecture and massively parallel low parameter models for implementation.

What caught my eye were the phrases “distributed AI”, “high-parameter models for architecture” and “massively parallel low-parameter models for implementation”. Those are the central value proposition of the project, and they got me thinking about the economics of Squeaky.

The two ways to lose

Every team pointing an AI at its codebase juggles two failure modes. Which two? Well, the small models are cheap, fast, and prone to making things up: ask one for a whole feature and it’ll cheerfully invent a method that never existed. The big models are steadier, but they cost more per token, and the cost climbs faster than linearly with context, which a big problem forces. Two paths, two tolls. For a while I just bounced between them.

Where the money hides, and how splitting beats it

So where does the money go? Well, when a model underperforms the instinct is to feed it more: more context, more codebase pasted in. But, that runs face-first into how these models work. Attention (the mechanism under every language model shipping today) scales with the square of the input length. Double the context and you roughly quadruple the attention work. The bill tracks that curve, the latency tracks it, and so does the error rate: fifty thousand tokens is fifty thousand chances to fixate on the wrong one. A long context stops reading like a convenience you pay a little for, and starts reading like the bill itself.

So don’t let one model see the whole problem. A capable, expensive model looks once and does only the architectural work: the modules, the classes, what each is on the hook for, how they depend. It writes the plan, not the implementation. Take a payment flow: it decides on a coordinating service, interchangeable processors behind one interface, a repository, and a couple of value objects, then fans those out to separate agents at once. Each agent sees a single class specification. Nothing else. Its context is a few hundred tokens instead of tens of thousands, so it has almost nowhere to hallucinate.

Here’s what made me commit. Say a problem carries ten thousand tokens of context whole; on one big model the attention cost scales with ten thousand squared. Chop it into twenty classes of five hundred tokens, and each pays five hundred squared, so twenty together come to one twentieth of the whole. Split into k pieces and you cut the attention work by roughly a factor of k. The pieces then run on models that cost far less per token, and because they don’t depend on each other they run at once, so twenty classes take about as long as the slowest one. None of it is free (nothing ever is): the architect still runs, splitting and stitching carries overhead, and some jobs flop and get retried. Even so, the curve holds: a quadratic punishes big contexts, and the surest way to beat one is to quit feeding it.

What keeps the cheap models honest

So why doesn’t this collapse into twenty disconnected classes? Well, the small models are cheap because their world is small, and it’s small because the architecture tier bricked hard walls around each one first. Each agent gets a compact specification, on the order of two hundred characters, naming its class, the pattern to implement, its methods, and the exact classes it may depend on. That spec is frozen and machine-checkable: a validator throws out anything that doesn’t parse before an implementation model sees it, so the cheap tier never interprets prose or guesses at intent. It gets a contract it satisfies mechanically, and it can’t reach past the boundary, because the boundary is all it can see.

On top of that sit rules the code must satisfy no matter which model coughed it up: one class to a file, a ceiling on methods per class and arguments per method, and a dependency rule that forbids inner layers from importing outer ones, checked as a graph instead of trusted to good intentions. Turned loose with no constraints, a cheap model hands you exactly the mess the skeptics picture. Fence it into one well-specified class, though, and what comes back is small enough to verify by machine.

Where it stops working

I won’t oversell it: the approach has an edge past which it stops working. It rests on the problem decomposing cleanly into classes with narrow, stable interfaces. Plenty oblige. Some flatly don’t. When behavior is tangled, when getting one class right means juggling three others, splitting across isolated agents fights the problem, and the architecture tier must swallow that complexity or the run degrades. That tier is also the ceiling: every agent is only as well-aimed as the plan it was handed, so a weak decomposition can’t be rescued by strong implementation. The small models also fail more often than the big ones, which means retries, which means the real cost sits north of the clean arithmetic. The framework carries an evaluation harness for exactly this reason: I don’t trust the theory on its own. It measures the actual tokens, failures, and architecture, so the whole thing gets checked against numbers, not hope.

Why I keep building this way

Strip away the machinery and the idea Martin pointed at is almost boring in its simplicity. Use the expensive, capable thing for the few decisions that need judgment, and the cheap, plentiful things for the large volume that only needs doing correctly inside clear walls. We organize human teams this way, and built compilers the same way for decades: a clever front end plans, a mechanical back end emits. What’s new is that the cheap, plentiful thing can write code now, and the only way to trust it is to keep its world small and its walls firm. The economics make it practical; the constraints make it safe. I found that worth building on, and apparently worth a second look from a man whose books taught me most of what I know about drawing those walls. I’ll take that as reason enough to keep going, and to keep showing the math.