I’ve spent the last few weeks building across a few different AI coding tools, and I stopped paying much attention to the model underneath. You spend enough time around AI and you start to believe the model is the part that matters, since it’s what everyone talks about and all the benchmarking is against. But build across enough tools in a short stretch and a different picture shows up. The model is closer to an interchangeable component, and the setup wrapped around it is what decides success.
I hadn’t put it that cleanly until a few nights ago, at the Prolific and AI Circle panel in London, “Fine-Tuning the Agent Stack,” with people from Overmind, Meta and ElevenLabs on the stage. The panel’s whole argument boiled down to one line: “The systems around the agent decide more than the raw model does, and a weaker model with a strong harness beats a stronger model with a bad one.” I’d been circling that idea for weeks without realising it, and hearing it said out loud made it click.
The expensive model changed almost nothing
I’d actually run the experiment myself a few weeks earlier, without clocking it as one. Kiro on Auto mode normally routes each task to the optimal model automatically. But I had credits going spare on a build I’d already finished, so I pinned Opus instead, at about 2.2 times the normal Kiro credit rate, and ran it back over the same work to see what the extra credits bought. The difference barely showed, and I mean barely: a slightly different turn of phrase in a comment here and there, nothing I’d have caught if I hadn’t gone looking. I paid 2.2x for the model, and the output came back the same. The one thing that did change was speed, it was quicker, I’ll give it that.
The reason it made no difference is that Kiro’s harness had already done the work. Surfdeck was built spec-driven, so every task started from steering docs that held the conventions and a tracker that held what was already proven. Each task began knowing the rules and knowing what passed, which left almost nothing for a smarter model to fix.
Changing the model barely changed the result because the task arrived with its context, constraints, and checks already in place.
The harness ran the loop
A week later I hit the same point from the other direction, building an Android app at a running hackathon. It ran on Devin, from Cognition, which doesn’t offer Opus, so there was no model to swap, but the model was never the thing that mattered here. This was my first Android app and I was building it between laps on a track in the rain, which is not a situation that flatters anyone’s reasoning.
Devin’s harness is what carried me. It works inside its own cloud machine, so it could stand up an Android emulator and run the whole loop, build, run, read the error, try again, without me leaving the tool or standing anything up by hand. Because that machine is yours to configure, you can keep adding to the harness: Devin has a system it calls Blueprints, version-controlled setup that describes the environment so every session boots into the same place. I only looked into that on the day without using it, but it’s the first thing I want to pull into my own setup next, either into my Claude harness or onto a machine of its own. The model was writing the code, but the scaffolding was doing everything around it.
Building between laps in the rain made the value of a self-contained environment unusually obvious. Photo: Tobie Charlton.
Where the harness closes the gap
None of this is mine to claim, though, because the people who study harnesses have found the same and put a value on it I couldn’t. The scaffolding matters most exactly where the model is weakest, so a weak model is a hostage to its harness and swapping it makes the scores swing wildly, while a strong model absorbs a worse setup and needs the scaffolding less. Which means it isn’t a fixed multiplier on the model, more a support whose value falls as the model climbs. My Opus swap doing nothing is that same finding read from the strong end of the curve.
There’s a real limit to all this worth keeping in mind. The harness only closes the gap up to the reasoning the base model can actually do, and if a task needs something the model can’t reason through, no amount of config or deterministic checks will supply it. Kiro’s work was scoped and test-gated, which isn’t true of everything. But for most of what I ask these tools to do, the model has stopped being the variable worth spending on, and the setup around it is the part I own. It travels with me or it doesn’t, and on identical hardware that was the whole difference between a good build and a worse one. So I’ve started treating my config like a real dependency, mostly getting the Mac and Windows setups to match, so next time the work doesn’t get worse.