Behavioral harnesses made a cheap coding model reliable, and a frontier one pricier

A controlled test of coding agents with and without a behavioral harness, decoded for developers deciding between cheap models plus checks and frontier models.

Picture a two-person API shop building a backend-for-frontend for a retail client. One developer runs the top-tier model through Codex because it gets the work right, and the token bill is starting to show up in client invoices. They tried the cheaper model on the same brief. It produced something that looked done, but it handled the awkward cases wrong: the timeout that should become a 202 with a poll link, the transient failure that needs a 429 and a retry interval. Now they're stuck. Pay frontier prices for every task, or pay a human to catch the cheap model's misses?

An experiment published on martinfowler.com on 13 October 2026 tests that exact trade-off with numbers. It compares coding agents building the same API service with and without a behavioral harness. GPT-6 Luna, a cost-optimized model, passed in 3 of 10 runs without the harness and in all 10 runs with it. That cost $0.273 per run, against $0.406 for the frontier GPT-6 Sol running without a harness. For Sol itself, the harness bought nothing measurable and cost about 7 times more.

What it is

This is a write-up of a controlled experiment, not a product. The author borrows Birgitta Böckeler's framing: a behavioral harness combines guides, which steer the agent before it acts, and sensors, which observe what it did and feed back so it can self-correct. Working Surface's 14 October issue attributes the experiment to Naresh Jain, founder of the API testing company Specmatic. The post that surfaced it, from @bibryam, credits @thoughtworks.

The task was a product-discovery BFF. It had to validate requests and downstream responses, call an Order API, publish only the first returned product to Kafka, and map failures to specific status codes (202, 429, 503, 502). These are the edge cases where the lede's cheap model trips. Each capability of the harness lines up with one of those failures:

The author built the harness with Specmatic and writes that "equivalent contract-testing and service virtualization tools could provide the same capabilities." Evaluation was kept separate. Each run happened in an isolated Docker container with a fresh Codex home, and afterwards an outer loop ran 178 acceptance tests that the inner harness never had access to.

The alternatives, each from its own source:

Who it's for

Pick this up if you run repeated, well-specified agent tasks, especially API work with clear contracts, and want to move them to a cheaper model without losing reliability. The result also applies to anyone accepting agent output on the agent's own say-so. The author argues the same sensors serve as an independent acceptance check "which matters for every model." daily.dev's summary reads the conclusion as: cost-optimized models benefit from continuous feedback, while frontier models only need a final independent verification gate.

Skip the full inner-loop harness if you're already on a frontier model and it passes your acceptance checks consistently. For Sol, the harness used about 10.5 times the tokens and nearly 3 times the runtime, with no change in accepted runs. Skip it too if you don't have contracts or examples to build guides from. This harness was only as good as the specs fed into it.

Before and after

Before (cheap model, no harness), as the experiment ran it:

  1. The agent gets the brief and the baseline repo instructions.
  2. It writes the service on its own interpretation.
  3. External tests run afterwards. With Luna, 7 of 10 runs failed acceptance.

After (cheap model, behavioral harness):

  1. The agent reads the brief alongside the OpenAPI/AsyncAPI specs and examples.
  2. The skill directs it to map components and link each required behavior to its downstream interactions and checks. Helper scripts scaffold those documents and the build/start/test commands.
  3. It runs the project's test command. That validates the examples against the specs, runs build and unit checks, starts the mocks, launches the app and runs the contract and behavior checks.
  4. Reports name specific contract, value and interaction mismatches. The agent fixes the code, reruns the affected scenarios, then runs the full suite as a final gate.
  5. External acceptance tests run. With Luna, 10 of 10 runs passed.

For a frontier model, the evidence points to a lighter workflow: let it build unaided, then run an independent acceptance suite that the agent can't edit. In exploratory runs, Claude Sonnet 4.6, Claude Opus 4.6 and Gemini 3.8 Flash also completed the task unassisted on the first try, all at low reasoning effort, according to the article.

Try it

The sources do not publish the harness code, the application brief or the 178-test suite, so you can't rerun this experiment as-is. What you can reproduce is the method.

  1. Write the acceptance check outside the agent's reach first. The article's acceptance bar was strict: the app had to start and pass every external test. SpecStory's walkthrough shows the small version. Add a browser or API test under an acceptance/ path, add a deny rule so the agent's edit tools can't change that path, and add a Stop hook that runs the test when the agent tries to end its turn.
  2. Add guides only where the model misses. SpecStory's example adds one AGENTS.md line ("Changing the delivery address must not change the cart") after a specific failure.
  3. If you have API contracts, use them as sensors. The article generated contract checks deterministically from OpenAPI/AsyncAPI specs and validated example payloads against the specs before tests ran. That kept the guides and sensors from contradicting each other.
  4. Measure on your own task. Run the same brief several times per condition in clean, isolated workspaces, as the article did with Docker and a fresh Codex home. Count accepted runs, tokens, time and estimated cost separately.

SpecStory's wording on scope is "illustrative" and "simplified," and the sources give no commands, versions or pricing for setting any of this up.

Where it breaks

Sources