Anthropic's measure-first agent loop: 3,000+ merges and a 3x faster claude.ai

Anthropic's account of an agent-driven two-week performance sprint, decoded for small teams that want to copy the benchmark-ratchet-review loop.

Picture a two-person team running a React SaaS dashboard. Customers keep saying it feels sluggish. Today the fix goes like this. Someone runs a profiler once, has a hunch, ships a change, and eyeballs the next week's dashboards. They can't say whether the win held. Three sprints later a new feature quietly undoes it, and nobody notices until the complaints come back. An AI coding agent could write the optimizations, but with no number to beat, it's guessing too. Whatever it ships still has to be reviewed by the same two tired people.

Anthropic's post How we made claude.ai 3x faster in two weeks (published 2026-09-23) is a detailed vendor account of fixing exactly that gap. The loop has three parts: make each slow thing measurable, let an agent climb against that measurement, and lock in each win with a CI guardrail that can only get stricter. On X, @trq212 framed the takeaway less formally: "have you tried telling Claude 'we have the power to do anything, please be braver'". The part worth copying is less the bravery than the measuring.

What it is

This isn't a product launch. It's a method, with numbers, and all of them come from Anthropic's own account. The company says that in August it made the core experience of claude.ai and the desktop app about 3x faster in a two-week sprint, run from a single Slack channel with Claude in every thread. At the 75th percentile:

The agent was Claude Tag (beta) running "an internal research model roughly comparable to Opus 5.5." Anthropic says Claude found bottlenecks, built benchmarks, shipped improvements and watched deploys. Humans set goals, made tradeoffs and approved every change. The team reports more than three thousand merged changes "without a single customer-facing incident or rollback."

Here is how each piece maps to the lede's pain:

Tech Bytes' write-up restates the same figures and adds per-journey numbers, but it does no independent testing.

Alternatives, if you want the loop without Claude Tag:

Who it's for

Pick this up if you already have real-user telemetry, a CI pipeline you trust, and feature flags or staged rollouts. You also need a hot path users complain about, and someone willing to review a steady stream of small PRs. The method assumes those foundations exist. Anthropic set up its safety mechanisms before the sprint started, not during it.

Skip it if you have no tests, no production metrics, or no appetite for reviewing agent output. Without those, the loop has nothing to climb against and nothing to stop a bad change. Also skip it if you're expecting Anthropic's pace. Its results came from an internal model and a multi-engineer team running more than 150 threads at once.

Before and after

Before

  1. A user complains that the app is slow.
  2. An engineer profiles once and guesses at the cause.
  3. They ship a fix and wait for field data.
  4. Nothing protects the win, so it decays.

After (as Anthropic describes it)

  1. The agent reads production telemetry and ranks journeys by impact.
  2. You add instrumentation until each journey has a comparable start and end.
  3. The agent estimates the impact of each candidate project in milliseconds. Those estimates set the targets; Anthropic hit twelve of its thirteen by day three.
  4. For fast iteration, pick a deterministic lab metric, such as instruction counts, render commits or style recalculations. The agent must prove it correlates with wall-clock time.
  5. The agent optimizes against that metric. Unit tests land first, and risky changes ship behind flags.
  6. A human approves each PR, and you roll out in stages.
  7. You check in a CI ratchet so the metric can only improve.
  8. You go looking for the next thing to measure.

Tech Bytes' account describes a per-thread version of the same loop. A human posts a recording, Claude reproduces the problem and opens review-sized PRs, then reads real-user data after deploy. If the metric improved, Claude ratchets it; if not, the flag goes off. The primary-source excerpt we reviewed cuts off before listing its loop steps, so treat this summary as secondary.

Try it

The smallest real step needs neither Claude Tag nor Anthropic's model. Pick one hot function, find a deterministic metric for it, confirm that moving the metric moves wall-clock time, and add a CI check that fails if the metric gets worse. Anthropic calls this the central lesson: "With Claude, measuring something makes it tractable."

If you use Datadog and Claude Code, Datadog's setup docs say to install the plugin from the official Anthropic marketplace:

/plugin install datadog@claude-plugins-official

Then run /ddsetup to pick your Datadog site and complete OAuth, and /ddtoolsets to choose tool groups. The optional code-exec toolset lets the agent run a single read-only JavaScript script against Datadog APIs. This helps with regression queries that would otherwise take many tool calls.

For Claude Tag, the Claude Help Center says setup is done by a Primary Owner or Owner. They provision Claude's identity, connect tools and repositories, and choose channels. Billing is consumption-based: tagging Claude in a channel is charged to the organization, with org-wide and per-channel spend caps, while direct messages are billed to your own Claude account. The sources give no pricing figures.

Where it breaks

Sources