Claude Code's build-eval and hillclimb: eval design with a held-out check

Two claude-api skill commands in Claude Code build an eval from real cases, then tune your app against it with a held-out set. For Claude API builders.

Picture a two-person team shipping an email-routing feature on the Claude API inside a small support product. Today, "testing" means one of them edits the system prompt, pastes in the five emails they remember failing last week, eyeballs the output and merges. Sometimes the fix holds. Often it quietly breaks a case nobody re-checked. A model swap they tried last month got reverted on a hunch, because nobody could say whether it was better or just different. They have no grader they trust, no baseline, and no way to tell a real gain from run-to-run noise.

On September 28, 2026, Anthropic published Automating eval design and hillclimbing with Claude. The post adds two commands to the claude-api skill in Claude Code. /claude-api build-eval builds an evaluation inside your codebase. /claude-api hillclimb improves your application against that eval "one change at a time, with a held-out set of examples to catch overfitting." That targets the step between vibe-coded prompt tweaks and changes you can defend with a number.

What it is

Anthropic says the release is two things: written principles for eval design and hillclimbing, and two guided commands that apply them. Here is what each piece does, mapped to the team's pain.

No trustworthy test set → build-eval samples real cases first. Anthropic's guide warns against choosing cases only because today's model fails them. Model capability is "jagged," so that approach measures one model's "failure fingerprint." It also warns that raw user traffic may skew easy. According to Superpower Daily and The Neuron, the command draws on sources in this order:

  1. Production transcripts, after it asks about data retention and sensitive information.
  2. Bug reports and support tickets.
  3. Five to ten cases you write yourself.
  4. Synthetic cases generated from the codebase.

Claude builds a simple local page listing every input and waits for you to confirm them.

Eyeballed grading → a validated grader. Claude proposes "the cheapest grader that fits" your output. explainx.ai lays out the ladder:

Claude then grades a handful of cases and asks whether you would have scored any of them differently. Per Superpower Daily, it also checks whether the grader gives the same verdict when it sees the same answer twice, and whether errors or timeouts are distorting the baseline.

No baseline → a scored run with error bars. Before running, the skill tells you the size of the run (cases × repeats × models) and roughly how long it will take. It then runs a baseline and prints the score with a confidence interval. You get back:

Hunch-driven changes → one patch per round, checked against held-out cases. Hillclimb first asks what you want to optimize, such as performance, or lower cost while performance holds. It then splits the eval at random into train and test sets and checks that the eval's noise is smaller than the smallest improvement you'd act on. Superpower Daily lists the surfaces you can let it edit: prompts, instruction files, tool descriptions, model settings and application code.

Each round, Claude reads the train transcripts and proposes one patch that aims at the root cause rather than rewording a line. The rules for keeping it are simple:

When the score stalls for two or three rounds, Claude sorts the remaining failures by cause, which can surface ambiguous cases or harness bugs. It finishes on the version that did best on the test set and recommends against merging if the gain is within noise.

Alternatives. We did not research competing eval tools in this cycle, so we can't give sourced trade-offs. The one sourced alternative is do-it-yourself. One team planning to adopt the method notes on GitHub that "only the method carries over" when their runtime model isn't Anthropic's, and they may run the same loop through their own harness instead.

Who it's for

Pick this up if you have an app, agent or skill that calls the Claude API, and you already hold real failure material: bug reports, tickets or traces. One public example is a repo planning a gate eval built from recorded P0/P1 defects, each with its pre-fix diff. It's also a fit if you want to cut cost without losing quality. Anthropic's worked example was exactly a cost-plus-accuracy climb.

Skip it (for now) if any of these apply:

Before and after

Before

  1. Remember a few recent failures.
  2. Edit the prompt, or swap the model.
  3. Rerun those few cases and eyeball the output.
  4. Merge if it "looks better."
  5. Find out in production whether it was.

After (as Anthropic describes it)

  1. Run /claude-api build-eval. Claude interviews you and proposes inputs from transcripts, tickets, your own cases or synthetics. You approve them on a local page.
  2. Approve the grader after reviewing a few cases it scored.
  3. Get a baseline score with a confidence interval, plus per-case transcripts.
  4. Run /claude-api hillclimb with a goal and a set of allowed edit surfaces. Claude checks noise first, then patches one change per round, keeping or reverting each based on train and test scores.
  5. Receive a test-set result against the baseline, and a recommendation not to merge if the gain is within noise.

Anthropic's own run shows the shape. On an internal support benchmark of 44 tickets (30 for search, 14 held out), it started with Opus 4.8 at high effort: 74.4% accuracy on the search tickets at 4.6 cents per ticket. From there:

  1. The climb audited the prompt, removing mandatory tool calls, a scratchpad step and contradictory rules.
  2. It moved to Opus 5.5 at low effort: 87.8% at 1.9 cents.
  3. It stepped down to Sonnet 5 at low effort: 88.9% at about 1 cent.
  4. Prompt changes brought Sonnet 5 to 98.9% on the search tickets.

On the 14 held-out tickets, the guide reports 90.5% against the original setup's 78.6%, at about one fifth of the cost. KuCoin relayed the same held-out figures. Separately, The Neuron and explainx.ai report that Anthropic's hillclimb on the claude-api skill itself went from about 66% to 88%. explainx.ai says that took 24 rounds.

Try it

The sources describe running both commands directly in Claude Code through the claude-api skill. You can steer build-eval by giving it access to examples such as traces. If a command is missing, a community command reference says to type / to see what your version and plan include, and to run claude update.

The sources give no minimum version and no pricing for the skill itself. Your cost is the API usage of the eval runs, which none of the sources quantify for a small project.

There is a sensible first move, suggested in a post on X: ask Claude to estimate API fees before running anything, and run hillclimb on a new branch. The post's sample prompt ends:

"Show me the report first. Don't merge anything until I confirm."

The GitHub gate-eval plan adds the same guardrails from a working repo: get the budget approved first, measure the noise floor before hillclimbing, and never paste failures into prompts.

Where it breaks

Sources