Google's agent-eval playbook: replacing vibe checks with trace-based regression gates

A decode of Google's agent-eval playbook and open-source toolkit: CUJs, boolean judges and CI gates for developers shipping LangGraph-style agents.

Picture a two-person consultancy that built a LangGraph agent for a client's support desk. It looks up orders, checks a policy and issues refunds. Testing today means running four or five prompts in a terminal and reading the replies. Last week someone tweaked the system prompt to fix one edge case. The refund flow kept producing polite, correct-sounding answers, but the agent had quietly stopped calling the order-lookup tool first. Nobody noticed until a client asked why refunds were going out against the wrong orders.

That failure is the subject of Google's Developers Blog post "Evaluating AI agents: moving past the vibe check", published October 15, 2026, and flagged on X by @bibryam on October 24. In the post's words, an agent "can get stuck in a retry loop, fire off multiple tool calls, yet still return a polite, plausible-sounding final answer." The post is a playbook for a staged eval pipeline built around Google's open-source agent-eval toolkit. It is the companion to an AI Agent Clinic episode in which Google builds that pipeline for a LangGraph agent in 60 minutes.

What it is

The post is two things: a method, and a toolkit that runs it. Mapped to the consultancy's problem:

daily.dev's summary describes the same core: CUJs, simulated users, boolean rubrics, cost tracking and a Day 0 to Day 2 rollout.

Alternatives, all from sources other than Google:

Who it's for

Pick this up if you're shipping an agent that calls tools and you're already on, or open to, Google's stack. That's where the toolkit's judge and root-cause steps run. The method also suits anyone whose agent has broken after a prompt edit and who wants a staged plan rather than a platform purchase.

Skip the toolkit, but keep the method, if your data can't go to Google's evaluation service or your team already runs LangSmith, DeepEval or a self-hosted tracer. Most of the post's advice is framework-neutral, and the alternatives above cover the same layers.

Before and after

Before:

  1. Edit the system prompt.
  2. Run a few prompts in a terminal.
  3. Read the final answers. If they look reasonable, ship.
  4. Learn about the regression from a user.

After (Google's Day 0 to Day 2 rollout):

  1. Day 0: Write CUJs, break them into scenarios, run them through simulated users and read the first 5 to 20 traces "with a red pen." Mark where the agent hesitates, picks the wrong tool or rambles.
  2. Day 1: Turn those notes into deterministic assertions and boolean judge checks. Freeze the golden dataset and change one variable at a time. Google says reaching production quality "often takes 50 to 60 small iterations" of tweaks like clarifying a tool parameter description.
  3. Day 2: Lock the checks into pull-request merge gates. Export traces and scorecards to BigQuery or Cloud Storage. Promote failing production traces back into the golden dataset.

Where cheap checks stop and judges start is the key design choice. Developers Digest's guide puts it plainly: tool correctness, trajectory order, and arguments and budgets are "plain code" that cost nothing and give the same answer every time. The outcome judge "costs money, adds variance, and needs calibration." Run the deterministic layers on every case and save the LLM judge for what code can't read.

Try it

The smallest first step needs no tooling: write one CUJ for your agent, turn it into a handful of scenarios, run them and read the full traces, not just the final replies. That is Google's Day 0.

The Google post, as fetched, does not give install commands, a repo layout or library versions for agent-eval. Check the post and the toolkit's own repo before planning around it. The post also doesn't say which CI runner its gate uses.

For the Day 2 gate, here is one concrete pattern. It comes from SIGNAL DRIFT's practical evals guide, not from Google, and it uses GitHub Actions:

name: evals
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      # compare against the main-branch baseline, not an absolute score
      - run: python -m evals.run --set dev --baseline main --report dev.json
      - uses: actions/upload-artifact@v4
        with:
          name: eval-report
          path: dev.json

The evals.run module is the guide's own runner; you supply yours. If you're on LangGraph and want pytest-style gates today, DeepEval's integration page documents the callback setup.

Where it breaks

This is vendor content. A third-party review filed on GitHub calls the post first-party material that promotes agent-eval and Gemini's Agent Evaluation service. It advises treating the 50–60 iteration figure and the "e.g. 95%" pass threshold as illustrative, not measured. It also notes the boolean-judge recommendation comes with no measurements in the post. We found no independent hands-on test of the toolkit itself.

Gemini coupling is unclear. The toolkit's judge and root-cause steps run on Gemini's evaluation service. How well agent-eval works with other judge models is not confirmed by the sources.

Judges drift, and so do you. In a dbt Labs podcast interview, Shreya Shankar describes "criteria drift": in her "Who Validates the Validators?" study, developers' criteria changed as they saw more outputs. A rubric written on Day 0 will need revising. TestMu AI's eval-driven development guide adds that you should pin the judge model, prompt and dataset version, and rescore the baseline whenever the judge changes.

Judges can reward the wrong thing. TestMu cites AlpacaEval's recorded results: appending "Answer with as much detail as possible" to Mixtral 8x7B Instruct prompts raised the judged win rate from 18.26% to 24.61%, while the length-controlled win rate fell from 23.69% to 23.22%. Without a length or cost guard, a judge-only gate would have passed a change that mostly made answers longer.

Absolute thresholds have critics. Google's gate pairs a global pass bar with zero regressions on previously passing metrics, per the GitHub review. SIGNAL DRIFT argues against absolute bars: a 90% threshold "licensed 10% of your cases to fail silently." It recommends gating on regressions against the main branch, and rerunning unexpected failures because outputs vary between runs.

The alternatives have bugs too. An open agentevals issue reports that in "subset" and "unordered" trajectory modes, tool-argument matchers receive their arguments in swapped order. As a result, the same inputs score True or False depending on the mode. Whichever tool you pick, check that your matchers behave as documented before trusting a green build.

Sources