Three changed prompt words broke two of four agent evals while dashboards stayed green

CloudAura shows a three-word system-prompt edit flipping two of four deterministic evals to FAIL with HTTP 200s and flat latency, and proposes a fix.

A three-word edit to an agent's system prompt flipped two of its four deterministic evaluators from PASS to FAIL. The agent kept answering every question, latency stayed flat, and none of the SRE dashboards would have caught it, according to CloudAura's write-up (May 19, 2026), which @bibryam flagged on X on October 9. If you run an agent in production, HTTP 200s are not telling you whether its behavior changed.

The experiment was small. On kagent with Solo.io's agentevals as the eval backend, CloudAura ran a minimal mistral-small agent told to answer in exactly two sentences and end with kthxbye. Version 2 swapped terse for verbose, two for three and kthxbye for goodbye now. The two shape-specific checks failed. The two generic checks, length range and expected substrings, stayed green. No LLM judge was involved.

CloudAura proposes two artifacts: a Profile, a repeated series of deterministic evals against a baseline config, and a Fingerprint, a content-addressed hash of that config. The Fingerprint is layered so vendor-side noise doesn't invalidate the Profile on every reconcile, and a change triggers re-evaluation. This is a proposal built on a lab prototype. We couldn't find public code for its AgentProfile CRD.

You can apply the pattern without it. A separate author built kagent-agentevals, a golden-suite runner that exits non-zero when an agent's tool-calling behavior changes. Outside Kubernetes, promptfoo runs evals from a declarative config on the CLI or in CI. Luca Berton's tutorial pairs deterministic checks with an LLM-judge rubric to block pull requests.

Sources