AutoRubric: a Python library for rubric-based LLM judges, with a cheap Jev first pass

AutoRubric is an open-source Python library for rubric-based LLM judging, for builders who eval open-ended output and want rigor without a homemade harness.

Picture a two-person team at a small B2B SaaS company shipping an AI support assistant. After every prompt change, they run 200 recent tickets through the bot and grade the replies with a script. It asks a model three questions: does this reply resolve the issue, is it polite, does it invent policy? That works until it doesn't. Someone swaps the judge model and the scores shift. Nobody knows whether the judge agrees with what a human would say. The bill climbs with every criterion they add. And when an API key expires, the script quietly grades less than it should. They built their evals the way they built the bot, by vibe.

AutoRubric is aimed at that homemade harness. On October 8, 2026, Delip Rao presented the AutoRubric paper at COLM 2026. In that thread he argues that rubric judges "can silently go wrong in multiple ways" and that teams pay a "reinvention tax" rebuilding evaluation pipelines. The library's cookbook packages the fixes as recipes. It also offers a cheap first pass: a decision model grades everything, and only the criteria it is unsure about go to an LLM.

What it is

AutoRubric is a Python library for grading text outputs against a rubric of weighted criteria. A GitHub issue on the repo shows the shape: a Rubric is a list of Criterion objects, each with a name, a weight (negative for penalties such as "States a factual error") and a plain-language requirement. Here is how each capability maps to the team's problems, according to the cookbook:

Rao also claims multi-vendor support, artifact versioning, reports and budget tracking in his thread. Treat those as the author's description.

Alternatives: our sources don't compare AutoRubric with other eval frameworks, so we won't invent trade-offs. What they do compare is judge setups you can run inside AutoRubric:

Who it's for

Pick it up if you write Python 3.11+ and grade open-ended output: support replies, summaries, RAG answers, agent skills. It suits you best if you already run LLM-judge evals and want ensembles, calibration and human-agreement checks without writing them yourself. It fits even better if you have some human labels to validate against, or are willing to create them.

Skip it if your output is verifiable by tests. Rao frames AutoRubric around non-verifiable tasks. Skip the cheap Jev path if your criteria need the judge to derive a verdict rather than read it off the text. A September 28 arXiv paper on Jev as a judge found it trails GPT-6 by 7 to 28 points on expert knowledge, code, math and logic puzzles. Skip that path too if you can't add another vendor key: decision-model recipes need a TypeSafe API key.

Before and after

Before:

  1. Write one prompt that asks a model to score each reply.
  2. Parse the scores, average them, eyeball a few outliers.
  3. Change the judge model or the prompt and hope the numbers still mean the same thing.
  4. Notice cost or failures only when the run ends, or never.

After, following the cookbook's structure:

  1. Define a rubric as weighted criteria, with penalties as negative weights.
  2. Pick a judge kind. Use an LLM judge per criterion when you need explanations, llm_calls="per_item" to cut calls, or a decision model for probabilities.
  3. Run the Judge Validation recipe to measure agreement with human labels before trusting the scores.
  4. Add few-shot calibration, an ensemble or fixed seeds where agreement is weak or runs aren't reproducible.
  5. For large batches, use checkpointing and cost tracking. Optionally put Jev first and send uncertain criteria to an LLM, after validating the threshold on your own labelled data.

Try it

The cookbook's prerequisites are Python 3.11+, pip install autorubric, an API key for at least one supported LLM provider, and basic familiarity with async/await, since recipes use asyncio.run(). Decision-model recipes also need the TypeSafe extra and a TypeSafe key. The exact command for that extra isn't in the page text we have. The cookbook says to start with the "Your First Evaluation" recipe, where each recipe ends with a complete runnable script. We didn't have that code or a package version.

The imports and rubric definition below are copied from issue #40's reproduction, written against commit 0d5d953. It is an excerpt, not a complete script:

from autorubric import Criterion, LLMConfig, Rubric
from autorubric.graders import CriterionGrader, JudgeSpec

# Weighted criteria; a negative weight acts as a penalty
rubric = Rubric([
    Criterion(name="cites", weight=1.0, requirement="Cites a source"),
    Criterion(name="wrong", weight=-1.0, requirement="States a factual error"),
])

In that reproduction, grading is an awaited call: await rubric.grade(to_grade=..., grader=CriterionGrader(judge_model_config=...)). The sources don't state a price for the library itself. You pay your LLM provider, and TypeSafe for Jev.

Where it breaks

Silent abstention on misconfiguration. Issue #40, filed September 27, 2026, reports a specific bug. A misconfigured LLM judge (unsupported parameter, bad API key, wrong model name) doesn't raise an error. Instead it is filed as an "infrastructure" failure and abstains. With a single judge, every item fails, but you only see it after the whole run. With a panel, the bad judge contributes nothing, and the run can finish "successfully" with no error in the report. The reporter hit this with LiteLLM 1.95.0 and openai/gpt-6-luna with thinking="low". The workaround given is extra_params={"allowed_openai_params": ["reasoning_effort"]}. Decision-model configuration errors, by contrast, fail when the grader is built. Until this is resolved, check judge_scores for None entries.

Cascades save money more than they add accuracy. The rubric-judge comparison found the LLM judges repeat nearly all of Jev's most confident errors. Replayed on recorded verdicts, a cascade gained at most 1.5 points over the best single judge with cross-fitted thresholds, and 2.0 with oracle thresholds. The Jev-as-a-Judge paper is more favourable. It reports a frozen-threshold cascade 0.9 points more accurate than GPT-6 on 1,610 held-out pairs, at 41% of its fee. That paper also says confidence routing weakens on style-adversarial pairs and reference-free prose, and it recommends validating thresholds locally. The two papers test different setups, so neither result transfers to your rubric without checking.

Judges can agree with each other and still be wrong. On graded criteria, the comparison paper found all four judges agreed more with one another than with human labels, and mostly assigned lower levels than raters did. Per Redreamality, a constant predictor on HelpSteer2 scored 60.0%, above every judge. An ensemble's consensus is not validation. Human labels are.

Vendor and independence. The cookbook's decision-model path is built around TypeSafe's Jev. Per Redreamality, the rubric comparison's authors are Delip Rao and Chris Callison-Burch, and that paper cites AutoRubric as their own work. Read it as the authors evaluating options in their own harness, not as an independent audit.

Unconfirmed:

Sources