OpenAI's Decisions API: typed probabilities from gpt-6-luna, now in an LLM plugin

OpenAI's beta Decisions API returns probabilities instead of prose for classifiers and filters; Simon Willison's plugin makes it a one-command test.

Picture a two-person online shop selling refurbished kitchenware. Every morning the owner works through a returns inbox: customer emails, many with a photo of a chipped pan or a cracked lid. Today they paste each one into a chat model with a prompt like "reply with JSON: damaged true or false, and which department." It mostly works. But the answer is just a label. When they want to approve obvious cases automatically and send the borderline ones to a human, there's nothing to set a threshold on, and every call pays for output tokens. The Anthus team described the same problem from their own work: confidence read from OpenAI models' token log-probabilities was "so concentrated that they looked certain about nearly every answer."

On October 6, 2026, OpenAI put its Decisions API into public beta, according to OpenAI's Decisions guide. It takes text, images or both and returns a probability, a choice or a score instead of written text. The same day, Simon Willison released llm-openai-decisions 0.1a0, a plugin for his LLM command-line tool, so you can try it from a terminal before writing any integration code.

What it is

The API is a dedicated endpoint, POST /v1/decisions, and gpt-6-luna is the only model it accepts, per OpenAI's guide. A request has three parts: the model, the input (a text string, or user messages mixing text and images) and a list of named questions. Each question is one of three types:

On price, the guide says you pay only for input, at $0.10 per million input tokens, with no output, cache-read or cache-write charges; regional processing premiums and long-context multipliers still apply (CellCog, October 6, 2026). OpenAI claims it returns typed answers "about 10x faster than the Responses API"; at DevDay OpenAI cited 150 milliseconds against 1.6 seconds for a standard Luna call, The New Stack reported. The endpoint supports Zero Data Retention and HIPAA for eligible customers, with US and EU data residency.

Simon's plugin wraps all this for the LLM CLI. He notes that he had GPT-6 Astra read OpenAI's documentation and build the plugin, modelled on his existing llm-typesafe plugin for Jev.

The alternatives the sources describe:

Who it's for

Pick it up if you already have an OpenAI account, run high-volume classification, routing or eval gates, and some of your inputs are images. Image support is the clearest difference from Jev. The ZDR and HIPAA eligibility matters if you're handling customer data under contract.

Skip it if your traffic is text only and cost is the main lever: by CellCog's arithmetic, a million 500-token classifications cost about $50 on OpenAI against about $21 on Jev. Skip it if your data can't leave your network; this is a hosted API. And skip it for decisions that need several chained inference steps, for the reasons in Where it breaks.

Before and after

Before, for the returns inbox:

  1. Send the email text and photo to a chat model with a prompt asking for a JSON label.
  2. Parse the reply, and handle the cases where it isn't valid JSON.
  3. Treat every label as equally certain, because there's no usable confidence, so either a human checks everything or nothing.
  4. Pay for input and output on every call.

After:

  1. Send one request with the email and the photo as inline base64, plus a predicate question ("visible damage?") and a choice question (department, with an "other" fallback). Independent questions can share one request, per Unite.AI.
  2. Read typed answers keyed by the names you gave each question.
  3. Approve automatically above a threshold, send everything else to a person. OpenAI's advice is to set that threshold from your own labeled examples, weighing the cost of a false positive against a false negative.
  4. Pay for input only.

If a decision depends on an earlier answer (confirm damage, then ask for a repair category), that's a second request (MIXED).

Try it

The smallest step is Simon's plugin. From his release post:

llm install llm-openai-decisions

His example query against an image attachment:

llm -m openai-decisions/gpt-6-luna \
-a https://static.simonwillison.net/static/2025/two-pelicans.jpg \
-s 'Does this image contain any mammals?'

And the output he got:

{"type": "predicate", "name": "evaluation", "probability": 0.0}

He points to the plugin's README for choice and score questions. We couldn't see the README, so we can't confirm the minimum LLM version, how the plugin expects your OpenAI API key, or the syntax for the other two types. Note also that OpenAI's endpoint only takes inline base64 images, while Simon's example passes a hosted URL; the plugin appears to handle the encoding, but check the README.

To call the API directly, OpenAI's guide lists minimum SDK versions (Python 3.26.0, JavaScript 7.30.0, Go 3.73.0, Ruby 0.101.0, Java 4.78.0) and offers a Playground. Its routing example, as published:

curl https://api.openai.com/v1/decisions \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-6-luna",
"input": "I was charged twice for my order.",
"questions": [{
"type": "choice",
"name": "department",
"instructions": "Which department should handle this complaint?",
"choices": [
{"value": "billing", "description": "Payments, invoices, and refunds."},
{"value": "technical", "description": "Problems using the product."},
{"value": "shipping", "description": "Delivery and tracking."},
{"value": "other", "description": "Requests outside these categories."}
]
}]
}'

Before trusting any threshold, run a few hundred of your own labeled emails through it.

Where it breaks

The probabilities are not yet shown to be trustworthy. No independent accuracy, calibration or latency benchmark of the Decisions endpoint exists yet; CellCog says neither OpenAI nor TypeSafe has published head-to-head accuracy, and JevBench doesn't include it. The closest evidence is Anthus's test of plain GPT-6 Luna with reasoning off, run before the endpoint was documented: on 3,600 ProofWriter reasoning problems, when Luna said it was 99% sure or more it was right 68% of the time, and by five chained inferences it was at roughly a coin flip. Anthus is explicit that OpenAI uses a specialized version of Luna, so this is a warning, not a verdict. Whether Decisions probabilities are calibrated enough to threshold on, rather than raw scores, is open.

The speed claim is OpenAI's. MIXED notes the 10x figure comes with no benchmark attached.

It's a beta with one model. GA is promised for "the coming weeks," not a date; CellCog suggests planning for field names and limits to move. MIXED also found the gpt-6-luna model page's endpoint table doesn't list Decisions.

Input constraints. Images must be inline base64; hosted URLs and file IDs are refused. MIXED reports a cap of 128 image parts per request, user role only, and no files, audio or function calls. Any question can return a refusal instead of an answer, so handle that path.

Score lands between levels. A 1.1 isn't any level you defined; decide how your code maps it.

Unconfirmed costs and limits. The input token limit and context size for gpt-6-luna on this endpoint, and how image tokens are priced, weren't in the sources we could check.

The plugin is an alpha. Version 0.1a0, built by a model reading the docs. Good for trying the API, not something to put in a production pipeline unchanged.

Alternatives have their own gaps. Liquid's claims about d1's performance against Jev are its own, and DataNorth notes Liquid publishes no calibration figures, telling users to measure on their own traffic. That's the right advice for all three vendors.

Sources