Calibration & benchmarks

A probability is useful only if it is honest: of all the answers given with 80% confidence, about 80% should be right. Models vary a lot here:

• logprob and logit reads are usually close, and are often a little over-confident.
• verbalized reads (Claude, reasoning models) round to tidy numbers and are often over-confident.

polyjev measures this on your data and corrects it.

Labelled data

One JSON object per line: a state, a question and the right answer.

{"state": "Everything is down and we have a demo at noon.", "question": {"type": "noul", "instructions": "Does the customer need a reply within the hour?"}, "answer": "yes"}
{"state": "I was billed for 12 seats but we only have 9 users.", "question": {"type": "choice", "instructions": "Which team owns this ticket?", "criteria": {"billing": null, "outage": null, "feature": null}}, "answer": "billing"}
{"state": "UNACCEPTABLE. We are losing money every minute!!!", "question": {"type": "score", "instructions": "How upset is the customer?", "criteria": ["calm", "annoyed", "furious"]}, "answer": "furious"}

bench/data/sample.jsonl has 36 such items to start with. For real use, label 100–500 examples from your own traffic.

Compare models

polyjev bench --models vllm/local,anthropic/claude-opus-5,openai/gpt-5.6-luna \
  --data bench/data/sample.jsonl --markdown bench/results.md

metric	meaning (lower is better unless noted)
accuracy	share of top answers that are right (higher is better)
ECE	expected calibration error: the average gap between confidence and accuracy, over 10 confidence bins
Brier	mean squared error of the whole distribution
NLL	negative log-likelihood of the right answer
mean conf	average confidence; compare it with accuracy
ms / decision	wall time per decision

Paid APIs cost money, so the command asks before it starts (--yes skips that).

What to expect

polyjev-bench measured 7 open models on 1,777 questions from six public datasets (calibrating on one half, reporting on the other):

Read settings on the same model: shuffling options lowers raw calibration error, and verbalized confidence is both less accurate and slower than logprobs.

A real run on bench/data/sample.jsonl with Qwen3-8B on vLLM (one RTX 5000 Ada):

model	n	accuracy	ECE	Brier	NLL	mean conf	ms / decision
qwen3-8b	36	0.861	0.124	0.236	1.023	0.985	174

Mean confidence (0.985) is well above accuracy (0.861). The model's logprobs are over-confident, which is typical for instruction-tuned models. Its rare mistakes come with near-certain probabilities, which is what drives the high NLL.

Fit and use a profile

polyjev calibrate anthropic/claude-opus-5 --data labelled.jsonl --out calibration-claude.json

This fits one temperature per question type (temperature scaling, p_i ∝ p_i^(1/T)): T > 1 softens an over-confident model and T < 1 sharpens a timid one. Accuracy does not change; the probabilities become honest.

A question type keeps T = 1 when it has too few records (--min-records, default 20) or no wrong answers. With no mistakes there is no evidence which way the model is off, and the likelihood alone would sharpen it without limit. The command prints the reason for any type it leaves alone.

Use the profile in the config:

models:
  claude:
    provider: anthropic
    model: claude-opus-5
    calibration: calibration-claude.json

Decisions on that model then report diagnostics.calibrated: true.

From Python:

from polyjev.calibration import load_items, records_from, fit_profile, metrics

items = load_items("labelled.jsonl")
decisions = [judge.decide(it.state, {"q": it.question}) for it in items]
records = records_from(items, decisions)
print(metrics(records))
fit_profile("my-model", records).save("calibration.json")