Can a one-pass decision model make the calls a business makes — and know how sure it is?

The families follow the published System One use-case taxonomy: workflow control (inbox triage, intent routing, typed tool dispatch, confidence-gated actions), retrieval and knowledge (passage filtering, reranking, citation verification, entity alignment), safety and quality (guardrails, compliance, self-consistency, semantic linting),data and operations (hierarchical classification, structured extraction, composite scoring, tabular prediction) andreal-time and agents(tool and skill routing, real-time control, agent-run judging). Each task is a decision a buyer in its vertical would recognise as real work — a regulator's complaint desk, a contract-review queue, a clinical-trial screen, a moderation pipeline, a Kubernetes triage rota, an ICU vitals window.

Every system receives the identical compact-JSON state and the identical typed questions. A yes/no question is scored on P(yes); a choice on the returned distribution over its options; an ordinal score on the distribution over its levels. Items a system structurally cannot take (over its input limit or option cap) are recorded as unanswered, never as wrong — and they count as zero on the full-roster metric, so coverage is always visible.

Mean per-item proper score over the full roster, 0-100: per item, the mean over its gold questions of 1 - normalised Brier of the returned probabilities; an item a system could not answer scores 0.

Because it is a strictly proper scoring rule, a system cannot improve it by hedging or by over-claiming: the best strategy is to report its true belief. Beside it, every suite reports accuracy (top label), ROC-AUC on yes/no questions, log loss, Brier and expected calibration error (10 equal-width bins), and the answered count. A strict “pass” on the leaderboard is a top-label-correct item. Latency and cost are reported per system on this page but are not ranked: hosted APIs are measured from a laptop, open weights on a GPU pod, and the two boundaries are not comparable.

Open replicas publish their training registries; we read them at source. A dataset a system trained on (its train split, same task) is in-task; a synthetic clone of the same distribution is near; a source absent from a documented registry is held-out; a closed system with undisclosed training data is unknown. Fresh synthetic tasks generated here are held-out for every open system. The per-suite tables above show the tier next to each number, and the benchmark scorecard penalises in-task pairs.

Panels are stratified, seeded (20260919) and carry sampling weights; token counts use the Qwen/Qwen3-0.6B tokenizer, the tightest among the six systems. The generator, adapters, metrics and scorecard live in benchmark/typed_decision_bench; the frozen items are the suites directory the runner consumes. This is an independent project, not affiliated with or endorsed by any model vendor.