Blobfish ResearchTyped Decision Bench v0.3.0Results published

Can a one-pass decision model make the calls a business makes — and know how sure it is?

Typed Decision Bench measures System One models — models that take a state and typed questions and return calibrated probabilities in a single forward pass, with no text generation — on the decisions practitioners actually delegate to them: triage, routing, tool dispatch, passage filtering, citation checks, guardrails, compliance, extraction, scoring, control and run judging. One suite per use-case family, several industry verticals in each, three primitives (yes/no, choice, ordinal score). Every label is ground truth from a public dataset or a seeded generator; every answer is scored as a probability. No LLM judge.

Try Jev live Browse the 25 tasks Results Run report: every item, answer and verdict Per-task scores and samples Open source on GitHub Dataset on Hugging Face

Scoring Contamination tiers Licences and provenance Builders, metrics and scorecard (MIT)Leaderboard

5,387frozen items

25tasks in 5 suites

16industry verticals

3primitives: choice, noul, score

0LLM grading calls

Try the decision loop

Decide. Measure. Improve.

jev-1.13.0

What happened

Customer: I was charged twice for order A-104. Please refund the duplicate charge.

Questions sent to Jev

{
  "refund": {
    "type": "noul",
    "instructions": "Does the customer ask for a refund?"
  },
  "team": {
    "type": "choice",
    "instructions": "Which team should handle this?",
    "criteria": {
      "billing": "Charges, invoices, refunds",
      "shipping": "Delivery problems",
      "other": null
    }
  },
  "anger": {
    "type": "score",
    "instructions": "How frustrated is the customer?",
    "criteria": [
      "calm",
      "frustrated",
      "very frustrated"
    ]
  }
}

Awaiting your call

Run the request to see typed answers and probabilities from Jev.

What it measures

One suite per use-case family, the verticals inside it

The families follow the published System One use-case taxonomy: workflow control (inbox triage, intent routing, typed tool dispatch, confidence-gated actions), retrieval and knowledge (passage filtering, reranking, citation verification, entity alignment), safety and quality (guardrails, compliance, self-consistency, semantic linting),data and operations (hierarchical classification, structured extraction, composite scoring, tabular prediction) andreal-time and agents (tool and skill routing, real-time control, agent-run judging). Each task is a decision a buyer in its vertical would recognise as real work — a regulator's complaint desk, a contract-review queue, a clinical-trial screen, a moderation pipeline, a Kubernetes triage rota, an ICU vitals window.

Every system receives the identical compact-JSON state and the identical typed questions. A yes/no question is scored on P(yes); a choice on the returned distribution over its options; an ordinal score on the distribution over its levels. Items a system structurally cannot take (over its input limit or option cap) are recorded as unanswered, never as wrong — and they count as zero on the full-roster metric, so coverage is always visible.

Suites and tasks

25 tasks, 5,387 items

Data and operations 981 items · 4 tasks

Task	Use case	Vertical	Primitive	Items	Options	Max state tokens	Labels	Source
`arxiv-archive-subcategory`	Hierarchical classification	science publishing	choice	200	11	264	logged outcome	gfissore/arxiv-abstracts-2021
`dbpedia-ontology-hierarchy`	Hierarchical classification	media and reference	choice	200	9	405 long tier	logged outcome	DeveloperOats/DBPedia_Classes
`finer-xbrl-fact-tag`	Structured extraction	finance	choice	181	28	251	expert labels	nlpaueb/finer-139
`trial-outcomes`	Feature extraction for ML	saas growth	noul	400	—	282	generator oracle	generated here (seeded oracle)

Real-time and agents 1,199 items · 6 tasks

Task	Use case	Vertical	Primitive	Items	Options	Max state tokens	Labels	Source
`bfcl-tool-dispatch`	Agent routing and skill selection	software engineering	choice	200	8	306 long tier	expert labels	minpeter/bfcl-v1-non-live-ast-parsed
`k8s-issue-kind-routing`	Agent routing and skill selection	devops	choice	200	7	429 long tier	logged outcome	open-index/open-github-issues
`swe-rebench-run-resolved`	Agent-run judging	software engineering	noul	200	—	2170 long tier	executed verifier	nebius/SWE-rebench-openhands-trajectories
`swe-smith-run-resolved`	Agent-run judging	software engineering	noul	200	—	1287 long tier	executed verifier	SWE-bench/SWE-smith-trajectories
`tau2-airline-run-success`	Agent-run judging	customer operations	noul	200	—	473 long tier	executed verifier	snorkelai/Tau2-Bench-Verified-Airline-With-Code-Agents
`tau2-airline-transcript-success`	Agent-run judging	customer operations	noul	199	—	3936 long tier	executed verifier	snorkelai/Tau2-Bench-Verified-Airline-With-Code-Agents

Retrieval and knowledge 1,249 items · 5 tasks

Task	Use case	Vertical	Primitive	Items	Options	Max state tokens	Labels	Source
`abt-buy-product-match`	Entity alignment	e commerce	score	200	3	259	expert labels	matchbench/Abt-Buy
`casehold-cited-holding`	Citation verification	legal	choice	200	5	437 long tier	logged outcome	coastalcph/lex_glue
`esci-product-rerank`	Semantic reranking	e commerce	noul	490	—	421 long tier	human annotation	tasksource/esci
`hotpot-context-filter`	RAG passage filtering	knowledge work	noul	200	—	1403 long tier	human annotation	hotpotqa/hotpot_qa
`trialgpt-criterion-support`	Citation verification	healthcare	choice	159	4	297	expert labels	ncbi/TrialGPT-Criterion-Annotations

Safety and quality 1,162 items · 6 tasks

Task	Use case	Vertical	Primitive	Items	Options	Max state tokens	Labels	Source
`cuad-checklist`	Compliance verification	legal	noul	204	—	288	expert labels	nguha/legalbench
`humanevalpack-spec-match`	Semantic linting	software engineering	noul	164	—	448 long tier	executed verifier	bigcode/humanevalpack
`moderation-hate-severity`	LLM guardrails	trust and safety	score	200	3	366 long tier	expert labels	mmathys/openai-moderation-api-evaluation
`prompt-injection`	LLM guardrails	security	noul	194	—	221	human annotation	deepset/prompt-injections
`unfair-tos-clause-type`	Compliance verification	consumer legal	choice	200	9	174	expert labels	nguha/legalbench
`vitaminc-claim-consistency`	Self-consistency checks	media	choice	200	3	183	human annotation	tals/vitaminc

Workflow control 796 items · 4 tasks

Task	Use case	Vertical	Primitive	Items	Options	Max state tokens	Labels	Source
`cfpb-complaint-product-queue`	Support inbox triage	finance	choice	199	7	292	logged outcome	liri-uzh/cfpb-complaints-mini
`hwu64-tool-select`	Typed tool dispatch	consumer assistant	choice	200	64	27	human annotation	deutsche-telekom/NLU-Evaluation-Data-en-de
`ledgar-clause-review-routing`	Intent and model routing	legal	choice	199	64	274	logged outcome	coastalcph/lex_glue
`unfair-tos-publish-gate`	Confidence-gated actions	consumer legal	noul	198	—	161	expert labels	coastalcph/lex_glue

Scoring

DecisionScore: a proper score on the probability, not just the argmax

Mean per-item proper score over the full roster, 0-100: per item, the mean over its gold questions of 1 - normalised Brier of the returned probabilities; an item a system could not answer scores 0.

Because it is a strictly proper scoring rule, a system cannot improve it by hedging or by over-claiming: the best strategy is to report its true belief. Beside it, every suite reports accuracy (top label), ROC-AUC on yes/no questions, log loss, Brier and expected calibration error (10 equal-width bins), and the answered count. A strict “pass” on the leaderboard is a top-label-correct item. Latency and cost are reported per system on this page but are not ranked: hosted APIs are measured from a laptop, open weights on a GPU pod, and the two boundaries are not comparable.

Results

6 systems on the same 5,387 items

#	System	DecisionScore	Accuracy	Answered	Truncated	Held-out suites	Weights
1	Jev `jev-1.13.0` Closed weights; training data undisclosed, so every contamination tier is unknown. Probabilities are returned rounded to 2 decimals.	81.7	76.8%	5,387 / 5,387	0	0.0%	proprietary API (closed weights)
2	JevFish `jevfish-v4-r4b` First-party model published by the same team as this benchmark. Full fine-tune of Mapika/decider-2b at lr 1e-6 with a KL anchor to the starting model and a floor-gated early stop, on verifier- and human-labelled public agent runs (no labels distilled from a closed vendor). It replaces round 3 (75.82), which scored below its own base. Against decider-2b as published it is +0.69 [+0.39, +0.98]; against the SAME decider-2b calibrated with the same procedure (78.41, not a published row) it is a statistical tie, -0.17 [-0.43, +0.08]. In-task on the SWE-smith and SWE-rebench corpora it trained on (161 of the 200 SWE-smith items share a task instruction with training rows, different agent runs). Measured effect: drop both SWE tasks (4,987 items) and JevFish scores 78.94 (decider-2b 78.20, System One Scorer 77.61, OpenJev 77.20, Jev 82.84) - its rank does not change. Known limits: JSON-object states rank worse than the same content as text, and on code runs it is pass-leaning at p=0.5. Per-question-type calibration temperatures: noul 1.15, ordinal 1.09, choice 3-5 1.30, choice 11+ 1.32.	78.2	70.1%	5,387 / 5,387	0	80.0%	code MIT; weights not released
3	decider-2b `decider-2b-v8` In-task on several public sources it trained on (see contamination tiers); latency is local MPS and not comparable with hosted APIs.	77.6	68.8%	5,387 / 5,387	0	100.0%	Apache-2.0
4	System One Scorer `s1-scorer-4b-v2b` Scores one sequence per option with a 384-token limit; longer states are cut from their end (flagged as truncated).	77.2	67.6%	5,387 / 5,387	1571	80.0%	CC-BY-NC-4.0 (weights; evaluation only)
5	OpenJev `openjev-latest` Zero-shot open replica served by a third party; no task training.	75.6	73.7%	5,387 / 5,387	0	0.0%	Apache-2.0 (code)
6	NanoJev `nanojev-0.6b` A nano replica trained on maze/navigation decisions; items over its 512-token limit are rejected and count as unanswered.	56.8	43.7%	4,582 / 5,387	0	100.0%	code MIT; the weights card declares no licence

Data and operations — per-system detail

System	DecisionScore	Accuracy	AUROC	ECE	Log loss	Brier	Answered	Contamination
Jev	83.9	81.9%	0.716	0.064	0.590	0.201	981 / 981	unknown
JevFish	78.7	77.5%	0.626	0.101	0.325	0.092	981 / 981	held-out
decider-2b	78.9	75.8%	0.629	0.051	0.335	0.095	981 / 981	held-out
System One Scorer	77.9	77.5%	0.571	0.256	0.508	0.160	981 / 981	held-out
OpenJev	76.3	80.4%	0.544	0.082	0.418	0.105	981 / 981	unknown
NanoJev	64.2	40.1%	0.498	0.234	0.353	0.098	981 / 981	held-out

Real-time and agents — per-system detail

System	DecisionScore	Accuracy	AUROC	ECE	Log loss	Brier	Answered	Contamination
Jev	75.8	65.5%	0.599	0.109	0.774	0.274	1199 / 1199	unknown
JevFish	73.0	58.7%	0.459	0.087	0.878	0.321	1199 / 1199	in-task
decider-2b	73.0	55.3%	0.367	0.112	1.073	0.392	1199 / 1199	held-out
System One Scorer	74.8	56.9%	0.468	0.079	0.784	0.287	1199 / 1199	held-out
OpenJev	66.6	61.4%	0.480	0.226	3.038	0.460	1199 / 1199	unknown
NanoJev	33.3	43.1%	0.480	0.105	0.800	0.293	598 / 1199	held-out

Retrieval and knowledge — per-system detail

System	DecisionScore	Accuracy	AUROC	ECE	Log loss	Brier	Answered	Contamination
Jev	83.4	79.6%	0.877	0.042	0.517	0.160	1249 / 1249	unknown
JevFish	79.8	72.1%	0.814	0.147	0.560	0.192	1249 / 1249	held-out
decider-2b	78.3	70.9%	0.783	0.123	0.582	0.200	1249 / 1249	held-out
System One Scorer	75.5	62.5%	0.652	0.039	0.683	0.243	1249 / 1249	near
OpenJev	76.5	75.4%	0.845	0.149	0.976	0.189	1249 / 1249	unknown
NanoJev	57.1	48.8%	0.542	0.130	0.711	0.256	1046 / 1249	held-out

Safety and quality — per-system detail

System	DecisionScore	Accuracy	AUROC	ECE	Log loss	Brier	Answered	Contamination
Jev	88.9	85.9%	0.983	0.095	0.323	0.100	1162 / 1162	unknown
JevFish	84.6	77.7%	0.962	0.054	0.306	0.088	1162 / 1162	held-out
decider-2b	83.5	77.1%	0.947	0.085	0.289	0.077	1162 / 1162	held-out
System One Scorer	82.9	77.5%	0.971	0.079	0.347	0.096	1162 / 1162	held-out
OpenJev	85.9	82.5%	0.972	0.160	0.638	0.103	1162 / 1162	unknown
NanoJev	69.2	45.6%	0.571	0.304	0.737	0.270	1161 / 1162	held-out

Workflow control — per-system detail

System	DecisionScore	Accuracy	AUROC	ECE	Log loss	Brier	Answered	Contamination
Jev	74.7	70.1%	0.906	0.209	0.546	0.169	796 / 796	unknown
JevFish	73.8	64.2%	0.859	0.107	0.320	0.097	796 / 796	held-out
decider-2b	72.9	65.0%	0.845	0.172	0.339	0.106	796 / 796	held-out
System One Scorer	74.7	65.2%	0.895	0.299	0.542	0.178	796 / 796	held-out
OpenJev	71.8	68.5%	0.902	0.257	1.010	0.206	796 / 796	unknown
NanoJev	64.5	39.1%	0.593	0.093	0.625	0.216	796 / 796	held-out

Fairness

Every system × dataset carries a contamination tier

Open replicas publish their training registries; we read them at source. A dataset a system trained on (its train split, same task) is in-task; a synthetic clone of the same distribution is near; a source absent from a documented registry is held-out; a closed system with undisclosed training data is unknown. Fresh synthetic tasks generated here are held-out for every open system. The per-suite tables above show the tier next to each number, and the benchmark scorecard penalises in-task pairs.

Task	jev	openjev	decider-2b	s1-scorer	nanojev	jevfish
`arxiv-archive-subcategory`	unknown	unknown	held-out	held-out	held-out	held-out
`dbpedia-ontology-hierarchy`	unknown	unknown	held-out	held-out	held-out	held-out
`finer-xbrl-fact-tag`	unknown	unknown	held-out	held-out	held-out	held-out
`trial-outcomes`	unknown	unknown	held-out	held-out	held-out	held-out
`bfcl-tool-dispatch`	unknown	unknown	held-out	held-out	held-out	held-out
`k8s-issue-kind-routing`	unknown	unknown	held-out	held-out	held-out	held-out
`swe-rebench-run-resolved`	unknown	unknown	held-out	held-out	held-out	in-task
`swe-smith-run-resolved`	unknown	unknown	held-out	held-out	held-out	in-task
`tau2-airline-run-success`	unknown	unknown	held-out	held-out	held-out	held-out
`tau2-airline-transcript-success`	unknown	unknown	held-out	held-out	held-out	held-out
`abt-buy-product-match`	unknown	unknown	held-out	held-out	held-out	held-out
`casehold-cited-holding`	unknown	unknown	held-out	held-out	held-out	held-out
`esci-product-rerank`	unknown	unknown	held-out	held-out	held-out	held-out
`hotpot-context-filter`	unknown	unknown	held-out	near	held-out	held-out
`trialgpt-criterion-support`	unknown	unknown	held-out	held-out	held-out	held-out
`cuad-checklist`	unknown	unknown	held-out	held-out	held-out	held-out
`humanevalpack-spec-match`	unknown	unknown	held-out	held-out	held-out	held-out
`moderation-hate-severity`	unknown	unknown	held-out	held-out	held-out	held-out
`prompt-injection`	unknown	unknown	held-out	held-out	held-out	held-out
`unfair-tos-clause-type`	unknown	unknown	held-out	held-out	held-out	held-out
`vitaminc-claim-consistency`	unknown	unknown	held-out	held-out	held-out	held-out
`cfpb-complaint-product-queue`	unknown	unknown	held-out	held-out	held-out	held-out
`hwu64-tool-select`	unknown	unknown	held-out	held-out	held-out	held-out
`ledgar-clause-review-routing`	unknown	unknown	held-out	held-out	held-out	held-out
`unfair-tos-publish-gate`	unknown	unknown	held-out	held-out	held-out	held-out

Benchmark quality

The benchmark is scored too

Axis	Score	Bar
realism	0.701	0.70	clears
diversity	0.985	0.70	clears
complexity	0.705	0.60	clears
verifiability	1.000	0.95	clears
fairness	0.817	0.60	clears
headroom	0.800	0.60	clears
power	0.779	0.70	clears

Weakest axis: none — every scored axis clears its bar

Realism weighs operational records over academic sentences and verifier or outcome labels over crowd labels; diversity counts families, verticals, primitives and datasets and rejects near-duplicate states; complexity rewards abstention options, multi-question items, rare events and a long tier; verifiability requires ground-truth labels inside the option set; fairness is the held-out share; headroom is measured on the ranked systems; power is the expected confidence-interval width at each task's size. Each release is scored before it is published and the weakest axis is the next change.

Provenance

Licences, sources, reproducibility

Dataset	Licence	Tasks
gfissore/arxiv-abstracts-2021	CC0-1.0	arxiv-archive-subcategory
DeveloperOats/DBPedia_Classes	CC0-1.0	dbpedia-ontology-hierarchy
nlpaueb/finer-139	CC-BY-SA-4.0	finer-xbrl-fact-tag
synthetic:saas-trial-v1 (generated here)	CC-BY-4.0 (generated here)	trial-outcomes
minpeter/bfcl-v1-non-live-ast-parsed	Apache-2.0	bfcl-tool-dispatch
open-index/open-github-issues	ODC-BY-1.0	k8s-issue-kind-routing
nebius/SWE-rebench-openhands-trajectories	CC-BY-4.0	swe-rebench-run-resolved
SWE-bench/SWE-smith-trajectories	MIT	swe-smith-run-resolved
snorkelai/Tau2-Bench-Verified-Airline-With-Code-Agents	Apache-2.0	tau2-airline-run-success, tau2-airline-transcript-success
matchbench/Abt-Buy	CC (variant unconfirmed on the source page) — FLAGGED	abt-buy-product-match
coastalcph/lex_glue	CC-BY-4.0	casehold-cited-holding, ledgar-clause-review-routing, unfair-tos-publish-gate
tasksource/esci	Apache-2.0	esci-product-rerank
hotpotqa/hotpot_qa	CC-BY-SA-4.0	hotpot-context-filter
ncbi/TrialGPT-Criterion-Annotations	Public domain (NCBI)	trialgpt-criterion-support
nguha/legalbench	CC-BY-4.0	cuad-checklist, unfair-tos-clause-type
bigcode/humanevalpack	MIT	humanevalpack-spec-match
mmathys/openai-moderation-api-evaluation	MIT	moderation-hate-severity
deepset/prompt-injections	Apache-2.0	prompt-injection
tals/vitaminc	CC-BY-SA-3.0	vitaminc-claim-consistency
liri-uzh/cfpb-complaints-mini	CC0-1.0 (CFPB public domain)	cfpb-complaint-product-queue
deutsche-telekom/NLU-Evaluation-Data-en-de	CC-BY-4.0	hwu64-tool-select

Panels are stratified, seeded (20260919) and carry sampling weights; token counts use the Qwen/Qwen3-0.6B tokenizer, the tightest among the six systems. This is an independent project, not affiliated with or endorsed by any model vendor.

Everything here is open. The generator, adapters, metrics, contamination registry and quality scorecard are MIT-licensed at github.com/blobfishai/jevfish, next to the trainer and server of the first-party model ranked above. The frozen items, each with the licence of its source, and every system's answer to every item are on Hugging Face at SamuelChien821/typed-decision-bench. Two tasks whose sources state no clear redistribution licence (Abt-Buy, TrialGPT) ship there as item ids only; python -m typed_decision_bench.freeze --seed 20260919 rebuilds them, and SHA256SUMS checks the rebuild. The first-party model's weights are not released yet.