Skip to content
Blobfish AI
BenchmarksTalk to usOpen app

Typed decisions

Typed Decision Bench

One suite per System One use-case family (workflow control, retrieval and knowledge, safety and quality, data and operations, real-time and agents) across 16 industry verticals: yes/no, choice and ordinal-score decisions on public labelled data and seeded generators, scored as calibrated probabilities.

Source benchmark Methodology

Typed decisions

Model performance

6 recorded configurations · 5,387 source tasks

Updated
Higher is better · 0–100 · execution errors score 0
J
Jevjev-1.13.0
81.7 DecisionScore4138 / 5387 strict passes · 5387 graded · 0 execution errors
Full benchmark5387 tasks attempted
View run
Run details
Evaluation
Typed Decision Bench · v0.3.0
Environment
System One wire format · 6 systems
Grader
Ground-truth labels
Verifier
typed-decision-bench:0.3.0
Coverage
5387 of 5387 source tasks attempted; 5387 graded.
Open task evidence
J
JevFishjevfish-v4-r4b
78.2 DecisionScore3778 / 5387 strict passes · 5387 graded · 0 execution errors
Full benchmark5387 tasks attempted
View run
Run details
Evaluation
Typed Decision Bench · v0.3.0
Environment
System One wire format · 6 systems
Grader
Ground-truth labels
Verifier
typed-decision-bench:0.3.0
Coverage
5387 of 5387 source tasks attempted; 5387 graded.
Open task evidence
d
decider-2bdecider-2b-v8
77.6 DecisionScore3706 / 5387 strict passes · 5387 graded · 0 execution errors
Full benchmark5387 tasks attempted
View run
Run details
Evaluation
Typed Decision Bench · v0.3.0
Environment
System One wire format · 6 systems
Grader
Ground-truth labels
Verifier
typed-decision-bench:0.3.0
Coverage
5387 of 5387 source tasks attempted; 5387 graded.
Open task evidence
S
System One Scorers1-scorer-4b-v2b
77.2 DecisionScore3642 / 5387 strict passes · 5387 graded · 0 execution errors
Full benchmark5387 tasks attempted
View run
Run details
Evaluation
Typed Decision Bench · v0.3.0
Environment
System One wire format · 6 systems
Grader
Ground-truth labels
Verifier
typed-decision-bench:0.3.0
Coverage
5387 of 5387 source tasks attempted; 5387 graded.
Open task evidence
O
OpenJevopenjev-latest
75.6 DecisionScore3971 / 5387 strict passes · 5387 graded · 0 execution errors
Full benchmark5387 tasks attempted
View run
Run details
Evaluation
Typed Decision Bench · v0.3.0
Environment
System One wire format · 6 systems
Grader
Ground-truth labels
Verifier
typed-decision-bench:0.3.0
Coverage
5387 of 5387 source tasks attempted; 5387 graded.
Open task evidence
N
NanoJevnanojev-0.6b
56.8 DecisionScore2002 / 5387 strict passes · 4582 graded · 805 execution errors
Full benchmark5387 tasks attempted
View run
Run details
Evaluation
Typed Decision Bench · v0.3.0
Environment
System One wire format · 6 systems
Grader
Ground-truth labels
Verifier
typed-decision-bench:0.3.0
Execution errors
805
Coverage
5387 of 5387 source tasks attempted; 4582 graded.
Open task evidence

DecisionScore is the mean per-task score from the release's own verifier over the full roster. Strict passes are that verifier's all-or-nothing verdicts (version under Run details); open the run for the per-task verdicts.

About Typed Decision Bench

What it measures

DecisionScore: the mean per-item proper score (1 - normalised Brier) over the full roster, unanswered items as 0; accuracy, ROC-AUC, ECE, log loss and Brier per suite.

Scope & limitations

Public source datasets overlap with some systems' training data; every system and suite carries a contamination tier on the benchmark page. Latency is not ranked: hosted APIs and local open-weight runs have different measurement boundaries.

Evaluation environment

Each system receives the identical compact-JSON state and typed questions through the System One wire format; structurally unsupported items are recorded as unanswered.

Source

Original benchmark

Dataset & task catalog ↗