Skip to content
Blobfish AI
BenchmarksTalk to usOpen app

Published evaluation

Typed Decision Bench

Read the actual model outputs. Follow each task from its instructions to the final response and evaluator verdict.

EnvironmentSample task

Model run report

decider-2b

decider-2b-v8 · 2026-09-20

69.0%Task success345 / 500 tasks passed
500 / 5,387Tasks attemptedPublished task coverage
8.9sMean task time500 recorded measurements

abt-buy-product-match

retrieval-and-knowledge--abtbuy-1043

Passed

ITEM retrieval-and-knowledge:abtbuy-1043 · task abt-buy-product-match · e-commerce STATE (record, 164 tokens) { "record_a": { "source": "abt", "name": "olympus silver 8 megapixel digital camera fe370s", "description": "olympus silver 8 megapixel digital camera fe370s dual image stabilization 5x optical zoom face detection intelligent auto smile shot 2.7 ' high resolution lcd 18 shooting modes 8.0 megapixels silver finish", "price": null }, "record_b": { "source": "buy", "name": "canon powershot sx110 is digital camera silver 2666b001", "description": "canon powershot sx110 is 9 megapixel digital camera w / 10x optical zoom , 3 ' lcd , iso 1600 , optical image stabilizer silver", "price": null } } QUESTION link_state (ordinal score) The state holds two product records, record_a from retailer abt and record_b from retailer buy (name, description, price). Rate how the two records relate as products. - 0: Two different products (different model or item, or only the same brand/category). - 1: Closely related products that may or may not be the same one: a colour or bundle variant, or a name that could refer to either. - 2: One and the same product (same manufacturer model, possibly worded or abbreviated differently). Answer every question with a calibrated probability distribution over its options.

decider-2b4.7s1 recorded actions

Agent response

link_state: 0 (0 0.73, 1 0.19, 2 0.08)

Original response text
link_state: 0 (0 0.73, 1 0.19, 2 0.08)
Raw outputsResults CSVSummaryDownload HTMLReport JSON
Run configuration & methodology
Run
typed-decision-bench-v03-20260919:decider-2b
Model
decider-2b-v8
Environment
open weights (Mapika/decider-2b) · Apple M-series MPS (eager engine, no flash-linear-attention) · model default · jevfish_bench runner · jevfish_bench.competitors.decider_server · local MPS
Task budget
One request per item · temperature 1.3 · 32768-token input limit
Primary evaluator
native
Benchmark version
0.3.0
Evaluation
Executable grading against each dataset's own ground truth: the top-probability answer is compared with the expected answer, and the full distribution is scored with a proper score (1 - normalised Brier). No LLM judge.

Typed Decision Bench 0.3.0: 5,387 frozen items over five use-case family suites (workflow control, retrieval and knowledge, safety and quality, data and operations, real-time and agents), drawn from public datasets with their own ground truth. Every system answered every item through the System One wire format — one request carrying the state and its typed questions, one calibrated distribution back. This report publishes 500 of those items (20 per task, drawn with seed 20260919 — the same items for every system) with the request, the answer and the grade for each; raw_outputs.jsonl carries the same 500 items as machine-readable records. The ranked metric on the leaderboard, DecisionScore, is measured over the complete 5,387-item roster. Grading is executable: the top-probability answer is compared with the dataset's expected answer and the full distribution is scored with a proper score (1 - normalised Brier). No LLM judge.

Typed Decision Bench 0.3.0: 5387 frozen items over five use-case family suites. Every system answered every item through the System One wire format; this report publishes 500 of them (20 per task, drawn with seed 20260919 — the same items for every system), and raw_outputs.jsonl carries the complete per-item record. Executable grading against each dataset's own ground truth: the top-probability answer is compared with the expected answer, and the full distribution is scored with a proper score (1 - normalised Brier). No LLM judge.

Environment

What decider-2b had to work with.

decider-2bdecider-2b-v82026-09-20T05:27:22Z → 2026-09-20T05:27:22Z

Agent & scaffold

Model
decider-2b-v8
Agent
decider-2b
Harness
System One wire format · 6 systems
Environment
open weights (Mapika/decider-2b) · Apple M-series MPS (eager engine, no flash-linear-attention) · model default · jevfish_bench runner · jevfish_bench.competitors.decider_server · local MPS

Budget

Budget
One request per item · temperature 1.3 · 32768-token input limit

Grader

Primary evaluator
native
Evaluator
Ground-truth labels
Verdicts by grader
Original tests → 500 verdicts

Recorded window

Recorded window
2026-09-20T05:27:22Z → 2026-09-20T05:27:22Z
Task roster
500 attempted of 5387 tasks
Task source
0.3.0

Environment and sample task above belong to decider-2b. Other configurations in this evaluation: Jev · JevFish · NanoJev · OpenJev · System One Scorer.

Sample task from this run (1 of 500 attempted)

arxiv-archive-subcategory · data-and-operations--arxiv-3f0cfd7f28a7

One real task from this run on Typed Decision Bench: the prompt, the answer as given, and the grader’s verdict.

This sample is for illustration; a single task does not reflect a configuration’s overall result.

Task

ITEM data-and-operations:arxiv-3f0cfd7f28a7 · task arxiv-archive-subcategory · science-publishing STATE (text, 115 tokens) { "title": "Interaction induced topological phase transition in Bernevig-Hughes-Zhang model", "abstract": "We study interaction induced topological phase transition in Bernevig-Hughes-Zhang model. Topological nature of the phase transition is revealed by directly calculating the Z2 index of the interacting system from the single-particle Green's function. The interacting Z2 index is also consistently checked through the edge spectra. Combined with ab initio methods, present approach is a useful tool searching for correlated topological insulating materials from the first-principle point of view." } QUESTION archive (single choice) Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in? - astro_ph: Astrophysics - cond_mat: Condensed matter physics - cs: Computer science - econ: Economics - eess: Electrical engineering and systems science - math: Mathematics - nlin: Nonlinear sciences - physics: Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...) - q_bio: Quantitative biology - q_fin: Quantitative finance - stat: Statistics QUESTION subcategory (single choice) Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories. - cond_mat_dis_nn: cond.mat.dis.nn - cond_mat_mes_hall: cond.mat.mes.hall - cond_mat_mtrl_sci: cond.mat.mtrl.sci - cond_mat_other: cond.mat.other - cond_mat_quant_gas: cond.mat.quant.gas - cond_mat_soft: cond.mat.soft - cond_mat_stat_mech: cond.mat.stat.mech - cond_mat_str_el: cond.mat.str.el - cond_mat_supr_con: cond.mat.supr.con Answer every question with a calibrated probability distribution over its options.

decider-2b

decider-2b-v8System One wire format · 6 systems3.7 s1 steps

Pass

Graded by Original tests · Typed Decision Bench executable grader · score 0.72

Criteria · Original tests
CriterionCriterion typeVerdict
archive: expected cond_mat — answered cond_mat (P(expected) 0.88, proper score 0.99)Task criterionYes
subcategory: expected cond_mat_str_el — answered cond_mat_mtrl_sci (P(expected) 0.07, proper score 0.46)Task criterionNo

Grader reasoning

Deciding question `archive`: expected cond_mat, answered cond_mat — correct. It gave the expected answer a probability of 0.88; proper score 0.72.

Full trajectory ↗Model report ↗

Trajectory

decider-2b

decider-2b-v8System One wire format · 6 systems3.7 s1 steps

Pass

Final answer

archive: cond_mat (cond_mat 0.88, physics 0.07, math 0.02, nlin 0.02, eess 0.00, cs 0.00)
subcategory: cond_mat_mtrl_sci (cond_mat_mtrl_sci 0.40, cond_mat_stat_mech 0.23, cond_mat_supr_con 0.08, cond_mat_str_el 0.07, cond_mat_other 0.07, cond_mat_dis_nn 0.06)

Steps, newest first

Final answer

archive: cond_mat (cond_mat 0.88, physics 0.07, math 0.02, nlin 0.02, eess 0.00, cs 0.00)
subcategory: cond_mat_mtrl_sci (cond_mat_mtrl_sci 0.40, cond_mat_stat_mech 0.23, cond_mat_supr_con 0.08, cond_mat_str_el 0.07, cond_mat_other 0.07, cond_mat_dis_nn 0.06)
  1. Recorded browser state
    Deciding question `archive`: expected cond_mat, answered cond_mat — correct. It gave the expected answer a probability of 0.88; proper score 0.72.
    Full input and result
    {
 "archive": "cond_mat",
 "subcategory": "cond_mat_str_el"
}
    Deciding question `archive`: expected cond_mat, answered cond_mat — correct. It gave the expected answer a probability of 0.88; proper score 0.72.
  2. +3.7 s
    POST /v1/systemone{ "archive": { "type": "choice", "instructions": "Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary cat…
    {"archive":{"type":"choice","choice":"cond_mat","confidence":0.869003099690031,"probabilities":{"astro_ph":0.0014998500149985001,"cond_mat":0.8809119088091191,"…
    Full input and result
    {
 "archive": {
  "type": "choice",
  "instructions": "Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in?",
  "options": {
   "astro_ph": "Astrophysics",
   "cond_mat": "Condensed matter physics",
   "cs": "Computer science",
   "econ": "Economics",
   "eess": "Electrical engineering and systems science",
   "math": "Mathematics",
   "nlin": "Nonlinear sciences",
   "physics": "Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...)",
   "q_bio": "Quantitative biology",
   "q_fin": "Quantitative finance",
   "stat": "Statistics"
  }
 },
 "subcategory": {
  "type": "choice",
  "instructions": "Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories.",
  "options": {
   "cond_mat_dis_nn": "cond.mat.dis.nn",
   "cond_mat_mes_hall": "cond.mat.mes.hall",
   "cond_mat_mtrl_sci": "cond.mat.mtrl.sci",
   "cond_mat_other": "cond.mat.other",
   "cond_mat_quant_gas": "cond.mat.quant.gas",
   "cond_mat_soft": "cond.mat.soft",
   "con
… [133 more characters in raw_outputs.jsonl]
    {
 "archive": {
  "type": "choice",
  "choice": "cond_mat",
  "confidence": 0.869003099690031,
  "probabilities": {
   "astro_ph": 0.0014998500149985001,
   "cond_mat": 0.8809119088091191,
   "cs": 0.0020997900209979003,
   "econ": 0.000999900009999,
   "eess": 0.0023997600239976003,
   "math": 0.022797720227977204,
   "nlin": 0.017098290170982904,
   "physics": 0.06889311068893111,
   "q_bio": 0.0005999400059994001,
   "q_fin": 0.0007999200079992001,
   "stat": 0.0018998100189981002
  },
  "extensions": {
   "native": {
    "certainty": 0.7843
   }
  }
 },
 "subcategory": {
  "type": "choice",
  "choice": "cond_mat_mtrl_sci",
  "confidence": 0.3198694869486948,
  "probabilities": {
   "cond_mat_dis_nn": 0.0636063606360636,
   "cond_mat_mes_hall": 0.014301430143014302,
   "cond_mat_mtrl_sci": 0.3954395439543954,
   "cond_mat_other": 0.0701070107010701,
   "cond_mat_quant_gas": 0.0636063606360636,
   "cond_mat_soft": 0.0055005500550055,
   "cond_mat_stat_mech": 0.23302330233023302,
   "cond_mat_str_el": 0.0735073507350735,
   "cond_mat_supr_con": 0.0809080908090809
  },
  "extensions": {
   "native": {
    "certainty": 0.2136
   }
  }
 }
}
  3. Recorded browser state
    {"title":"Interaction induced topological phase transition in Bernevig-Hughes-Zhang model","abstract":"We study interaction induced topological phase transition…
    Full input and result
    {
 "title": "Interaction induced topological phase transition in Bernevig-Hughes-Zhang model",
 "abstract": "We study interaction induced topological phase transition in Bernevig-Hughes-Zhang model. Topological nature of the phase transition is revealed by directly calculating the Z2 index of the interacting system from the single-particle Green's function. The interacting Z2 index is also consistently checked through the edge spectra. Combined with ab initio methods, present approach is a useful tool searching for correlated topological insulating materials from the first-principle point of view."
}
Full trajectory ↗Model report ↗