Blobfish Benchmark Portfolio · 2026
Evaluate the work domain by domain—with public Harbor tasks, disclosed tool contracts, stateful MCP worlds, and the evidence behind every result.
A model score appears only after a full run matches the exact released package. Solvability controls stay visibly separate from model claims.
The evidence contract
The source evaluation, executable environment, and recorded run are separate layers. Each has its own identity and evidence, so a local control can never masquerade as an upstream score.
Keep the public task, sandbox, verifier, version, and scoring semantics intact.
Inspect the companion schema, tools, seeded state, task split, and deterministic checks over MCP.
Resolve provider identity, configuration, trajectories, cost, and immutable artifacts to one exact release.
628/628 chain-audited tasks pass. Admission checks the full dependent chain: required investigations, derived intermediates, three costed and authority-checked alternatives, and an exact multi-field answer. No LLM judge is used.
Inspect the executable realism standard ↗The portfolio
Start with the domain, then open its evidence page to inspect tasks, assets, tool contracts, controls, executable world proofs, and any version-pinned model runs that meet the publication rule.
First-party releases: built, oracle-qualified, and published by Blobfish, each with a purpose-built page carrying full task, asset, contract, and trajectory evidence.
Can an agent finish a legal matter—not just answer a question?
Long-horizon legal work across evidence rooms, custody checks, source-grounded findings, and final deliverables.
harbor run -d blobfishai/counselbench-100Can an agent run the revenue workflow—not just summarize the call?
Long-horizon revenue work across Salesforce, HubSpot, Gong, seeded account rooms, controlled CRM mutations, and executive deliverables.
harbor run -d blobfishai/salesbench-100Can an agent run the incident, ship the fix, and prove it landed?
Long-horizon DevOps/SRE work over one executable NovaCart world: root-cause analysis, code changes, canaried deploys, migrations, flags, and incident closure.
harbor run -d blobfishai/devopsbench-100Can an agent run the finance desk—not just look up the number?
Corporate-finance work across 22 families: a D365-shaped ERP, Odoo, subsidiary books, drive, email, documents, and real SEC XBRL filings behind 8 MCP servers.
harbor run -d blobfishai/ledgerbench-100Can an agent answer the factory question—and safely carry the decision through?
Realistic manufacturing decisions framed as high-level employee requests. Agents must discover the relevant Oracle Fusion, Gmail, Drive, Sheets, and Slack evidence, reconcile records, calculate and compare options, then execute only the supported outcome.
harbor run -d blobfishai/factorybench-100Can an agent contain the semiconductor exception—and prove the fab is safe?
Long-horizon semiconductor exception work across lot genealogy, plasma etch, CMP, lithography, maintenance, materials, final test, reliability, FA/CAPA, and supply traceability.
harbor run -d blobfishai/semikongbench-100Can an agent run the deal—not just calculate one valuation cell?
Long-horizon investment-banking execution across source control, QoE, comps, precedents, DCF, LBO, merger math, bids, model-to-deck consistency, approvals, and review-ready client handoffs.
harbor run -d blobfishai/dealbench-100-suiteCan an agent run the ERP transaction end to end—not just read the record?
Long-horizon ERP execution across order import, shipment verification, receipt application, reorder requisitions, receiving and three-way match, worker document compliance, shift rollups, channel-order sync, hiring within quota, and effective-dated price batches.
harbor run -d blobfishai/erpbench-100-suiteCan an agent finish the CRM workflow without breaking the account?
Independent synthetic CRM work across contact onboarding, stage correction, bounded quotes, quote replacement, unsigned contracts and corrected document associations. All source conversations were inspected; zero source rows are reproduced.
harbor run -d blobfishai/arc-crm-6@sha256:fc9ca3b80d6a1a3c3e5bd1b975c54f1f3ada21b49668cb5f25e2912d67b69dfa -a oracle -e docker -n 1 -k 1 -r 0Benchmark-shaped environments with inspectable tasks, state, tools, verifiers, and trajectories. They are listed in the portfolio but kept outside leaderboard comparisons until one frozen evaluation identity has comparable model runs.
Can an agent coordinate real state changes across customer, support, billing, engineering, and knowledge systems without touching the wrong records?
The interactive Northstar release contains 100 frozen tasks with executable verifiers and task-level traces. Its page separately labels a larger 296-tool customer release and 100 single-model receipts, preventing either artifact from being mistaken for a cross-model leaderboard.
13 independently authored, oracle-proven domain families inspired by the Harbor Hub inventory. These shared worlds are not individual adaptations of every source dataset. Inspect source classifications and pending adaptations →
Can one agent handle employee decisions across 13 professional domains?
Each released family is a self-contained SQLite world behind provider-shaped MCP tools and a terminal CLI, where the question is only answerable through a dependent chain of evidence and a deterministic verifier grades the whole chain — options, intermediates, authority, and the exact answer.
Public Harbor evaluations from the wider ecosystem, each paired with a separate executable Blobfish diagnostic world.
Can an agent find the number, show its provenance, and update the model?
Public financial-research questions paired with a deterministic DCF, assumptions, period-data, and review-note world.
harbor run -d vals/financeagentCan an agent follow policy while the customer and the system keep changing?
Multi-turn service-agent evaluation paired with an SLA, ticket, escalation, message, and knowledge-base world.
harbor run -d sierra-research/tau3-benchCan an agent turn scientific data into a reproducible result?
Scientific-programming tasks paired with an assay, compound, target, QC-result, and literature world.
harbor run -d scienceagentbench/scienceagentbenchCan an agent harden a service without reaching beyond the sandbox?
Local defensive-hardening tasks paired with a permission-aware search, document, employee, and access-grant world.
harbor run -d polyvorlabs/cyberdefense-benchCan an agent join the systems behind a real operational decision?
Cross-functional enterprise questions paired with vendors, processes, changes, controls, and review workflows.
harbor run -d Enterprise-Bench/l1-l2-benchCan an agent resolve the issue in the repository—not just propose a patch?
Expert-verified GitHub issues paired with a service, repository, pull-request, incident, ADR, and standup world.
harbor run -d swe-bench/swe-bench-verifiedResult integrity
The five exact current packages pass 500/500 Harbor Oracle solvability trials with zero exceptions or retries. FactoryBench publishes one version-pinned full-suite model run. CounselBench, SalesBench, DevOpsBench, and LedgerBench likewise withhold older, partial, or pre-release rows until a full model run exists on the exact current release.
The six companion suites expose a 24/24 reference ceiling, a 0/24 wrong-evidence floor, inspectable task and asset records, and paired trajectories. Reference replay proves solvability, not model difficulty.
Harbor inventory observed 2026-08-29; 282 public datasets were visible in the live catalog. Source licenses and runtime requirements remain authoritative at the linked upstream pages.
Your workflow, made executable
Bring the systems, policies, and failure modes that matter. Blobfish turns them into inspectable tasks, stateful tools, deterministic verification, and a release you can rerun.