Blobfish Benchmark Portfolio · 2026
One leaderboard cannot describe an agent. We evaluate the work domain by domain—against public Harbor tasks, real tool contracts, and worlds you can inspect over MCP.
The portfolio
The task count belongs to the named Harbor source suite. The world count belongs to the separate Blobfish companion environment. Keeping those layers explicit prevents a local diagnostic from masquerading as an upstream result.
Can an agent finish a legal matter—not just answer a question?
Long-horizon legal work across evidence rooms, custody checks, source-grounded findings, and final deliverables.
harbor run -d blobfishai/counselbench-100Can an agent find the number, show its provenance, and update the model?
Public financial-research questions paired with a deterministic DCF, assumptions, period-data, and review-note world.
harbor run -d vals/financeagentCan an agent follow policy while the customer and the system keep changing?
Multi-turn service-agent evaluation paired with an SLA, ticket, escalation, message, and knowledge-base world.
harbor run -d sierra-research/tau3-benchCan an agent turn scientific data into a reproducible result?
Scientific-programming tasks paired with an assay, compound, target, QC-result, and literature world.
harbor run -d scienceagentbench/scienceagentbenchCan an agent harden a service without reaching beyond the sandbox?
Local defensive-hardening tasks paired with a permission-aware search, document, employee, and access-grant world.
harbor run -d polyvorlabs/cyberdefense-benchCan an agent join the systems behind a real operational decision?
Cross-functional enterprise questions paired with vendors, processes, changes, controls, and review workflows.
harbor run -d Enterprise-Bench/l1-l2-benchCan an agent resolve the issue in the repository—not just propose a patch?
Expert-verified GitHub issues paired with a service, repository, pull-request, incident, ADR, and standup world.
harbor run -d swe-bench/swe-bench-verifiedOne publication contract
Inspired by the clarity of professional-work leaderboards, every page separates what was evaluated, what can be reproduced, and what has not been run yet.
Keep the upstream task, sandbox, verifier, version, and scoring semantics intact.
Open the companion schema, tools, seeded state, task split, and exact-state verifier locally.
Attach provider identity, configuration, trajectories, cost, and immutable artifacts before showing a score.
Result integrity
CounselBench has published, inspectable model runs. The six new suites now expose a 24/24 reference ceiling, a 0/24 seeded-random floor, real task and asset records, and paired trajectories. Model rows remain reserved for version-pinned executions.
See a fully published benchmark →Harbor inventory observed 2026-08-26; 282 public datasets were visible in the live catalog. Source licenses and runtime requirements remain authoritative at the linked upstream pages.