Blobfish AI
BenchmarksTalk to usOpen app
Benchmark operations / observed 2026-09-16 19:22 UTC

Every benchmark. Evidence of progress.

Track native results, active model cohorts and preparation across all 17 portfolio entries and 20 product categories. Every result keeps its original task set and metric.

17 portfolio entries20 product categories3 registered full native packages1 dedicated GKE cluster

Retained native evidence

Completed results, with their original scope.

These evaluations preserve each benchmark’s own metric and task set. Active campaigns have no final score at this observation.

Model cohort finished4 companion tasks · 102 upstream tasks separate

DiscoveryBench-102 · four-task companion

DeepSeek V4 Pro · all 4 original attempts retained · cost not reported.

0/4 native passes4/4 graded attempts0 unscored execution errors3/4 final-state checks0/4 required-evidence checks

Native passes and diagnostic checks cover graded attempts only. All 4 attempts are retained; execution errors have no native verdict. Final state and required evidence are diagnostics within the original grader.

Every original task · deepseek-v4-pro · 2026-09-16 09:56 UTC
TaskNative verdictFinal stateRequired evidenceAttempt elapsedCost
dedupe assay resultFailPassFail8.5 minNot reported
duplicate compound nameFailPassFail4.5 minNot reported
invalidate failed runFailPassFail5.3 minNot reported
promote hit compoundFailFailFail10.8 minNot reported

Attempt elapsed includes runtime setup and cleanup. The original 900-second solver and 120-second verifier budgets are preserved. Disposable runtime resources were removed after collection.

Frozen source and retained evidence

Source version: 3.3.2:revision9:gke-1. The download retains each original archive’s SHA-256 and the verifier identity for every graded attempt.

Original result SHA-256: 7ff62a2acaf7dc52f8886f0e8ec66bc8437638d2670513c2329920c02435a3d6

Download task verdicts and source identities ↓

Model cohort finished4 companion tasks · 375 upstream tasks separate

ServiceBench-375 · four-task companion

DeepSeek V4 Pro · all 4 original attempts retained · cost not reported.

0/1 native passes1/4 graded attempts3 unscored execution errors1/1 final-state checks0/1 required-evidence checks

Native passes and diagnostic checks cover graded attempts only. All 4 attempts are retained; execution errors have no native verdict. Final state and required evidence are diagnostics within the original grader.

Every original task · deepseek-v4-pro · 2026-09-16 14:01 UTC
TaskNative verdictFinal stateRequired evidenceAttempt elapsedCost
cs close resolved ticketsUnscoredSolver time limitNot gradedNot graded15.2 minNot reported
cs dedupe kb articleFailPassFail13.2 minNot reported
cs duplicate customer nameUnscoredSolver service errorNot gradedNot graded7.7 minNot reported
cs escalate sla breachUnscoredSolver time limitNot gradedNot graded15.2 minNot reported

Attempt elapsed includes runtime setup and cleanup. The original 900-second solver and 120-second verifier budgets are preserved. Disposable runtime resources were removed after collection.

Frozen source and retained evidence

Source version: 3.3.2:revision9:gke-1. The download retains each original archive’s SHA-256 and the verifier identity for every graded attempt.

Original result SHA-256: f216e6284ce70c4b1ea9a3bc3ae7b5bf1a21c88b8378537f2846b9358abbd82d

Download task verdicts and source identities ↓

Model cohort finished4 companion tasks · 500 upstream tasks separate

RepoBench-500 · four-task companion

DeepSeek V4 Pro · all 4 original attempts retained · cost not reported.

0/3 native passes3/4 graded attempts1 unscored execution errors3/3 final-state checks0/3 required-evidence checks

Native passes and diagnostic checks cover graded attempts only. All 4 attempts are retained; execution errors have no native verdict. Final state and required evidence are diagnostics within the original grader.

Every original task · deepseek-v4-pro · 2026-09-16 14:29 UTC
TaskNative verdictFinal stateRequired evidenceAttempt elapsedCost
eng dedupe standup noteFailPassFail7.9 minNot reported
eng duplicate repository nameFailPassFail1.6 minNot reported
eng reopen pr with red ciFailPassFail2.8 minNot reported
eng resolve ingest incidentUnscoredSolver time limitNot gradedNot graded15.2 minNot reported

Attempt elapsed includes runtime setup and cleanup. The original 900-second solver and 120-second verifier budgets are preserved. Disposable runtime resources were removed after collection.

Frozen source and retained evidence

Source version: 3.3.2:revision9:gke-1. The download retains each original archive’s SHA-256 and the verifier identity for every graded attempt.

Original result SHA-256: e38dc3a540854d9457ecbfe8490840a3ee37a9bfedef3a3b2360c2235a33becd

Download task verdicts and source identities ↓

Model cohort finished4 companion tasks · 100 upstream tasks separate

DefenseBench-100 · four-task companion

DeepSeek V4 Pro · all 4 original attempts retained · cost not reported.

0/4 native passes4/4 graded attempts0 unscored execution errors4/4 final-state checks0/4 required-evidence checks

Native passes and diagnostic checks cover graded attempts only. All 4 attempts are retained; execution errors have no native verdict. Final state and required evidence are diagnostics within the original grader.

Every original task · deepseek-v4-pro · 2026-09-16 16:03 UTC
TaskNative verdictFinal stateRequired evidenceAttempt elapsedCost
es dedupe document chunkFailPassFail1.7 minNot reported
es duplicate person nameFailPassFail5.2 minNot reported
es lock down confidential documentFailPassFail4.7 minNot reported
es mark disconnected source staleFailPassFail7.4 minNot reported

Attempt elapsed includes runtime setup and cleanup. The original 900-second solver and 120-second verifier budgets are preserved. Disposable runtime resources were removed after collection.

Frozen source and retained evidence

Source version: 3.3.2:revision9:gke-1. The download retains each original archive’s SHA-256 and the verifier identity for every graded attempt.

Original result SHA-256: 78c4824533b98a8ca0be1416bd802470a4fe9354e6fa3a4820e690e31f74bd94

Download task verdicts and source identities ↓

Model cohort finished4 companion tasks · 50 upstream tasks separate

FilingBench-50 · four-task companion

DeepSeek V4 Pro · all 4 original attempts retained · cost not reported.

0/3 native passes3/4 graded attempts1 unscored execution errors2/3 final-state checks0/3 required-evidence checks

Native passes and diagnostic checks cover graded attempts only. All 4 attempts are retained; execution errors have no native verdict. Final state and required evidence are diagnostics within the original grader.

Every original task · deepseek-v4-pro · 2026-09-16 16:35 UTC
TaskNative verdictFinal stateRequired evidenceAttempt elapsedCost
fa dedupe dcf assumptionFailPassFail11.5 minNot reported
fa duplicate company listingFailPassFail6.3 minNot reported
fa finalize solstice dcfFailFailFail12.9 minNot reported
fa revert model without discount rateUnscoredRuntime API timeoutNot gradedNot graded0.4 minNot reported

Attempt elapsed includes runtime setup and cleanup. The original 900-second solver and 120-second verifier budgets are preserved. Disposable runtime resources were removed after collection.

Frozen source and retained evidence

Source version: 3.3.2:revision9:gke-1. The download retains each original archive’s SHA-256 and the verifier identity for every graded attempt.

Original result SHA-256: 1795ccf496707342fe75e9cb721a58fec79ba4046a94f821766ce8082d8dc693

Download task verdicts and source identities ↓

Model cohort finished4 companion tasks · 14 upstream tasks separate

OpsBench-14 · four-task companion

DeepSeek V4 Pro · all 4 original attempts retained · cost not reported.

0/4 native passes4/4 graded attempts0 unscored execution errors3/4 final-state checks0/4 required-evidence checks

Native passes and diagnostic checks cover graded attempts only. All 4 attempts are retained; execution errors have no native verdict. Final state and required evidence are diagnostics within the original grader.

Every original task · deepseek-v4-pro · 2026-09-16 17:03 UTC
TaskNative verdictFinal stateRequired evidenceAttempt elapsedCost
ops dedupe process stepFailPassFail2.5 minNot reported
ops duplicate vendor nameFailPassFail9.9 minNot reported
ops flag overdue vendor renewalFailPassFail7.2 minNot reported
ops reject uncontrolled changeFailFailFail8.4 minNot reported

Attempt elapsed includes runtime setup and cleanup. The original 900-second solver and 120-second verifier budgets are preserved. Disposable runtime resources were removed after collection.

Frozen source and retained evidence

Source version: 3.3.2:revision9:gke-1. The download retains each original archive’s SHA-256 and the verifier identity for every graded attempt.

Original result SHA-256: 975dc87dd4d52f2fd439f4f8dddb121e8c6e49deb70fe1d57b0226cf7ed42f85

Download task verdicts and source identities ↓

Model cohort running at observation

CyberDefense · original 100 tasks

Original 100-task CyberDefense benchmark, separate from the four-task DefenseBench companion.

All 100 reference and 100 no-op native controls passed. The original DeepSeek V4 Pro cohort remained active at the dated Job observation; no final model score is available.

One native task complete

MTEB · AILAStatutes

0.50353 NDCG@10

One complete native MTEB retrieval task: 50 test queries, 82 documents and 217 relevance judgments. Full MTEB/MMTEB coverage remains pending.

Voyage 4 Large scored 0.50353 NDCG@10 using the pinned native MTEB harness. Perfect and irrelevant ranking controls each covered all 50 queries. Cost is unknown; API model weights are not immutable. Original results and scoring evidence are retained.

Download native results, controls and query scores ↓

One native task complete

MTEB · AILACasedocs

0.45300 NDCG@10

One complete native MTEB retrieval task: 50 test queries, 186 documents and 195 relevance judgments. Full MTEB/MMTEB coverage remains pending.

Voyage 4 Large scored 0.45300 NDCG@10 using the pinned native MTEB harness. Perfect and irrelevant ranking controls each covered all 50 queries. Long inputs retain the native adapter’s default truncation. Cost is unknown; API model weights are not immutable. Original results and scoring evidence are retained.

Download native results, controls and query scores ↓

The complete public portfolio

17 entries, with their real scope.

Nine original benchmarks, six companion worlds, Northstar and HubBench. A companion’s four tasks are distinct from the upstream benchmark’s full task set.

Explore the portfolio ↗
Execution observed 2026-09-16 19:22 UTC · full model matrix pending
Benchmark / worldEvaluation scopeTasksExecutionEvidence and next action
CounselBench-100 ↗Original benchmarkOriginal release identity required100Qualification requiredbf-benchmarks · dedicated run JobPrepare the exact complete published release, immutable images and original verifier; qualify world services, MCP tools, evidence mounts and native task timeouts.
SalesBench-100 ↗Original benchmarkOriginal release identity required100Qualification requiredbf-benchmarks · dedicated run JobPrepare the exact complete published release, immutable images and original verifier; qualify world services, MCP tools, evidence mounts and native task timeouts.
DevOpsBench-100 ↗Original benchmarkOriginal release identity required100Full package registeredbf-benchmarks · dedicated run JobAll 100 original tasks and immutable runtime images are registered. Complete model attempts and grading remain required; earlier ten-task controls and pilots retain their original scope.
LedgerBench-100 ↗Original benchmarkOriginal release identity required100Qualification requiredbf-benchmarks · dedicated run JobPrepare the exact complete published release, immutable images and original verifier; qualify world services, MCP tools, evidence mounts and native task timeouts.
FactoryBench-100 ↗Original benchmarkOriginal release identity required100Qualification requiredbf-benchmarks · dedicated run JobPrepare the exact complete published release, immutable images and original verifier; qualify world services, MCP tools, evidence mounts and native task timeouts.
SemiKongBench-100 ↗Original benchmarkOriginal release identity required100Qualification requiredbf-benchmarks · dedicated run JobPrepare the exact complete published release, immutable images and original verifier; qualify world services, MCP tools, evidence mounts and native task timeouts.
DealBench-100 ↗Original benchmarkOriginal release identity required100Qualification requiredbf-benchmarks · dedicated run JobPrepare the exact complete published release, immutable images and original verifier; qualify world services, MCP tools, evidence mounts and native task timeouts.
ERPBench-100 ↗Original benchmarkOriginal release identity required100Qualification requiredbf-benchmarks · dedicated run JobPrepare the exact complete published release, immutable images and original verifier; qualify world services, MCP tools, evidence mounts and native task timeouts.
Arc CRM 6 ↗Original benchmarkOriginal release identity required6Qualification requiredbf-benchmarks · dedicated run JobPrepare the exact complete published release, immutable images and original verifier; qualify world services, MCP tools, evidence mounts and native task timeouts.
FilingBench-50 ↗Companion world50 upstream source tasks are separate4Model cohort finished4/4 reference · 4/4 no-op controls0/3 native passes · 4/4 attempted · 1 unscored execution errorsDeepSeek V4 Pro’s original four-task companion reached terminal outcomes: 3 graded native failures and 1 unscored execution error. No native passes. All original attempts and errors are retained without replay. The 50-task upstream benchmark remains separate.

Inspect GKE control results →

ServiceBench-375 ↗Companion world375 upstream source tasks are separate4Model cohort finished4/4 reference · 4/4 no-op controls0/1 native passes · 4/4 attempted · 3 unscored execution errorsDeepSeek V4 Pro’s original four-task companion reached terminal outcomes: 1 graded native failure and 3 unscored execution errors. No native passes. All original attempts and errors are retained without replay. The 375-task upstream benchmark remains separate.

Inspect GKE control results →

DiscoveryBench-102 ↗Companion world102 upstream source tasks are separate4Model cohort finished4/4 reference · 4/4 no-op controls0/4 native passes · 4/4 attempted · 0 unscored execution errorsDeepSeek V4 Pro completed 4/4 companion tasks with 0/4 native passes. Final-state checks passed 3/4; required evidence checks passed 0/4. These diagnostics do not replace the native verdict. The 102-task upstream benchmark is separate.

Inspect GKE control results →

DefenseBench-100 ↗Companion world100 upstream source tasks are separate4Model cohort finished4/4 reference · 4/4 no-op controls0/4 native passes · 4/4 attempted · 0 unscored execution errorsDeepSeek V4 Pro’s original four-task companion reached terminal outcomes: 4 graded native failures and 0 unscored execution errors. No native passes. All original attempts and errors are retained without replay. The 100-task upstream benchmark remains separate.

Inspect GKE control results →

OpsBench-14 ↗Companion world14 upstream source tasks are separate4Model cohort finished4/4 reference · 4/4 no-op controls0/4 native passes · 4/4 attempted · 0 unscored execution errorsDeepSeek V4 Pro’s original four-task companion reached terminal outcomes: 4 graded native failures and 0 unscored execution errors. No native passes. All original attempts and errors are retained without replay. The 14-task upstream benchmark remains separate.

Inspect GKE control results →

RepoBench-500 ↗Companion world500 upstream source tasks are separate4Model cohort finished4/4 reference · 4/4 no-op controls0/3 native passes · 4/4 attempted · 1 unscored execution errorsDeepSeek V4 Pro’s original four-task companion reached terminal outcomes: 3 graded native failures and 1 unscored execution error. No native passes. All original attempts and errors are retained without replay. The 500-task upstream benchmark remains separate.

Inspect GKE control results →

Northstar Composite Ops ↗World proofOriginal release identity required100Model cohort running at observationbf-benchmarks · dedicated run JobThe original 100-task DeepSeek cohort remained active at the dated Job observation, with prior attempts and errors retained. Live native controls cover one reference and one no-op check (1/100 tasks); complete GKE control coverage is pending. No final model score is available.

Inspect GKE control results →

HubBench ↗13-family suiteOriginal release identity required104Qualification requiredbf-benchmarks · dedicated run JobQualify the complete 104-task release, multiple MCP endpoints, named internal networks and original 1,200-second task budgets.

Existing model results and reference controls keep their original release, task set and provenance. This matrix does not replace those historical results with a new score.

20 product categories

Match the benchmark to the work.

Each benchmark has an execution target and a concrete preparation requirement. Company fit describes candidates for evaluation.

20 / 20 categories

Browser agents · 3 benchmark entries

Successful completion of natural-language browser tasks.

Execution profile: Browser / Harbor. Prepare the complete task roster and benchmark-specific grading; live sites and self-hosted resettable sites require different environments.

Desktop and mobile agents · 4 benchmark entries

Application control, visual grounding and verified changes to application state.

Execution profile: Desktop or emulator. A qualified VM/emulator or grounding adapter, benchmark assets and native evaluator are required; the CPU container lane is insufficient.

Coding agents · 4 benchmark entries

Issue resolution, executable terminal work, regressions, time and cost.

Execution profile: Harbor / CPU Linux. Freeze the complete release, build task and verifier images, and qualify original positive and negative controls.

Customer-service agents · 3 benchmark entries

Correct actions, policy adherence, completion and repeated-conversation consistency.

Execution profile: Tool and conversation simulation. Pin the simulator, policies, domain state and native evaluator; supply required service access and reset behavior.

Harness qualification requiredFull suite pending

Berkeley Function Calling Leaderboard (BFCL) ↗

Full benchmark run pending.

Tool-call model adapter or instrumented agent interface.

Execution target: bf-benchmarks · Tool and conversation simulation. See the dated evidence above for admitted evaluations.

Alibaba Qwen ↗Anthropic ↗Cohere ↗Composio ↗DeepSeek ↗Glean ↗Google DeepMind ↗Meta ↗MiniMax ↗Mistral ↗OpenAI ↗Salesforce ↗Z.ai ↗Zapier ↗n8n ↗xAI ↗

Original benchmark and access requirements ↗

CRM agents · 2 benchmark entries

Correct record queries and mutations with an explicit schema and tool mapping.

Execution profile: Tool and conversation simulation. Pin the simulator, policies, domain state and native evaluator; supply required service access and reset behavior.

Enterprise workflow agents · 3 benchmark entries

Multi-step completion across applications, documents and business records.

Execution profile: Multi-service world / Harbor. Qualify each world service, MCP endpoint, evidence mount and deterministic reset with its original grader.

Search and research agents · 4 benchmark entries

Difficult retrieval, supported answers, research quality and citation quality.

Execution profile: Browser / Harbor. Prepare the complete task roster and benchmark-specific grading; live sites and self-hosted resettable sites require different environments.

Embeddings and retrieval · 3 benchmark entries

Document relevance and ranking; latency and throughput for complete pipelines.

Execution profile: Retrieval and structured evaluation. Prepare the corpus and relevance labels, embedding/search or answer adapter, and the benchmark-native scoring pipeline.

SQL and business-analysis agents · 3 benchmark entries

SQL execution correctness and multi-step business-analysis accuracy.

Execution profile: Data and artifact evaluation. Prepare databases/workbooks and original grading tools; external warehouses need their benchmark-specific access.

Spreadsheet agents · 2 benchmark entries

Correct workbook artifacts, formulas and graded formatting.

Execution profile: Data and artifact evaluation. Prepare databases/workbooks and original grading tools; external warehouses need their benchmark-specific access.

Document parsing and extraction · 4 benchmark entries

Text, tables, reading order, formulas, fields and supporting evidence.

Execution profile: Document evaluation. Prepare source documents, parser interface and pinned text/table/layout evaluator images.

Legal research and contracts · 4 benchmark entries

Reasoning, evidence retrieval, clause extraction and interpretation.

Execution profile: Retrieval and structured evaluation. Prepare the corpus and relevance labels, embedding/search or answer adapter, and the benchmark-native scoring pipeline.

Financial research · 3 benchmark entries

Evidence-backed answers, numerical reasoning and professional deliverables.

Execution profile: Data and artifact evaluation. Prepare databases/workbooks and original grading tools; external warehouses need their benchmark-specific access.

Speech recognition · 4 benchmark entries

Transcription across languages and domains; diarization scored separately.

Execution profile: Speech input/output. Qualify an ASR or TTS endpoint, source audio and native scoring; reference-speaker and language conditions must be preserved.

Speech synthesis and cloning · 1 benchmark entries

Intelligibility and reference-speaker similarity; separate listening and latency tests.

Execution profile: Speech input/output. Qualify an ASR or TTS endpoint, source audio and native scoring; reference-speaker and language conditions must be preserved.

Conversational voice agents · 2 benchmark entries

Spoken task completion, tool use, interruptions and response latency.

Execution profile: Streaming voice simulation. Qualify streaming audio, interruption timing, tool/user simulation and separate task, ASR and TTS scores.

Image generation · 3 benchmark entries

Prompt adherence, counts, attributes and spatial relationships.

Execution profile: Media generation and scoring. Qualify image/video generation and benchmark-native evaluator images; GPU requirements need a separately bounded execution profile.

Video generation · 1 benchmark entries

Temporal consistency, quality, motion and prompt alignment.

Execution profile: Media generation and scoring. Qualify image/video generation and benchmark-native evaluator images; GPU requirements need a separately bounded execution profile.

Multimodal understanding · 3 benchmark entries

Reasoning and understanding over images and video.

Execution profile: Image and video understanding. Prepare licensed media, modality-capable model input and benchmark-native scoring with the original sampling protocol.

Agent security · 2 benchmark entries

Prompt-injection resistance while preserving legitimate task success.

Execution profile: Tool and conversation simulation. Pin the simulator, policies, domain state and native evaluator; supply required service access and reset behavior.

Dedicated benchmark compute

Isolated tasks. Bounded runs.

Each admitted evaluation uses its own GKE Job, frozen manifest and retained artifacts. Benchmark compute is separate from production world storage.

Inspect harness records ↗

Original harness and graders

Harbor 0.21.0 and HUD 0.6.17 are installed. Benchmark-specific adapters and original positive/negative controls determine readiness.

Dedicated execution

gVisor task pods; a bounded dedicated Job for each admitted evaluation. The task pool scales between 0 and 2 nodes.

Retained outcomes

Retain original results and verify their hashes before removing each completed campaign's Job, credential and task pods. Temporary campaign pools are removed after their owned work or at their absolute deadline. Benchmark artifacts and world data are retained.

Runtime limits and remaining adapter work

The default API execution profile; native campaigns preserve their separately recorded phase and Job deadlines. Default target: bf-benchmarks, us-central1-a. System node: e2-standard-2; default task nodes: e2-standard-4, maximum 2. Default API task limit: 2 CPU, 4096 MiB RAM, 900 seconds; Job limit: 10980 seconds.

Companion campaigns preserve 900-second solver and 120-second verifier phases. Native retrieval uses a pinned MTEB image and a 1,800-second Job deadline. Temporary campaign pools have absolute teardown deadlines independent of the default pool.

Additional profiles are required for arbitrary multi-service networks, evidence volumes, desktop VMs/emulators, GPU evaluators and streaming media protocols. Each requires its original published task budget and native controls before admission.

Registry observation: 2026-09-16T04:23:21.089063+00:00. This page records dated execution evidence; it is not a live availability monitor.