Every score has a task behind it.

What a run records

A run freezes the benchmark version, source archive digest, selected task IDs, environment image digests, agent scaffolds, model IDs, grading rules, and execution limits. Changing any of these creates a new evaluation. Different models and scaffold configurations can run against the same selected tasks.

The live registry is the source of available task packages. Controls verify the infrastructure and are labeled as controls. A one-task check does not establish full-suite performance.

Contained execution on GKE

Each task attempt gets a fresh gVisor environment. A durable Kubernetes job owns the run, so it continues when a browser closes. Task environments have no Kubernetes credentials, no host mounts, bounded CPU and memory, and an expiration deadline. Network policies block private networks, cluster services, and cloud metadata.

The initial execution lane accepts prepared, single-container CPU Linux environments. Compose stacks, GPUs, and dynamic hostname allowlists need a separately qualified runtime. Unsupported declarations are rejected during registration or task admission.

Three distinct grading dimensions

Native task grade

HUD returns its tasks.grade receipt. Harbor runs the task’s tests/test.sh and reads the configured reward metric. Every task must declare its grader version and pass threshold. Missing receipts and execution errors never supply a passing verdict.

Rubric evaluation

An optional, versioned judge evaluates task-specific criteria against the prompt, answer, and reference. Every criterion must be met for a pass. Its verdict, explanation, model, and token usage remain separate from the native score.

Vision evaluation

An optional visual judge evaluates captured PNG evidence against the task’s criteria. HUD captures the final RFB display; Harbor uses declared screenshot artifacts. Missing screenshots or invalid judge responses produce an evaluation error.

Rubric and vision are model judgments, not deterministic proof. A configured grader must have a credential at admission. An unconfigured dimension is reported as “not configured,” and a failed evaluator stays ungraded.

Reading the report

Task success counts successful recorded attempts, including execution errors in the denominator. Reliability measures completion without execution errors. The report shows selected coverage alongside source-suite coverage. Missing measurements remain unknown.

Rubric and vision pass rates require complete grading coverage. The evaluator summary also shows passes, graded tasks, and total selected tasks for each site. Cost includes only measured provider usage; judge usage is retained separately. End-to-end task latency includes setup, agent execution, native verification, and additional grading.

Competitive rankings require matching task sets, dataset version, grader contract, environment, and budgets. Imported results retain their submitted provenance; matching declarations do not mean Blobfish independently reran them.

Your files

Reports are private to the owning Blobfish API key. Share the downloaded files when you choose. Submitted provider credentials are redacted from retained artifacts and removed with the temporary job secret.

Add a benchmark

Registration is an operator workflow. Run callers select registered task packages; they do not upload executable controller code. Prepare and publish environment images, then pin their digests in the manifest. The runner does not build images during an evaluation.

1. Prepare the source. For Harbor, package task directories containing task.toml, instruction.md, and tests/test.sh. For HUD, package the source taskset and register each HUD Task data row with its prepared environment image.
2. Declare every grader. Supply a native grader version, metric, and threshold for each task. Add task-specific rubric and vision criteria where needed. HUD vision tasks declare final.png; Harbor paths refer to the retained trial artifact directory.
3. Freeze and register. Create a tar.gz archive, compute its SHA-256, and validate the manifest with the CLI. Registration checks archive containment, source integrity, task contracts, and placement compatibility.
4. Run and compare. Select the new registry entry, choose task coverage and scaffolds, and submit a bounded run. Keep its run ID for recovery or downloads.

# Generate the registry schema
uv run --project services/benchmark-runner python -m \
  blobfish_benchmark_runner.cli schema --output benchmark-schema.json

# Register a validated manifest and its frozen source archive
uv run --project services/benchmark-runner python -m \
  blobfish_benchmark_runner.cli register benchmark.json tasks.tar.gz \
  --bucket YOUR_BENCHMARK_BUCKET

WebVoyager and Mind2Web are not automatically executable just because a dataset is downloaded. Supply the task environment, supported browser tools, site metadata, grading criteria, and references for the specific version. The live registry shows whether that prepared package is available.

For existing evaluations, import report.json in the workspace. Download the example JSON there for the reporting schema. Import happens entirely in the browser.

Runtime formats: HUD Python and Harbor task format.

Rerun after a change

Register a new benchmark revision when task definitions or graders change. Submit a new run when the model or scaffold changes. Transport retries reuse the original request ID; execution failures are retained and are never silently replaced by a successful retry.