Every score has a task behind it.

Astra grading and executable checks

Independent model grading uses GPT-6 Astra with maximum reasoning and frozen task-specific criteria. Graders receive the original task and retained evidence in a fresh session. They do not accept the solving model’s claim of success as proof. Browser reports show DOM rubric and screenshot judgments separately, with their exact evidence scope.

Coding and environment benchmarks retain their native executable checks alongside any model review. Historical runs keep their original grader versions. Each report names its actual grader; missing or failed reviews remain ungraded.

What a managed run records

Managed environment runs freeze the benchmark version, source archive digest, selected task IDs, environment image digests, agent scaffolds, model IDs, grading rules, and execution limits. Changing any of these creates a new evaluation. Different models and scaffold configurations can run against the same selected tasks.

The benchmark directory is the source of available managed task packages. Controls verify the infrastructure and are labeled as controls. A one-task check does not establish full-suite performance. Existing managed run IDs remain recoverable through their original owner credentials.

Managed environment execution on GKE

Each managed task attempt gets a fresh gVisor environment. A durable Kubernetes job owns the managed run, so it continues when a browser closes. Task environments have no Kubernetes credentials, no host mounts, bounded CPU and memory, and an expiration deadline. Network policies block private networks, cluster services, and cloud metadata.

The execution lane accepts prepared CPU Linux environments and the qualified main/world service layout used by DevOpsBench. Other Compose layouts, GPUs, and dynamic hostname allowlists need a separately qualified runtime. Unsupported declarations are rejected during registration or task admission.

Published imports can originate from external browser providers. Each report identifies where its agent executed and which reviews ran separately. The Browser Use WebVoyager pilot uses hosted Browser Use sessions; its supplemental vision review runs in isolated GKE grader pods.

Subscription agents and matched pilots

Codex and Claude Code use connected subscription accounts in separate worker environments. DeepSeek uses its native API. All three receive the same task tools and bounded terminal access in their assigned environment. DevOpsBench exposes its original 97 world tools alongside the terminal; the report retains the exact tool catalog and every recorded call. Account credentials never enter task containers. Each task starts a fresh session; provider failures remain visible and do not trigger an API or account fallback.

The first matched pilots select ten fixed tasks each from HumanEval, MBPP and DevOpsBench-100. HumanEval and MBPP use original assertions in a separate verifier environment. These are agentic, single-attempt evaluations with terminal tools, not full-suite code-only leaderboard scores. DevOpsBench keeps the original state verifier and discloses its mix of source train and held-out tasks.

Three distinct grading dimensions

Native task grade

HUD returns its tasks.grade receipt. Harbor runs the task’s tests/test.sh and reads the configured reward metric. Every task must declare its grader version and pass threshold. Missing receipts and execution errors never supply a passing verdict.

Rubric evaluation

An optional, versioned judge evaluates task-specific criteria against the prompt, answer, and reference. Every criterion must be met for a pass. Its verdict, explanation, model, and token usage remain separate from the native score.

Vision evaluation

An optional visual judge evaluates captured PNG evidence against the task’s criteria. HUD captures the final RFB display; Harbor uses declared screenshot artifacts. Missing screenshots or invalid judge responses produce an evaluation error.

Rubric and vision are model judgments, not deterministic proof. Rubric and vision graders can use a connected Codex subscription account with Astra, or an explicitly configured API credential. Native executable graders remain independent. An unconfigured dimension is reported as “not configured,” and a failed evaluator stays ungraded.

Reading the report

Task success uses the run’s declared primary evaluator: native, rubric or vision. Registry-submitted GKE runs use native grading. Separately published browser pilots declare their own primary; the Online-Mind2Web Harbor pilot uses a frozen DOM rubric and a separate screenshot review. Success counts successful recorded attempts, including execution errors in the denominator. Reliability measures completion without execution errors. The report shows selected coverage alongside source-suite coverage. Missing measurements remain unknown.

Rubric and vision pass rates require complete grading coverage. The evaluator summary also shows passes, graded tasks, and total selected tasks for each site. Cost includes only measured provider usage; judge usage is retained separately. Task latency includes setup, agent execution and native verification. Additional rubric/vision grading time is recorded separately. Subscription usage is recorded without inventing a dollar cost. Imported reports declare their timing and cost scope in the run notes.

When the native verifier records individual checks, the task explorer shows each result and explanation, the recorded tool-call count, and any partial reward. The original pass/fail verdict still determines task success. The separate native-check download identifies the original task archive and verifier-file checksums.

Competitive rankings require matching task sets, dataset version, grader contract, environment, and budgets. Imported results retain their submitted provenance; matching declarations do not mean Blobfish independently reran them.

Your files

Managed runs are private to the owning Blobfish API key. Endpoint reports stay in your tab until you download them. Selected evaluations are published separately with fixed artifact checksums; private run submission never publishes a report. Submitted provider credentials are redacted from retained artifacts and removed with the temporary job secret.

Add a benchmark

Open the dataset recipe builder to pin Hugging Face or GitHub data files, map task fields, and freeze a reproducible selection. Compile the selected rows through a reviewed HUD or Harbor runtime and task-success grader.

Registration is an operator workflow. Run callers select registered task packages; they do not upload executable controller code. Prepare and publish environment images, then pin their digests in the manifest. The runner does not build images during an evaluation.

1. Prepare the source. For Harbor, package task directories containing task.toml, instruction.md, and tests/test.sh. For HUD, package the source taskset and register each HUD Task data row with its prepared environment image.
2. Declare every grader. Supply a native grader version, metric, and threshold for each task. Add task-specific rubric and vision criteria where needed. HUD vision tasks declare final.png; Harbor paths refer to the retained trial artifact directory.
3. Freeze and register. Create a tar.gz archive, compute its SHA-256, and validate the manifest with the CLI. Registration checks archive containment, source integrity, task contracts, and placement compatibility.
4. Run and compare. Select the new registry entry, choose task coverage and scaffolds, and submit a bounded run. Keep its run ID for recovery or downloads.

# Generate the registry schema
uv run --project services/benchmark-runner python -m \
  blobfish_benchmark_runner.cli schema --output benchmark-schema.json

# Register a validated manifest and its frozen source archive
uv run --project services/benchmark-runner python -m \
  blobfish_benchmark_runner.cli register benchmark.json tasks.tar.gz \
  --bucket YOUR_BENCHMARK_BUCKET

WebVoyager and Mind2Web are not automatically executable just because a dataset is downloaded. Supply the task environment, supported browser tools, site metadata, grading criteria, and references for the specific version. The benchmark directory shows whether that prepared package is available.

For existing evaluations, import report.json in the workspace. Published reports also include a compatible JSON download. Import happens entirely in the browser.

Runtime formats: HUD Python and Harbor task format.

Rerun after a change

Register a new benchmark revision when task definitions or graders change. Submit a new run when the model or scaffold changes. Managed transport retries reuse the original request ID. Execution failures are retained and are never silently replaced by a successful retry.