Benchmark runs

The single-task run

POST /api/v1/benchmarks/{slug}/runs with the model request shape (Models) and an optional max_steps(1–40, default 24) runs the benchmark’s public sandbox task inline, bounded at four minutes, and answers 201with the record: the grader’s score, strict_pass, passed_checks/total_checks, the recorded tool calls, the session evidence_url, and the exact full_suite_command to run the frozen release on Harbor with the same model. A bf_ key is required; a provider api_key on the request serves this run only and is never stored.

$ curl -sX POST https://blobfish.ai/api/v1/benchmarks/counselbench-100/runs \
    -H 'X-API-Key: bf_YOUR_KEY' -H 'Content-Type: application/json' \
    -d '{"provider":"xai","model":"grok-4","api_key":"xai-…","max_steps":24}'
# → 201 { kind: "task", run_id: "bmr_…", score, strict_pass, steps[], evidence_url, full_suite_command }

The same request is what the Run your model panel on each benchmark page sends when What to run is the public sandbox task.

The suite run

Add suite: "sample" (the first 10 hosted tasks, in bundle order) or suite: "full" (every hosted task) to the same POST and the server starts a background job instead: it answers 202 with the queued bmrs_ record, a Location to poll, and Retry-After: 5. max_tasks (1–100) bounds either suite. Each task runs through the same core as a single run — a fresh isolated session, the real MCP tools, the grader’s verdict — one at a time, and the record is updated after every task.

$ curl -si -X POST https://blobfish.ai/api/v1/benchmarks/counselbench-100/runs \
    -H 'X-API-Key: bf_YOUR_KEY' -H 'Content-Type: application/json' \
    -d '{"provider":"xai","model":"grok-4","api_key":"xai-…","suite":"sample","max_steps":24}'
# HTTP/1.1 202 Accepted
# Location: https://blobfish.ai/api/v1/benchmarks/counselbench-100/runs/bmrs_…
# Retry-After: 5

$ RUN=bmrs_…
$ while :; do
    curl -s https://blobfish.ai/api/v1/benchmarks/counselbench-100/runs/$RUN -H 'X-API-Key: bf_YOUR_KEY' > run.json
    jq -r '"\(.status) \(.progress.done)/\(.progress.total)"' run.json
    jq -e '.status == "queued" or .status == "running"' run.json > /dev/null || break
    sleep "$(jq -r '.poll_after_seconds // 5' run.json)"
  done

Every poll answers the whole record:

GET /api/v1/benchmarks/{slug}/runs lists your single runs (runs) and suite jobs (suite_runs) for that benchmark, newest first. One suite job runs per account at a time: a second POST answers 409 run_in_progress with the run_id to follow.

Publish it as a community run

A completedsuite run with at least one graded task can be published, once, onto the benchmark page’s community board — below the official leaderboard and never merged into it. The row carries your model, whose credential served it, the suite, tasks/graded/passed, mean score, pass rate, coverage, and the timestamps; every row is badged verbatim:

self-run · graded by the benchmark's own verifier · not the frozen official suite unless suite:full

$ curl -sX POST https://blobfish.ai/api/v1/benchmarks/counselbench-100/runs/$RUN/submit \
    -H 'X-API-Key: bf_YOUR_KEY' -H 'Content-Type: application/json' \
    -d '{"display_name":"grok-4 · my scaffold","notes":"temperature 0, 24 turns"}'   # ≤80 / ≤500 chars
# → 201 { community_run_id: "bcr_…", mine: true, badge, coverage, mean_score, pass_rate, links: { run, community, page, delete } }

$ curl -s https://blobfish.ai/api/v1/benchmarks/counselbench-100/community            # public; best mean score first
$ curl -s https://blobfish.ai/api/v1/benchmarks/counselbench-100/community -H 'X-API-Key: bf_YOUR_KEY'   # same board, your rows carry mine: true
$ curl -sX DELETE https://blobfish.ai/api/v1/benchmarks/counselbench-100/community/bcr_… -H 'X-API-Key: bf_YOUR_KEY'
# → 200 { deleted: true }; the suite run may be submitted again

Refusals are named: 409 not_a_suite_run for a bmr_ single-task id, 409 suite_run_not_completed while the job is still running or ended failed/interrupted, 409 suite_run_not_graded when no task was graded, 409 already_submitted (with the existing community_run_id) for a second submit, and 400 invalid_submission for an over-long name or notes. Only the key that started the run can submit or delete; a stranger reads 404. The Run your model panel does all of this from the page: pick a suite, watch it poll, press Publish as a community run, and the board below it refreshes.

CLI

The single-task verb blobfish benchmark run <slug> --model … exists today (Models). The suite and community shapes below land with the CLI lane; until that release is served, use the API calls above — the CLI verbs are thin wrappers over exactly those routes.

$ blobfish benchmark run <slug> --model <model> --suite sample|full [--max-tasks N]   # POST …/runs with suite, then polls links.self
$ blobfish benchmark runs <slug>                                                     # GET …/runs → runs + suite_runs
$ blobfish benchmark submit <slug> <run_id> [--display-name NAME]                    # POST …/runs/<run_id>/submit

The honesty boundary

1. Hosted tasks vs the official suite. A hosted suite run scores the frozen tasks this sandbox carries with your model — every task graded by the benchmark's own verifier. coverage.official_suite says whether it covered every official task; anything less is a self-run, not the frozen leaderboard. Today each first-party bundle carries one reviewed task of its 100-task release, so coverage.hosted_tasks is what this host can run and coverage.official_suite is false for it. The panel and the board print that from the record; the leaderboard stays the Harbor run of the frozen release.
2. One job per account. A second suite POST while one is in flight answers 409 run_in_progress naming the running run_id; follow it instead.
3. Interrupted, not silently running. The runner is in-process. If the process that owned a job is gone or its heartbeat is stale, the job reads interrupted with error.kind: "interrupted" and the tasks graded so far — never as still running, never as a score. Start a new run.
4. A rejected credential stops the suite. An auth failure on one task fails the job and the remaining tasks are not attempted; the record says so.
5. Keys are never stored. The bf_ key and any provider key travel on the request only; records carry the model name and credential_source (request | account | server), nothing secret. Session evidence expires after 30 minutes; recorded scores do not.

Next

Benchmarks · Models · Twins quickstart · API reference