Models

Hosted models

GET /api/v1/models returns this catalog and, with your API key, which credential would serve each provider: "account" (a key you saved), "server" (the operator’s key), or null. Pickers never offer a dead option.

Whose key runs it

Precedence for every run: an api_keyon the request (used once, never stored) → the provider key saved under your account → the server’s own key. A named model that cannot be served honestly is refused with model_unavailable; nothing is silently swapped for a cheaper policy.

$ curl -sX POST https://blobfish.ai/api/v1/accounts/providers \
    -H 'X-API-Key: bf_YOUR_KEY' -H 'Content-Type: application/json' \
    -d '{"provider":"xai","api_key":"xai-…"}'

# Your own OpenAI-compatible endpoint (vLLM, SGLang, TGI, a fine-tune behind a proxy)
$ curl -sX POST https://blobfish.ai/api/v1/accounts/providers \
    -H 'X-API-Key: bf_YOUR_KEY' -H 'Content-Type: application/json' \
    -d '{"provider":"custom","base_url":"https://vllm.example.com/v1","api_key":"…","default_model":"my-finetune"}'

Custom endpoints must be https on a public host and speak /chat/completions with tool calling. Any /v1/chat/completions server with tool calling: vLLM, SGLang, Ollama (OpenAI mode), TGI, a fine-tune behind a proxy. Point it at the twin and the same tool ledger and verifiers grade it. Once saved, --provider custom without --base-url (or a request carrying only provider: "custom") is served by that saved endpoint; a request that carries base_url must also carry api_key, and with neither saved nor sent the run is refused with no_custom_endpoint.

Run a twin with a model

$ blobfish models
$ blobfish twin-runs run twe_<id> --model claude-sonnet-5 --prompt "Find the overdue invoices and pay the smallest one"
$ blobfish twin-runs run twe_<id> --model gpt-5-mini --api-key-env OPENAI_API_KEY --prompt "…"
$ blobfish twin-runs run twe_<id> --model grok-4 --prompt "…"
$ blobfish twin-runs run twe_<id> --model deepseek-v4-flash --max-steps 12 --prompt "…"
$ blobfish twin-runs run twe_<id> --provider custom --base-url https://vllm.example.com/v1 --api-key-env MY_MODEL_KEY --model my-finetune --prompt "…"
# after saving a custom endpoint under your account, omit --base-url: the saved endpoint serves the run
$ blobfish twin-runs run twe_<id> --provider custom --model my-finetune --prompt "…"

$ curl -sX POST https://blobfish.ai/api/v1/twin/environments/twe_<id>/runs \
    -H 'X-API-Key: bf_YOUR_KEY' -H 'Content-Type: application/json' \
    -d '{"model":"grok-4","prompt":"List the open opportunities","max_steps":8,"max_tokens":2000,"timeout_seconds":120}'

The run drives the environment’s real tools in an isolated session, appends every call to the ledger, and returns the steps, the final answer, the finish reason, and credential_source. --api-key-env NAME reads the key from that environment variable; a key literal is never accepted on the command line.

In Studio or the runs API, set max_tokens to limit each model response (1–32,768; default 2,000) and timeout_seconds for each rollout (1–240; default 240). The deadline includes hosted tool discovery and cancels in-flight model and gateway requests. Batches shorten each rollout’s effective timeout to fit the remaining request budget. Inspect the saved run for both settings and its actual finish reason, including timeout. Cancellation is stored separately from the result, so a late server response cannot restore a cancelled run or count it as a pass. Unsupported settings, including temperature, reasoning effort, mode, and baseline digest, return a validation error.

Run the tasks — graded by the twin’s own verifiers

Every website-plane environment carries a task set: deterministic tasks minted from the twin’s own world by the same generator and executable VCode gate the training kit uses (or the world’s own tasks when it ships with verifiers). A run started with task_idis graded after it finishes — the run’s isolated session database is snapshotted before and after, and the task’s verifier executes in the sandbox runtime on the two states plus the call trace. The record carries score (0..1), passed, and the verifier detail. No LLM judge anywhere; verifier code and answer keys are never returned.

$ curl -sS https://blobfish.ai/api/v1/twin/environments/twe_<id>/tasks -H 'X-API-Key: bf_YOUR_KEY'
# → {"count": 8, "tasks": [{"task_id": "task_001", "instruction": "…", "difficulty": "medium", "kind": "state_change", "grades": ["state_diff"], "assertions": 3}, …]}
$ curl -sX POST https://blobfish.ai/api/v1/twin/environments/twe_<id>/runs \
    -H 'X-API-Key: bf_YOUR_KEY' -H 'Content-Type: application/json' \
    -d '{"model":"deepseek-v4-flash","task_id":"task_001","max_steps":8}'
# → {"run_id": "twrun_…", "task_id": "task_001", "status": "completed", "score": 1, "passed": true, "verifier": {"kind": "vcode", "assertions_passed": 3, "assertions_total": 3, …}, …}
# a bounded batch (≤10 tasks per request, sequential, inside the route's inline budget):
$ curl -sX POST https://blobfish.ai/api/v1/twin/environments/twe_<id>/runs \
    -H 'X-API-Key: bf_YOUR_KEY' -H 'Content-Type: application/json' \
    -d '{"model":"grok-4","task_ids":"all","max_tasks":5}'
# → {"batch_id": "twbat_…", "runs": [...], "skipped": [], "aggregate": {"tasks": 5, "completed": 5, "graded": 5, "passed": 3, "mean_score": 0.6}}
$ curl -sX POST https://blobfish.ai/api/v1/twin/environments/twe_<id>/runs/twrun_<id>/cancel -H 'X-API-Key: bf_YOUR_KEY'

$ blobfish twin-runs tasks twe_<id>
$ blobfish twin-runs run twe_<id> --model claude-sonnet-5 --task task_001
$ blobfish twin-runs run twe_<id> --model deepseek-v4-flash --tasks all --max-tasks 5
$ blobfish twin-runs cancel twe_<id> twrun_<id>

The platform MCP server exposes the same set as list_twin_tasks and accepts task_id / task_ids on run_model_on_twin, so a coding agent can score a model on a twin without leaving its session. A run still runningpast the route’s inline budget reads as failed (stale: true) rather than running forever; POST …/runs/{run_id}/cancel stops one early. Cluster-plane (tenv_) environments carry no world to mint tasks from and run with a prompt only.

Train an open-weight model on a twin

Choose a base from the open-weight catalog (default Qwen/Qwen3-8B), build a training kit from the environment, and run SFT or GRPO with blobfish train locally (Apple Silicon via MLX, or a CUDA box) or on RunPod. Launch is always yours — the site never starts a paid GPU on your behalf.

$ blobfish twin-runs train twe_<id> --base Qwen/Qwen3-8B --method grpo --target local-mlx
# → downloads ./twin-train-<job>/ and prints the exact command, e.g.
$ blobfish train --world ./twin-train-<job>/world --data ./twin-train-<job>/sft.jsonl --base Qwen/Qwen3-8B --method grpo --target local-mlx --out ./out

Details, kit contents, and the honesty rules for minted tasks are in Training.

Score your model on a public benchmark

$ blobfish benchmark run counselbench-100 --model grok-4 --api-key-env XAI_API_KEY
$ blobfish benchmark run salesbench-100 --provider custom --base-url https://vllm.example.com/v1 --api-key-env MY_MODEL_KEY --model my-finetune
$ blobfish benchmark run salesbench-100 --provider custom --model my-finetune   # the custom endpoint saved under your account

A hosted run scores the benchmark’s public sandbox task with your model; the sandbox grader — never the model — produces the score, checks, and strict pass. It is not a leaderboard entry. The response includes the exact Harbor command to run the frozen 100-task suite with the same model. See Benchmarks. To run every hosted frozen task as a background suite job and publish the result as a community run below the official leaderboard, see Benchmark runs.

Next

Quickstart · Training · Gateway · Twin reference