Guides

SDK evaluations

Run your environment with the Python SDK or CLI. Publish its completed rollouts to your account, inspect the recorded steps, and export the same traces for post-training.

Connect your account

Sign in and create a key in API keys. Set it as BLOBFISH_API_KEY in your local secret manager or shell environment, then verify it with blobfish login. Your model provider key is a separate credential.

Install the CLI and SDK

uv tool install --with "blobfish-sdk @ https://blobfish.ai/sdk/blobfish_sdk-0.6.2-py3-none-any.whl" "blobfish-cli @ https://blobfish.ai/sdk/blobfish_cli-0.1.13-py3-none-any.whl"

These downloadable wheels work while PyPI publisher configuration is pending. Use SDK 0.5.4 and CLI 0.1.3 or later for dashboard publication. A key created with an unverified contact email stays in its own scope; create the key while signed in to see its evaluations in your browser account.

Run three tasks

Set ANTHROPIC_API_KEY for Claude, or choose another configured provider. The scaffold supplies a letter-count environment and three tasks.

Evaluate and publish

blobfish login
blobfish init letter-count
cd letter-count
blobfish eval tasks.py --model claude
blobfish runs list --remote --json

The CLI evaluates all three tasks, saves local evidence, and publishes the completed job when a Blobfish key is available. A dashboard URL appears after the server acknowledges the upload. Rewards come from the actual grader.

For Python agents, publish the completed Job explicitly:

Python SDK

job = await tasks.run(my_agent, group=2, concurrency=4)
receipt = await job.publish(metadata={"model": "my-model"})
print(receipt["dashboard_url"])

Replay and export

Open Evaluations and select the uploaded job. Choose a rollout, move between recorded steps, and inspect the prompt, final answer, reward, structured grade, and captured training samples. Replay does not repeat tool calls.

Inspect retained evidence

blobfish runs inspect <evaluation-id> --json
blobfish runs replay <evaluation-id> --json

Use Export JSON or Export traces JSONL on the run, or filter jobs in Datasets and export the selected traces. Token IDs, sampling log probabilities, and lineage are retained when the agent captured them. Missing measurements stay absent.

Published jobs are private and labeled as executed and graded by your SDK client. Publication does not certify a benchmark or launch remote compute.

Retry without rerunning

An upload failure leaves local results available. Retry with the local job ID printed by the CLI:

Retry publication

blobfish runs publish <local-job-id> --json

Identical retries return the same evaluation. Changed evidence under an existing job ID is rejected. Set BLOBFISH_PUBLISH_EVALUATIONS=0 to keep CLI evaluations local. Set BLOBFISH_BASE_URL to an HTTPS origin, or HTTP on loopback, for a self-hosted dashboard.

Each publication accepts one completed job, 1–256 rollouts, and at most 1 MiB of JSON. Larger jobs remain local; split the taskset into smaller jobs before publishing.

Create a dataset through Studio

Open Studio and ask: “Find my completed evaluations and export these run IDs as a private JSONL dataset, including successes and failures.” Studio can select published SDK evaluations, synchronized training artifacts, recorded Twin runs, and benchmark runs. For a Twin or benchmark, name its environment or benchmark as well.

The conversation keeps a dataset receipt with its record count, byte count and SHA-256 checksum. Expand Create dataset and choose Download dataset (.jsonl). The download requires the same account, verifies its checksum, and remains available after refresh.

Exports retain recorded rewards, verifier details, tool observations and training samples when present. Benchmark suites contain their stored task summaries. This action does not manufacture missing observations, SFT messages, preferences or token masks.

Select 1–50 source IDs, with up to 500 records and 2 MiB per export. Unsupported formats, running Twin or benchmark jobs, missing records and oversized selections are rejected. An interrupted export can resume with the same request and selection; it does not rerun the model.

HTTP API

Private evaluation API

POST /api/v1/evaluations
GET /api/v1/evaluations?limit=50&offset=0
GET /api/v1/evaluations/<evaluation-id>
GET /api/v1/evaluations/<evaluation-id>?format=jsonl

Authenticate with your account session or X-API-Key. POST accepts a blobfish.sdk-evaluation.v1 envelope with a blobfish.job.v1 job and its complete blobfish.trace.v1 traces. The list endpoint returns summary records and a next_offset; detail and JSONL return full evidence. Secret fields are redacted before storage and export.