raw_outputs.jsonlInputs, outputs, timing, scores, metadata and retained evidence for every selected task.
Dataset preparation
Bring the tasks.
Keep the evidence.
Freeze a source version, choose a task set, and connect its runtime and graders. Use the same package to compare agent configurations.
01 Select data02 Attach graders03 Run on GKE04 Export results
01 / Source recipe
Choose your dataset.
This preset pins AllenAI’s public GitHub copy of the 300-task Online-Mind2Web benchmark. Its source revision is recorded separately from the current official dataset. You can also choose Hugging Face and supply an authorized source revision. Inspect the frozen GKE pilot and its task evidence →
Map the task fields
Use column names or dotted paths such as task.prompt. Reference answers stay out of the agent’s input files.
02 / Runtime and grading
Attach an executable task template.
Every runnable task needs a prepared environment and explicit success criteria. Registered HUD and Harbor packages use their native task graders. The browser pilot declares a rubric primary and a separate vision review.
Browser profiles require a reviewed browser success evaluator. The bundled exact-answer example cannot grade live-web navigation tasks. Screenshots alone and a provider’s success flag do not establish task success.
Downloads JSON; does not provision compute.
Prepare and register from your checkout
# From a Blobfish repository checkout; inspect current HF files with: # uv run --project packages/benchmark-datasets --extra huggingface bf-dataset inspect-hf OWNER/DATASET # Freeze source rows without running a model uv run --project packages/benchmark-datasets --extra huggingface --extra parquet \ bf-dataset prepare benchmark-recipe.json --output prepared-data # After adding a digest-pinned image to the recipe, attach a reviewed HARBOR template uv run --project packages/benchmark-datasets --extra huggingface --extra parquet \ bf-dataset prepare benchmark-recipe.json \ --template path/to/reviewed-browser-template --output prepared-benchmark # Validate and register in your configured benchmark bucket uv run --project services/benchmark-runner python -m blobfish_benchmark_runner.cli \ register prepared-benchmark/benchmark.json prepared-benchmark/tasks.tar.gz \ --bucket YOUR_BENCHMARK_BUCKET
The compiler preserves source IDs, file hashes, selected tasks and template identity in dataset.lock.json. Registration validates the resulting archive against the runner’s schema. Select the registered package in the leaderboard to start a bounded run.
Connect a browser scaffold
The agent, browser and grader are separate choices.
A dataset recipe prepares tasks. A reviewed runtime supplies the browser tools and agent adapter. The current runner’s registry shows which complete combinations you can execute.
| Component | What the task runtime needs |
|---|---|
| Browser Use, TinyFish, Stagehand, Skyvern | An agent adapter that records its configuration, final answer, actions and screenshots. |
| Browserbase, Steel | A selected agent or scaffold alongside the browser session provider. |
| Playwright | An agent that chooses actions through the browser tools, with an independent success evaluator. |
| Firecrawl | An extraction or navigation adapter whose supported actions fit the selected tasks. |
These are integration targets, not a claim that each provider is installed or has a measured score. A completed session is not evidence that its task succeeded.
From each run
Results you can take with you.
results_webvoyager_eval.csvSeparate native, rubric and vision verdicts, including each evaluator’s reasoning.
summary_webvoyager_eval.mdPer-site statistics, overall coverage, latency and measured cost availability.
Export filenames follow the benchmark ID. Open the resulting report in the workspace to explore tasks and download a standalone HTML report.
Read the grading and comparison methodology →