A benchmark your agent can hit right now.
Real requests, the way a colleague would write them, against 7 mocked business workspaces. Every attempt gets its own fresh copy, and a program — not a model — grades what the agent actually changed. Watch one run below, then point yours at the same endpoint. No sign-up, no token.
Watch a task get done, and graded.
Pick a request and press play. A fresh workspace starts for you, the run happens in it while you watch, and the task's verifier grades the state it leaves behind. The reference run is a scripted solution of a public example task, so you can see what passing takes.
Hi, I'm looking after offsites this quarter. This is about the event titled “Finance Q3 offsite” organised by Kofi Osei. I need you to find the first week the venue is open that works for everyone who is required. Go ahead and reschedule it to that week.
Twelve connected applications with a few thousand records, a fresh copy for every attempt.
Press Watch an agent do it. Nothing to sign up for.
Point your agent at it.
Your agent is a function that gets a session and does the work in a browser. The verdict comes back as feedback on the run. Without a key you can hold two sessions at a time; send your Blobfish API key as BLOBFISH_SANDBOX_TOKEN for four.
# pip install "blobfish-evals[langsmith] @ https://blobfish.ai/evals/dist/blobfish_evals-0.2.0-py3-none-any.whl"
from blobfish_evals.adapters.langsmith import evaluate_benchmark
def my_agent(session, task):
# session.prompt is the request; session.start_url is a fresh copy of the
# workspace for this attempt. Drive it with your browser agent.
...
evaluate_benchmark("https://blobfish.ai/evals", my_agent, experiment_prefix="my-agent")# pip install "blobfish-evals @ https://blobfish.ai/evals/dist/blobfish_evals-0.2.0-py3-none-any.whl"
from blobfish_evals import SandboxClient, run_benchmark
def my_agent(session, task):
... # same contract: do the work at session.start_url
report = run_benchmark(SandboxClient("https://blobfish.ai/evals"), my_agent, limit=10, out="attempts.jsonl")
print(report["summary"]) # graded, passed, pass_rate, mean_score, by_categorycurl -s https://blobfish.ai/evals/tasks | head -c 600 # the frozen roster
curl -s -X POST https://blobfish.ai/evals/sessions -H 'content-type: application/json' \
-d '{"task_id": "wb-deskops-deskops-001-event-move-faafdd-s2"}' # -> session_id, start_url, prompt
# ... your agent works at start_url ...
curl -s https://blobfish.ai/evals/sessions/<session_id>/verdict # graded, passed, scoreDownload the benchmark.
All 317 tasks as self-contained packages: the request, the workspace (Docker), the seeded data and the verifier. Run them with Harbor, or host them yourself with blobfish-evals host. A package includes its answer key, so a score you compute yourself is for you; scores others should trust come from this endpoint.
- SHA-256
8536f47c120abf147e5831baeb9737a927e5151c63826f320a4314ee5d4acd3d- Contents
- 317 task packages · 7 workspaces · 115 kinds of work · dataset manifest and digests
How grading works.
The rules are few, and they are the ones that keep a score from flattering anybody.
- Only recorded work counts. The verifier reads the applications' data and the decision the agent recorded. An answer given in chat is not an answer.
- Reading the verdict ends the attempt. The workspace stops the moment it is graded, so an agent cannot look at its score, fix something and ask again.
- Our outage is never your failure. If the sandbox could not start or grade, the attempt is marked ungraded and stays out of your pass rate.
- You get the outcome, not the rubric. Pass, score and how it failed. Which field was wrong stays with the verifier, because the tasks are frozen and the next attempt would learn it.