Blobfish AI / Independent evaluations
Different benchmarks. Real task outcomes.
Compare models, understand failures, and inspect every task.
Matched tasks and tools. Independent Astra grading. Recorded evidence.
Measured evaluations
Every result links to its task coverage, grading protocol and evidence.
Ten frozen HumanEval tasks, attempted once by ChatGPT and Claude through their subscription CLIs and by DeepSeek through its API. Native tests determine success; one Astra attempt reached the shared time limit before verification completed.
Task success · matched pilot
10/164 source tasks · 2026-09-16
Harbor · GKE
Ten matching tasks from the official MBPP test split, attempted once by ChatGPT and Claude through their subscription CLIs and by DeepSeek through its API. All three passed the original native tests on these selected tasks.
Task success · matched pilot
10/500 source tasks · 2026-09-16
Harbor · GKE
15 original tasks across 15 websites. Separate evidence review supports 10 outcomes; every failure and output is retained.
Task success · single-agent pilot
15/643 source tasks · 2026-09-16
Browser Use Cloud · GKE vision review
3/10 tasks passed the DOM rubric; 3 execution errors. The original GitHub snapshot, trajectories, screenshots and separate vision verdicts are retained.
Task success · single-agent pilot
10/300 source tasks · 2026-09-16
Harbor · GKE
Frozen tasks, isolated environments and retained execution evidence.
Compare Claude, ChatGPT and DeepSeek on the same task environments and budgets.
Inspect outcomes, latency, recorded costs and why individual tasks failed.
The benchmark directory
Browse by capability, company fit, or source catalog.
Catalog snapshot: 2026-09-15. Exact source matches are joined; versions and ports remain distinct. This includes every entry reviewed in our research, not every dataset on Hugging Face. Catalog inclusion and company fit do not imply a hosted integration or a measured result.
428 benchmark entries
End-to-end navigation and task completion on live websites; original release has 643 tasks across 15 sites.
For Browser Use, Browserbase / Stagehand, Firecrawl +4
300 real-world tasks spanning 136 websites; maintainers update invalid tasks.
For Browser Use, Browserbase / Stagehand, Firecrawl +4
Multi-step commerce, forum, CMS and developer-site workflows in resettable websites.
For Automation Anywhere, Browser Use, Browserbase / Stagehand +6
Browser tasks requiring visual understanding, including images and visually grounded shopping.
For Anthropic, Browser Use, Browserbase / Stagehand +3
ServiceNow knowledge-work tasks and compositional enterprise workflows.
For Automation Anywhere, Browser Use, Browserbase / Stagehand +3
Desktop tasks involving applications, files and cross-application workflows.
For Anthropic, Automation Anywhere, Google DeepMind +2
Long-horizon computer-use tasks with coordinated code, assets and mocked websites.
For Anthropic, Automation Anywhere, Google DeepMind +2
Locate the correct UI target in professional, high-resolution screenshots.
For Anthropic, Automation Anywhere, Google DeepMind +2
116 parameterized task templates across 20 Android apps.
For Anthropic, Google DeepMind, OpenAI +2
Resolve 500 human-validated GitHub issues.
For All Hands / OpenHands, Augment Code, Cline +6
Long-horizon repository-level software tasks; public Harbor snapshot has 731 tasks.
For All Hands / OpenHands, Augment Code, Cline +7
Repository issue resolution across nine programming languages, 300 test instances.
For All Hands / OpenHands, Augment Code, Cline +7
89 difficult terminal tasks across engineering, systems and scientific work.
For All Hands / OpenHands, Anthropic, Augment Code +9
Current continuous Terminal-Bench release with task and resource revisions; catalog snapshot lists 66 tasks.
For All Hands / OpenHands, Anthropic, Augment Code +9
Competition-style coding, with date-bounded problem releases.
For Alibaba Qwen, Anthropic, Cognition / Devin +7
225 programming exercises across six languages.
For Augment Code, Cline, Cognition / Devin +4
Python programming tasks using real libraries and APIs.
For All Hands / OpenHands, Augment Code, Cline +8
Small Python function synthesis with expanded tests.
For Alibaba Qwen, All Hands / OpenHands, Anthropic +17
113 original long-horizon repository tasks across five programming languages.
For All Hands / OpenHands, Augment Code, Cline +7
Codebase understanding, test authoring and refactoring as separate tracks.
For Augment Code, Cursor, Factory +3
100 software engineering tasks on Android repositories.
For Augment Code, Cognition / Devin, Cursor +3
46 tasks testing sustained useful work over hundreds of steps.
For All Hands / OpenHands, Augment Code, Cline +6
Policy-following conversations, tool actions and cooperative customer-service tasks.
For Ada, Decagon, Forethought +5
Customer service in airline, retail, telecom and banking-knowledge domains; Harbor lists 375 tasks.
For Ada, Decagon, Forethought +6
For teams building agents
We help teams evaluate browser, coding, voice, legal, search and long-horizon agents against real work.
