Coding and terminal agents / Benchmark leaderboard

HumanEval+ / MBPP+ (EvalPlus)

Small Python function synthesis with expanded tests.

No published company result yet.

This benchmark is in the research directory. Its task package, adapter and grading protocol need qualification before a hosted run can be offered.

Pass@1 / pass@k.

Lightweight isolated Python evaluation.

Good regression checks, weak standalone evidence for modern autonomous coding; public-data exposure and saturation limit differentiation.

Catalog entry. Request a managed evaluation to qualify your agent interface and the benchmark’s native grading requirements.

Research recommendations based on product capabilities. These companies have not necessarily run this benchmark or integrated with Blobfish.

Compatibility notes for each company

Alibaba Qwen: Capability-aligned; adapter/access to qualify

All Hands / OpenHands: Capability-aligned; adapter/access to qualify

Anthropic: Capability-aligned; adapter/access to qualify

Augment Code: Capability-aligned; adapter/access to qualify

Cline: Capability-aligned; adapter/access to qualify

Cognition / Devin: Capability-aligned; adapter/access to qualify

Cohere: Direct component API + selected system adapters

Cursor: Capability-aligned; adapter/access to qualify

DeepSeek: Capability-aligned; adapter/access to qualify

Factory: Capability-aligned; adapter/access to qualify

GitHub Copilot: Capability-aligned; adapter/access to qualify

Google DeepMind: Capability-aligned; adapter/access to qualify

JetBrains: Capability-aligned; adapter/access to qualify

Meta: Capability-aligned; adapter/access to qualify

MiniMax: Capability-aligned; adapter/access to qualify

Mistral: Capability-aligned; adapter/access to qualify

OpenAI: Capability-aligned; adapter/access to qualify

Replit: Capability-aligned; adapter/access to qualify

Z.ai: Capability-aligned; adapter/access to qualify

xAI: Capability-aligned; adapter/access to qualify