Retrieval, embeddings and long context / Benchmark leaderboard
MTEB / MMTEB
Embedding quality across retrieval, similarity, classification and multilingual tasks.
No published company result yet.
This benchmark is in the research directory. Its task package, adapter and grading protocol need qualification before a hosted run can be offered.
Arrange an evaluation for your company →What this benchmark measures
Metrics
Task-specific metrics; report retrieval nDCG separately from broad averages.
Execution requirements
Embedding/reranking endpoint through the selected MTEB task collection.
Scope and limitations
Vector databases should benchmark a declared embedding-plus-index pipeline; MTEB is not itself a database performance test.
Evaluation availability
Catalog entry. Request a managed evaluation to qualify your agent interface and the benchmark’s native grading requirements.
Sources and company fit
- Original benchmark ↗Established
- Hugging Face · mteb/arguana ↗Public · One MTEB/BEIR retrieval task; do not call this the complete MTEB suite.
Companies whose products may fit
Research recommendations based on product capabilities. These companies have not necessarily run this benchmark or integrated with Blobfish.
Compatibility notes for each company
Alibaba Qwen: Capability-aligned; adapter/access to qualify
Cohere: Direct component API + selected system adapters
Jina AI: Direct component API + selected system adapters
Mistral: Capability-aligned; adapter/access to qualify
NVIDIA: Research / simulator adaptation
Voyage AI: Direct component API + selected system adapters