DevOps and observability / Benchmark leaderboard
ITSMBench
53 multi-turn IT service tasks with policies and stateful business records.
No published company result yet.
This benchmark is in the research directory. Its task package, adapter and grading protocol need qualification before a hosted run can be offered.
Arrange an evaluation for your company →What this benchmark measures
Metrics
Binary final database-state correctness.
Execution requirements
Benchmark tenant, tool adapter and simulated colleague.
Scope and limitations
Requires benchmark tool semantics and identity/policy fidelity; useful niche fit rather than established broad fame.
Evaluation availability
Catalog entry. Request a managed evaluation to qualify your agent interface and the benchmark’s native grading requirements.
Sources and company fit
- Original benchmark ↗Newer specialist
- Harbor · vibrantlabsai/itsm-bench ↗53 source tasks · Public · Public detail metadata + family research
Companies whose products may fit
Research recommendations based on product capabilities. These companies have not necessarily run this benchmark or integrated with Blobfish.
Compatibility notes for each company
Automation Anywhere: Product-specific adapter / qualified access
Glean: Capability-aligned; adapter/access to qualify
Salesforce: Product-specific adapter / qualified access
ServiceNow: Product-specific adapter / qualified access
UiPath: Product-specific adapter / qualified access