Coding and terminal agents
BigCodeBench / BigCodeBench-Hard
Python programming tasks using real libraries and APIs.
About BigCodeBench / BigCodeBench-Hard
What it measures
Pass@1 using executable tests.
Scope & limitations
Full, Hard, Complete and Instruct tracks differ; Harbor Hard-Complete lists 145 tasks, so inspect membership before comparison.