Coding
HumanEval
Python function generation with original executable tests.
About HumanEval
What it measures
Native task success, errors, latency and recorded cost.
Scope & limitations
The published pilot is a subset. Inspect the run protocol before comparing scores.
Evaluation environment
Blobfish-hosted Harbor harness on GKE; frozen task subset.