Completed attempts clear the tests.
29 of 30 selected attempts passed the declared primary evaluation. 1 ended with an execution error or timeout. Every selected attempt stays in the report.
humaneval / Benchmark deep dive
Compare the outcomes, understand the gaps, and inspect the work behind every score.
29 of 30 selected attempts passed the declared primary evaluation. 1 ended with an execution error or timeout. Every selected attempt stays in the report.
10 selected tasks per model cover a small part of 164 benchmark tasks. One attempt per task measures this run; repeated-run consistency and full-suite leadership are not established.
Dollar cost is recorded for 0 of 30 attempts. Subscription usage and missing provider billing are not converted into an invented per-task price. Latency includes the scope declared in the protocol below.
Scores use the same selected tasks and the report’s declared primary evaluator.
| Model / agent | Success | Execution errors | Mean latency | Cost / task |
|---|---|---|---|---|
| ChatGPT subscriptiongpt-6-astra | 90.0%9/10 selected tasks | 1 | 135.8s | Not reported |
| Claude subscriptionclaude-sonnet-5 | 100.0%10/10 selected tasks | 0 | 48.1s | Not reported |
| DeepSeekdeepseek-v4-pro | 100.0%10/10 selected tasks | 0 | 49.9s | Not reported |
Pilot coverage: 10/164 source tasks. Small fixed selections do not establish a full-benchmark ranking. Missing grades remain visible.
Execution failure and an incorrect result are separate outcomes.
TimeoutError:
No failing task was recorded in this selected sample.
No failing task was recorded in this selected sample.
The work behind the score
Original instructions, observed actions, and the verdict for each attempt.
gpt-6-astra · 73.0s · 3 recorded actions
Complete the following Python snippet. Write only the completion (the code after this snippet) to /logs/artifacts/answer.py. Do not repeat the provided code. def truncate_number(number: float) -> float: """ Given a positive floating point number, it can be decomposed into and integer part (largest integer smaller than given number) and decimals (leftover part always smaller than 1). Return the decimal part of the number. >>> truncate_number(3.5) 0.5 """
Recorded benchmark evaluator · 6d43fb980f9fee3c892a914eda09951f772ad10d:original-tests-seed0
reward=1; pass threshold=1.0
Recorded benchmark evaluator · none
Rubric grading is not configured for this task.
Recorded benchmark evaluator · none
Vision grading is not configured for this task.
Reproducibility
Measured ten-task sample from 164 public evaluation tasks. One attempt per task with a common terminal scaffold; original tests, RNG seed 0. This is an agentic evaluation, not the original code-only sampling protocol. Source: https://github.com/openai/human-eval/tree/6d43fb980f9fee3c892a914eda09951f772ad10d. Registered source archive SHA-256: f373485244ee5745ff98af9f9e20ff824f5b4b68bb1f37a61dd66e20e2cb4581. Selected tasks: humaneval-2, humaneval-39, humaneval-48, humaneval-50, humaneval-61, humaneval-64, humaneval-75, humaneval-79, humaneval-100, humaneval-136. ChatGPT subscription: harbor-codex-subscription; observed chatgpt_subscription / 0.154.0 / max. Claude subscription: harbor-claude-subscription; observed claude_subscription / 2.1.273 / max. DeepSeek: harbor-deepseek; observed deepseek_api / max. 3 concurrent task attempts; 600 seconds and 100 terminal tool calls per task. Account queues are outside task clocks. Latency includes environment setup, agent execution and native verification. Subscription cost is unmeasured. Full traces and pinned task/world images are retained in raw_outputs.jsonl.