Blobfish analysis / measured evaluations

Understand the tradeoffs.

Compare task success, speed, reliability and reported cost. Choose a benchmark, inspect the distribution, and follow every number to the original task evidence.

4 published evaluations5 measured configurations85 recorded attempts

EvaluationFull report & task evidence

Matched evaluation10 selected / 164 source tasksNative tests2026-09-16

Highest task success

100.0%Claude subscription · DeepSeekAll selected tasks stay in the denominator

Lowest median latency

39.4sClaude subscriptionRecorded task time; scope in the protocol

Lowest reported cost / task

Not measuredNo complete measurement availableReported usage only; see excluded charges

Compare agents

ChatGPT subscriptionClaude subscriptionDeepSeek

PERFORMANCE

Task success

Higher is better · % of selected tasks

Claude subscription100.0%

claude-sonnet-510 pass · 0 errors

DeepSeek100.0%

deepseek-v4-pro10 pass · 0 errors

ChatGPT subscription90.0%

gpt-6-astra9 pass · 1 errors

A task must pass the declared primary grader. Completed but ungraded or uncollected tasks prevent a success ranking.

TRADEOFFS

Success vs. latency

More success, less time

ChatGPT subscriptionClaude subscriptionDeepSeek

Each point uses the same task selection, grading rules, environment and budget. This evaluation covers a subset of the source benchmark.

RELIABILITY

What happened to every task

Every selected task is accounted for

PassedFailedExecution errorUngraded / missing

ChatGPT subscription10/10 recorded

Claude subscription10/10 recorded

DeepSeek10/10 recorded

Execution errors count against task success. They are not presented as a grader verdict. Supplemental vision and provider judgments remain in the full report.

LATENCY DISTRIBUTION

Typical time and the long tail

Minimum → median → P95 → maximum

ChatGPT subscription10/10 timed

Median 67.4sP95 600.1s

Claude subscription10/10 timed

Median 39.4sP95 110.4s

DeepSeek10/10 timed

Median 44.7sP95 90.7s

P95 uses the nearest ranked observed task. Small selections can have a P95 equal to their slowest attempt. Missing timing records are disclosed.

ALL COMPARISON METRICS

Inspect the numbers.

HumanEval · agent evaluation · 10 selected tasks · 2026-09-16
Rank	Agent / model						Errors	Evidence
1	Claude subscriptionclaude-sonnet-5	100.0%10/10 passed	100.0%Execution, irrespective of grade	39.4s10/10 measured	110.4s10/10 measured	Not measured0/10 reported	00 missing · 0 ungraded	Task results ↗
1	DeepSeekdeepseek-v4-pro	100.0%10/10 passed	100.0%Execution, irrespective of grade	44.7s10/10 measured	90.7s10/10 measured	Not measured0/10 reported	00 missing · 0 ungraded	Task results ↗
3	ChatGPT subscriptiongpt-6-astra	90.0%9/10 passed	90.0%Execution, irrespective of grade	67.4s10/10 measured	600.1s10/10 measured	Not measured0/10 reported	10 missing · 0 ungraded	Task results ↗

Ranks compare task success within this frozen evaluation; ties share rank. Filtering and metric sorting do not change the success rank. Missing measurements are shown explicitly and sort last.

Measurement definitions, configuration & protocol

Measured ten-task sample from 164 public evaluation tasks. One attempt per task with a common terminal scaffold; original tests, RNG seed 0. This is an agentic evaluation, not the original code-only sampling protocol. Source: https://github.com/openai/human-eval/tree/6d43fb980f9fee3c892a914eda09951f772ad10d. Registered source archive SHA-256: f373485244ee5745ff98af9f9e20ff824f5b4b68bb1f37a61dd66e20e2cb4581. Selected tasks: humaneval-2, humaneval-39, humaneval-48, humaneval-50, humaneval-61, humaneval-64, humaneval-75, humaneval-79, humaneval-100, humaneval-136. ChatGPT subscription: harbor-codex-subscription; observed chatgpt_subscription / 0.154.0 / max. Claude subscription: harbor-claude-subscription; observed claude_subscription / 2.1.273 / max. DeepSeek: harbor-deepseek; observed deepseek_api / max. 3 concurrent task attempts; 600 seconds and 100 terminal tool calls per task. Account queues are outside task clocks. Latency includes environment setup, agent execution and native verification. Subscription cost is unmeasured. Full traces and pinned task/world images are retained in raw_outputs.jsonl.

Harness: Harbor · GKE
Primary evaluator: Native tests
Source coverage: 10 / 164 tasks

ChatGPT subscription: 600s/task; 100 max turns; no automatic retries. GKE gVisor; images pinned in manifest.

Claude subscription: 600s/task; 100 max turns; no automatic retries. GKE gVisor; images pinned in manifest.

DeepSeek: 600s/task; 100 max turns; no automatic retries. GKE gVisor; images pinned in manifest.

Cost reflects the charges present in the source report; hosting, subscription fees and independent grading may be excluded. Task latency uses the source report’s measurement window. These are not streaming inference measurements.

Inspect the source report and individual attempts ↗