# MBPP evaluation

Run: `br-389a9805af790d8deb25689546a16de2`
Dataset version: `f46ca8374b4c:pilot-1`
Frozen manifest SHA-256: `28cbcf6c4d6a792dd22b7441b347c709e292c4998ec7e3bb9e4e2a97d1ca97eb`
Source SHA-256: `d2043b7ed0e7cf551d80686875d8f5de146a4382d1259081a6fe13f876f38253`

Selected tasks per scaffold: 10. Registered: 10. Official total: 500.
These results cover the selected tasks and the recorded execution budgets only.

Native, rubric and vision verdicts are separate. Unavailable graders are not passes; a pass rate is reported only with complete grading coverage. No retries are silently substituted.

## ChatGPT subscription

Model: `gpt-6-astra`; scaffold: `harbor-codex-subscription`; scaffold revision: `terminal-mcp-v1`.

| Site | Tasks | Completed | Native passes / graded | Rubric passes / graded | Vision passes / graded | Mean latency |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Python | 10 | 10 | 10 / 10 (100.0%) | 0 / 0 (incomplete grading) | 0 / 0 (incomplete grading) | 70.87s |
| Overall | 10 | 10 | 10 / 10 (100.0%) | 0 / 0 (incomplete grading) | 0 / 0 (incomplete grading) | 70.87s |

P95 latency: 110.65s. Model cost: not fully measured.
Task budget: 600s; max agent steps: 100.

## Claude subscription

Model: `claude-sonnet-5`; scaffold: `harbor-claude-subscription`; scaffold revision: `terminal-mcp-v1`.

| Site | Tasks | Completed | Native passes / graded | Rubric passes / graded | Vision passes / graded | Mean latency |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Python | 10 | 10 | 10 / 10 (100.0%) | 0 / 0 (incomplete grading) | 0 / 0 (incomplete grading) | 48.21s |
| Overall | 10 | 10 | 10 / 10 (100.0%) | 0 / 0 (incomplete grading) | 0 / 0 (incomplete grading) | 48.21s |

P95 latency: 84.28s. Model cost: not fully measured.
Task budget: 600s; max agent steps: 100.

## DeepSeek

Model: `deepseek-v4-pro`; scaffold: `harbor-deepseek`; scaffold revision: `terminal-mcp-v1`.

| Site | Tasks | Completed | Native passes / graded | Rubric passes / graded | Vision passes / graded | Mean latency |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Python | 10 | 10 | 10 / 10 (100.0%) | 0 / 0 (incomplete grading) | 0 / 0 (incomplete grading) | 43.19s |
| Overall | 10 | 10 | 10 / 10 (100.0%) | 0 / 0 (incomplete grading) | 0 / 0 (incomplete grading) | 43.19s |

P95 latency: 127.15s. Model cost: not fully measured.
Task budget: 600s; max agent steps: 100.