The harness behind the results.

Full 100/100-task version-pinned model run derived from the exact Harbor trial artifacts; no oracle or deterministic-control trials are ranked. Scoring observes only released task state and answers, Codex built-in web search was disabled, and OS-level no-egress isolation is not claimed. Provider-native trace provenance is reported per task (100/100 native, 0/100 explicit Harbor agent fallbacks); trials with no recorded MCP operation retain that fallback rather than being presented as provider calls.

Full 137/137-task version-pinned run of terminus-2 on DeepSeek V4.1 Flash, derived from the exact Harbor trial artifacts; oracle, control and calibration episodes are never ranked. Each task's own verifier grades end state, answer fields and evidence reads; the one trial the verifier could not grade (a Harbor RuntimeError) counts as 0. The run executed on one local Docker host in two sittings: the job was interrupted at 124 finished trials and resumed with Harbor's job resume, which kept every finished trial and started only the unfinished tasks again, so every task has exactly one attempt. Known v0.3.0 defects, fixed in the engine after this release (#4866) and shipping in v0.3.1: 20 decision-only tasks word the sprint_id field ambiguously, and ids read off key-less list pages can be graded 'never rendered'. 11 of this run's 35 non-passing trials fail on those two leaves alone, so the pass rate understates this agent on the affected tasks.

Every system receives the identical compact-JSON state and typed questions; probabilities are scored, never generated text. Structurally unsupported items (over a system's input limit or option cap) are recorded as unanswered. Observed 2026-09-20.

Pages are generated from a WebBench world and served through a throw-away public tunnel; the generator knows every content unit it emitted. Pass = usable page: content recall ≥ 0.9 and signal ratio ≥ 0.3 (TinyFish's own usability floor). Score = content F1, the harmonic mean of content recall and signal ratio, so neither a clean stub nor a full-page dump scores well. A provider error counts as 0. No LLM judge. Profiles: clean, heavy chrome, consent overlay, client-side rendered, late content.

Pages are generated from a WebBench world and served through a throw-away public tunnel; the generator knows every content unit it emitted. Pass = usable page: content recall ≥ 0.9 and signal ratio ≥ 0.3 (TinyFish's own usability floor). Score = content F1, the harmonic mean of content recall and signal ratio, so neither a clean stub nor a full-page dump scores well. A provider error counts as 0. No LLM judge. Profiles: clean, heavy chrome, consent overlay, client-side rendered, late content.

Pages are generated from a WebBench world and served through a throw-away public tunnel; the generator knows every content unit it emitted. Pass = usable page: content recall ≥ 0.9 and signal ratio ≥ 0.3 (TinyFish's own usability floor). Score = content F1, the harmonic mean of content recall and signal ratio, so neither a clean stub nor a full-page dump scores well. A provider error counts as 0. No LLM judge. Profiles: clean, heavy chrome, consent overlay, client-side rendered, late content.

Live-web suite by necessity (a search index covers the public web). Items are drawn from arXiv's API on the run date, so the set is new each run. Pass = the paper's arXiv identifier appears in the rank-1 result URL; score = reciprocal rank within the top 10. No LLM judge. Latency correction (2026-09-20): the figures first published here were warm-cache reads. The same 60 items had been issued to TinyFish in two earlier sittings (p50 904 ms, then 956 ms, then 139 ms) and to Browserbase in one (774 ms, then 341 ms), and a never-seen query repeated three times went 1,654 ms, 237 ms, 114 ms with identical results. The latency shown is now each item's FIRST issue to that provider, pooled across the sittings: TinyFish 927 ms over 58 items, Browserbase 774 ms over 60. Ranking quality (MRR, hit@k) is unaffected. The suite now measures latency on first issue only.

Live-web suite by necessity (a search index covers the public web). Items are drawn from arXiv's API on the run date, so the set is new each run. Pass = the paper's arXiv identifier appears in the rank-1 result URL; score = reciprocal rank within the top 10. No LLM judge. Latency correction (2026-09-20): the figures first published here were warm-cache reads. The same 60 items had been issued to TinyFish in two earlier sittings (p50 904 ms, then 956 ms, then 139 ms) and to Browserbase in one (774 ms, then 341 ms), and a never-seen query repeated three times went 1,654 ms, 237 ms, 114 ms with identical results. The latency shown is now each item's FIRST issue to that provider, pooled across the sittings: TinyFish 927 ms over 58 items, Browserbase 774 ms over 60. Ranking quality (MRR, hit@k) is unaffected. The suite now measures latency on first issue only.

Each challenge is one page with one mechanic and a one-line human request; the page never shows a success token — the server compares the state the browser submits with the target. Pass = the final submission is correct; score = first-time pass (no rejected submission before it). 26 rejected submissions across the run. A run the provider cancelled on its own before the 300 s deadline is an execution error (scores 0, shown as an error, not a failed task). Cost computed from TinyFish's metered rates ($0.016/step, $0.002/browser-minute) and the recorded steps and wall time; provider-cancelled runs report no step count, so it is a lower bound. Mechanic-verified: 29 of 52 passes carried the challenge's gesture as a trusted browser event (29 counting only the final submission, which penalises an agent that re-opens a submit URL to check itself). A pass that the page state accepts but no gesture produced was reached by page script rather than by operating the interface.

Each challenge is one page with one mechanic and a one-line human request; the server compares the submitted state with the target. Pass = the final submission is correct; score = first-time pass. Mechanic-verified: 49 of 52 passes carried the challenge's gesture as a trusted browser event (44 counting only the final submission, which penalises an agent that re-opens a submit URL to check itself). A pass that the page state accepts but no gesture produced was reached by page script rather than by operating the interface.

Imported recorded evidence; not independently rerun by the report engine. Scaffold: browser-use-cloud-v4 at SDK 3.11.3. Unknown measurements remain unknown. 15/643 tasks; one per website, seed 20260915. Separate Codex AI evidence review, not official GPT-4V grading or human adjudication; review model version was not retained. Unverified requirements receive no success credit under the recorded rubric. A later, separate Astra vision review runs on GKE using only the final screenshot and answer. Its restricted evidence scope is supplemental and does not replace the original rubric primary. Vision latency and subscription token usage are recorded separately; no dollar cost is inferred. Provider trajectory judgments are retained separately. Latency measures create-to-terminal agent time. Cost is final provider-reported run cost including asynchronous judge updates; browser hosting is separate. Original fixed dates were preserved.