Search APIs asked for dated documents by the words a person would type (arXiv papers submitted days, a month and a year before the run): is the document in the top results, and how fresh is the index?
MRR and hit@1 by exact identifier match in the result URL, hit@5, evidence@5 (identifier or exact title in a top-5 title/snippet), per age bucket; latency p50/p95. No judge.
Scope & limitations
One document class (arXiv papers) on the live web; index freshness depends on the run date, so compare rows from the same sitting.
Evaluation environment
Items drawn from arXiv's API on the run date; one request per item at 1.5 s pacing, only HTTP 429 retried with backoff (same for every provider); every provider sees the same items.
