Blobfish ResearchLedgerBench-100 v1.0.0Public

Can an agent run the finance desk—not just look up the number?

LedgerBench-100 measures working finance operations: triage the anomaly, reconcile the bank feed, verify the filing against real SEC XBRL data, commit the payment run, and file the graded answer — across a D365-shaped ERP, Odoo, subsidiary books, drive, email, and documents behind 8 MCP servers.

View leaderboard Browse all 100 tasks

Run on Harbor ↗Dataset on Hugging Face ↗Source ↗

100finance tasks

8MCP servers

66tools under contract

46,686real SEC XBRL facts

0LLM grading calls

Read this first

25 of the 100 prompts are deliberate escalated variants that reuse a base persona message verbatim against a harder world (the governing policy is buried among decoy documents), so the release carries 75 distinct prompt texts. The duplication is disclosed, by design, and graded by different worlds.

See the full disclosure →

Measured performance

Leaderboard

REF

Reference oracleDeterministic reference solver (release qualification)

100 tasks

100.0%

15 calls avg.Solvability ceiling from the qualification run; not a model submission.

deepseek-v4-pronative tool loop (sim/run_batch.py) · pre-release world

45 tasks

78.5%

Pre-release run mapped onto the release by source-task id: only the released tasks that were run are counted (3 binary trials each; score is the per-task mean, strict pass requires all three trials). The escalated buried-policy released variants were not exercised.

100 distinct tasks

Task browser

Search tasksFilter by family100 shown

lgr100-001-duplicate-payment-marAnomaly triage

Duplicate Payment Mar

**Robin Vale · Treasury · Teams 08:05, Friday 6 March** Office-supplies run today — vendor group OFFICE, operating account. Cash is fine this week so it should be a quick one. Before you commit it, run the duplicate screen per SOP-AP-07. We got burne

Seeded files: 1
Reference path: 11 MCP calls

Complete task prompt

17 graded criteria summarized

**Robin Vale · Treasury · Teams 08:05, Friday 6 March**

Office-supplies run today — vendor group OFFICE, operating account. Cash is fine this week
so it should be a quick one.

Before you commit it, run the duplicate screen per SOP-AP-07. We got burned on this in Q4
and the auditors flagged it, so it's on every run now whether or not cash is tight. Be
careful though — last time someone binned a perfectly good invoice because the amount looked
familiar and the vendor was not happy.

Rubric summary

answer_checks: run_id (contains_all) expects ["PR-00001"]
answer_checks: duplicate_found (yes_no) expects yes
answer_checks: duplicate_invoice (contains_all) expects ["5521-OPS"]
answer_checks: duplicate_of (contains_all) expects ["OSINV-5521"]
answer_checks: total_paid (number) expects 34450
answer_checks: amount_prevented (number) expects 18400
state_checks: run_committed (sql) expects committed
state_checks: paid_set (sql) expects OSINV-5530,OSINV-5555,TPS-11907
state_checks: exactly_one_rejection (sql) expects 1
state_checks: duplicate_rejected_with_code (sql) expects duplicate
state_checks: lookalike_was_paid (sql) expects paid
state_checks: paid_total_rederived (sql) expects 34450
state_checks: original_untouched (sql) expects 1
state_checks: (writes_only)
trace_checks: (required_servers)
trace_checks: (min_calls)
trace_checks: (reads_before_submit)

Seeded evidence

Asset room

Search assets

duplicate-payment-control.mdtask_files/lgr100-001-duplicate-payment-mar/documentsContextMD1.6 KB

# SOP-AP-07 — Duplicate disbursement control > Contoso Entertainment System USA · Accounts Payable · effective 2026-01-01 · v1.2 > SIMULATION ONLY Duplicate payments are the most common avoidable loss in disbur

Runnable world

Environment and tool contract

get_company_info

books

CompanyInfo (GET /v3/company/{realmId}/companyinfo shape).

{
  "type": "object",
  "properties": {},
  "required": []
}

query

books

QBO query language (GET /v3/company/{realmId}/query shape). E.g. "SELECT * FROM Invoice WHERE CustomerRef = 'BC-114'". Entities: Customer, Invoice, CreditMemo, Payment. Operators: =, LIKE.

{
  "type": "object",
  "properties": {
    "q": {
      "type": "string"
    }
  },
  "required": [
    "q"
  ]
}

get_customer

books

Read a Customer by Id (GET /v3/.../customer/{id} shape; Balance = open invoice total).

{
  "type": "object",
  "properties": {
    "customer_id": {
      "type": "string"
    }
  },
  "required": [
    "customer_id"
  ]
}

get_invoice

books

Read an Invoice by Id or DocNumber (GET /v3/.../invoice/{id} shape).

{
  "type": "object",
  "properties": {
    "invoice": {
      "type": "string"
    }
  },
  "required": [
    "invoice"
  ]
}

get_creditmemo

books

Read a CreditMemo by Id or DocNumber (GET /v3/.../creditmemo/{id} shape).

{
  "type": "object",
  "properties": {
    "creditmemo": {
      "type": "string"
    }
  },
  "required": [
    "creditmemo"
  ]
}

get_payment

books

Read a Payment by Id (GET /v3/.../payment/{id} shape).

{
  "type": "object",
  "properties": {
    "payment_id": {
      "type": "string"
    }
  },
  "required": [
    "payment_id"
  ]
}

report_aged_receivables

books

AgedReceivables report (GET /v3/.../reports/AgedReceivables shape): open invoices bucketed by days past due. Credit memos are NOT netted (QBO behavior).

{
  "type": "object",
  "properties": {
    "as_of": {
      "type": "string"
    }
  },
  "required": []
}

report_customer_balance

books

CustomerBalance report shape: net open balance per customer.

{
  "type": "object",
  "properties": {},
  "required": []
}

report_transaction_list

books

TransactionList report shape: invoices, credit memos, payments in a date range.

{
  "type": "object",
  "properties": {
    "date_from": {
      "type": "string"
    },
    "date_to": {
      "type": "string"
    }
  },
  "required": []
}

create_invoice

books

POST /v3/.../invoice. This connection is accounting.read-scoped.

{
  "type": "object",
  "properties": {
    "invoice": {
      "type": "object"
    }
  },
  "required": [
    "invoice"
  ]
}

update_invoice

books

Sparse update (POST /v3/.../invoice). This connection is accounting.read-scoped.

{
  "type": "object",
  "properties": {
    "invoice": {
      "type": "object"
    }
  },
  "required": [
    "invoice"
  ]
}

void_invoice

books

Void (POST /v3/.../invoice?operation=void). This connection is accounting.read-scoped.

{
  "type": "object",
  "properties": {
    "invoice_id": {
      "type": "string"
    }
  },
  "required": [
    "invoice_id"
  ]
}

list_documents

docs

List documents, optionally by type (policy, sop, template, statement).

{
  "type": "object",
  "properties": {
    "doc_type": {
      "type": "string"
    }
  },
  "required": []
}

search_documents

docs

Keyword search over titles and bodies.

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string"
    }
  },
  "required": [
    "query"
  ]
}

get_document

docs

Fetch a full document body.

{
  "type": "object",
  "properties": {
    "doc_id": {
      "type": "string"
    }
  },
  "required": [
    "doc_id"
  ]
}

get_document_metadata

docs

Document metadata only (type, version, effective date) — check currency before relying on a policy.

{
  "type": "object",
  "properties": {
    "doc_id": {
      "type": "string"
    }
  },
  "required": [
    "doc_id"
  ]
}

list_document_types

docs

List document types with counts (policy, sop, template, statement, ...).

{
  "type": "object",
  "properties": {},
  "required": []
}

messages_list

Search messages (users.messages.list). `q` matches subject/body/sender; optional label (default INBOX).

{
  "type": "object",
  "properties": {
    "q": {
      "type": "string"
    },
    "label": {
      "type": "string"
    }
  },
  "required": [
    "q"
  ]
}

messages_get

Fetch a full message (users.messages.get, format=full): headers, snippet, body, attachment parts.

{
  "type": "object",
  "properties": {
    "id": {
      "type": "string"
    }
  },
  "required": [
    "id"
  ]
}

threads_get

Fetch a conversation thread (users.threads.get): all messages sharing the normalized subject.

{
  "type": "object",
  "properties": {
    "id": {
      "type": "string",
      "description": "a threadId from messages_list, or a message id"
    }
  },
  "required": [
    "id"
  ]
}

labels_list

List labels/folders with message counts (users.labels.list).

{
  "type": "object",
  "properties": {},
  "required": []
}

attachments_get

Fetch an attachment's content by message id (users.messages.attachments.get; text extraction).

{
  "type": "object",
  "properties": {
    "message_id": {
      "type": "string"
    }
  },
  "required": [
    "message_id"
  ]
}

send_message

Send a message from the shared finance mailbox (users.messages.send shape). Counterparties reply on their own schedule; a reply, if any, lands in the inbox and is returned here.

{
  "type": "object",
  "properties": {
    "to": {
      "type": "string"
    },
    "subject": {
      "type": "string"
    },
    "body": {
      "type": "string"
    }
  },
  "required": [
    "to",
    "subject",
    "body"
  ]
}

data_find_entity_type

erp

Find OData entity types matching a natural-language query. Returns multiple top hits; you decide which matches.

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "description": "e.g. 'customer invoices', 'payment terms'"
    }
  },
  "required": [
    "query"
  ]
}

data_get_entity_metadata

erp

Get metadata (field list) for an entity. Needed before find/create/update/delete entity calls.

{
  "type": "object",
  "properties": {
    "entity": {
      "type": "string"
    }
  },
  "required": [
    "entity"
  ]
}

data_find_entities

erp

Find/read data records for one entity with equality/contains filters. Paged (25 rows).

{
  "type": "object",
  "properties": {
    "entity": {
      "type": "string"
    },
    "filters": {
      "type": "object",
      "description": "field -> value; strings match case-insensitive substring, numbers match exactly"
    },
    "page": {
      "type": "integer"
    }
  },
  "required": [
    "entity"
  ]
}

data_find_entities_sql

erp

Find/read records using SQL (read-only single SELECT over erp_* tables; LIMIT 200 enforced). Replaces OData find in 10.0.48+.

{
  "type": "object",
  "properties": {
    "sql": {
      "type": "string"
    }
  },
  "required": [
    "sql"
  ]
}

data_create_entities

erp

Create data records using OData (no deep inserts). Subject to role security.

{
  "type": "object",
  "properties": {
    "entity": {
      "type": "string"
    },
    "records": {
      "type": "array"
    }
  },
  "required": [
    "entity",
    "records"
  ]
}

data_update_entities

erp

Update data records using OData. Subject to role security.

{
  "type": "object",
  "properties": {
    "entity": {
      "type": "string"
    },
    "updates": {
      "type": "array"
    }
  },
  "required": [
    "entity",
    "updates"
  ]
}

data_delete_entities

erp

Delete data records using OData. Subject to role security.

{
  "type": "object",
  "properties": {
    "entity": {
      "type": "string"
    },
    "keys": {
      "type": "array"
    }
  },
  "required": [
    "entity",
    "keys"
  ]
}

form_find_menu_item

erp

Find a menu item (application page) by search term.

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string"
    }
  },
  "required": [
    "query"
  ]
}

form_open_menu_item

erp

Open a menu item (form). Returns the form view model (grid page 1; tabs closed by default).

{
  "type": "object",
  "properties": {
    "menu_item": {
      "type": "string",
      "description": "menu item name or form name"
    }
  },
  "required": [
    "menu_item"
  ]
}

form_close_form

erp

Close an open form.

{
  "type": "object",
  "properties": {
    "form_id": {
      "type": "string"
    }
  },
  "required": [
    "form_id"
  ]
}

form_find_controls

erp

Find controls on an open form. One search term per call.

{
  "type": "object",
  "properties": {
    "form_id": {
      "type": "string"
    },
    "search": {
      "type": "string"
    }
  },
  "required": [
    "form_id",
    "search"
  ]
}

form_open_or_close_tab

erp

Open or close a tab on the form. Opening reveals the tab's fields for the selected row.

{
  "type": "object",
  "properties": {
    "form_id": {
      "type": "string"
    },
    "tab": {
      "type": "string"
    },
    "open": {
      "type": "boolean"
    }
  },
  "required": [
    "form_id",
    "tab"
  ]
}

form_filter_form

erp

Apply a quick filter across the form's grid columns.

{
  "type": "object",
  "properties": {
    "form_id": {
      "type": "string"
    },
    "value": {
      "type": "string"
    }
  },
  "required": [
    "form_id",
    "value"
  ]
}

form_filter_grid

erp

Filter the grid on one column. Only the 'matches' (substring) operator is supported.

{
  "type": "object",
  "properties": {
    "form_id": {
      "type": "string"
    },
    "column": {
      "type": "string"
    },
    "value": {
      "type": "string"
    },
    "operator": {
      "type": "string",
      "description": "only 'matches' is supported"
    }
  },
  "required": [
    "form_id",
    "column",
    "value"
  ]
}

form_sort_grid_column

erp

Sort the grid by a column.

{
  "type": "object",
  "properties": {
    "form_id": {
      "type": "string"
    },
    "column": {
      "type": "string"
    },
    "direction": {
      "type": "string",
      "description": "asc|desc"
    }
  },
  "required": [
    "form_id",
    "column"
  ]
}

form_select_grid_row

erp

Select a grid row by its _row id. Returns all fields incl. runtime-calculated values.

{
  "type": "object",
  "properties": {
    "form_id": {
      "type": "string"
    },
    "row": {
      "type": "integer"
    }
  },
  "required": [
    "form_id",
    "row"
  ]
}

form_click_control

erp

Click a control/action on the form (e.g. Collections, OpenTransactions, AgedBalances).

{
  "type": "object",
  "properties": {
    "form_id": {
      "type": "string"
    },
    "control": {
      "type": "string"
    }
  },
  "required": [
    "form_id",
    "control"
  ]
}

form_open_lookup

erp

Open a lookup control (e.g. payment_term, cash_disc_code) and list its valid values.

{
  "type": "object",
  "properties": {
    "form_id": {
      "type": "string"
    },
    "control": {
      "type": "string"
    }
  },
  "required": [
    "form_id",
    "control"
  ]
}

form_set_control_values

erp

Set values on form controls (not lookup controls). Subject to role security.

{
  "type": "object",
  "properties": {
    "form_id": {
      "type": "string"
    },
    "values": {
      "type": "object"
    }
  },
  "required": [
    "form_id",
    "values"
  ]
}

form_save_form

erp

Save the form. Subject to role security.

{
  "type": "object",
  "properties": {
    "form_id": {
      "type": "string"
    }
  },
  "required": [
    "form_id"
  ]
}

api_find_actions

erp

Finds actions (ICustomAPI AI tools) you can invoke.

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string"
    }
  },
  "required": []
}

api_invoke_action

erp

Invokes an action by name with parameters.

{
  "type": "object",
  "properties": {
    "action": {
      "type": "string"
    },
    "parameters": {
      "type": "object"
    }
  },
  "required": [
    "action"
  ]
}

lookup_company

filings

Resolve a company name or ticker to CIK (company_tickers.json shape).

{
  "type": "object",
  "properties": {
    "query": {
      "type": "string"
    }
  },
  "required": [
    "query"
  ]
}

list_available_concepts

filings

Snapshot index helper: XBRL concepts available for a company (no EDGAR equivalent).

{
  "type": "object",
  "properties": {
    "ticker": {
      "type": "string"
    }
  },
  "required": [
    "ticker"
  ]
}

get_company_concept

filings

All facts for one us-gaap concept (api/xbrl/companyconcept shape).

{
  "type": "object",
  "properties": {
    "ticker": {
      "type": "string"
    },
    "concept": {
      "type": "string",
      "description": "e.g. AssetsCurrent"
    }
  },
  "required": [
    "ticker",
    "concept"
  ]
}

get_company_facts

filings

ALL facts for a company across concepts (api/xbrl/companyfacts shape; large payload like the real endpoint).

{
  "type": "object",
  "properties": {
    "ticker": {
      "type": "string"
    }
  },
  "required": [
    "ticker"
  ]
}

get_xbrl_frames

filings

One concept, one annual period, across all companies in the snapshot (api/xbrl/frames shape).

{
  "type": "object",
  "properties": {
    "concept": {
      "type": "string"
    },
    "unit": {
      "type": "string"
    },
    "fy": {
      "type": "string",
      "description": "e.g. 2024"
    }
  },
  "required": [
    "concept",
    "fy"
  ]
}

get_submissions

filings

Company filing history (data.sec.gov/submissions shape: filings.recent columnar arrays).

{
  "type": "object",
  "properties": {
    "ticker": {
      "type": "string"
    }
  },
  "required": [
    "ticker"
  ]
}

full_text_search

filings

Full-text search across filing documents (efts.sec.gov search shape).

{
  "type": "object",
  "properties": {
    "q": {
      "type": "string"
    }
  },
  "required": [
    "q"
  ]
}

reporting_fields

harness

The fields this piece of work must be filed under, with their expected types. The request itself will not list them — read them here, the way you would read any reporting system's schema before filing into it.

{
  "type": "object",
  "properties": {},
  "required": []
}

submit_answer

harness

File your findings as an object, e.g. {"outstanding_balance": 12345.67, "currency": "USD"}. Call reporting_fields to see which fields are required. Resubmitting a field overwrites it. Use the literal string "none" where the answer is that the thing does not exist.

{
  "type": "object",
  "properties": {
    "answers": {
      "type": "object",
      "description": "field -> value map"
    }
  },
  "required": [
    "answers"
  ]
}

list_submitted

harness

See what you have submitted so far.

{
  "type": "object",
  "properties": {},
  "required": []
}

fields_get

odoo

List the fields of a model, with type. Discovery step before search_read.

{
  "type": "object",
  "properties": {
    "model": {
      "type": "string"
    }
  },
  "required": [
    "model"
  ]
}

search_read

odoo

Search and read records. `domain` is a list of [field, operator, value] leaves combined with AND, e.g. [["product_code","=","P123"],["price","<",100]].

{
  "type": "object",
  "properties": {
    "model": {
      "type": "string"
    },
    "domain": {
      "type": "array",
      "description": "list of [field, operator, value]"
    },
    "fields": {
      "type": "array",
      "description": "field names; omit for all"
    },
    "page": {
      "type": "integer"
    }
  },
  "required": [
    "model"
  ]
}

create

odoo

Create a record. Returns its name/id. Orders are created in state 'draft' — confirm them with action_confirm.

{
  "type": "object",
  "properties": {
    "model": {
      "type": "string"
    },
    "values": {
      "type": "object"
    }
  },
  "required": [
    "model",
    "values"
  ]
}

write

odoo

Update records matching a domain.

{
  "type": "object",
  "properties": {
    "model": {
      "type": "string"
    },
    "domain": {
      "type": "array"
    },
    "values": {
      "type": "object"
    }
  },
  "required": [
    "model",
    "domain",
    "values"
  ]
}

action_confirm

odoo

Confirm draft orders by name — sale.order -> 'sale', purchase.order -> 'purchase', mrp.production -> 'confirmed'. A draft order is not a commitment and is not graded as one.

{
  "type": "object",
  "properties": {
    "model": {
      "type": "string"
    },
    "names": {
      "type": "array"
    }
  },
  "required": [
    "model",
    "names"
  ]
}

list_drive_items

sheets

List workbook files on the finance shared drive (GET /drive/root/children shape).

{
  "type": "object",
  "properties": {},
  "required": []
}

get_drive_item

sheets

One file's metadata (GET /drive/items/{id} shape) — check lastModifiedDateTime before trusting a tracker.

{
  "type": "object",
  "properties": {
    "item": {
      "type": "string",
      "description": "file name/id"
    }
  },
  "required": [
    "item"
  ]
}

workbook_worksheets

sheets

List worksheets of a workbook (GET /workbook/worksheets shape).

{
  "type": "object",
  "properties": {
    "item": {
      "type": "string"
    }
  },
  "required": [
    "item"
  ]
}

workbook_range

sheets

Read a range (GET /workbook/worksheets/Sheet1/range(address='A1:C4') shape). Returns the Graph workbookRange object.

{
  "type": "object",
  "properties": {
    "item": {
      "type": "string"
    },
    "address": {
      "type": "string",
      "description": "A1 notation, e.g. A1:F4"
    }
  },
  "required": [
    "item",
    "address"
  ]
}

workbook_used_range

sheets

Read the used range of Sheet1 (GET /workbook/worksheets/Sheet1/usedRange shape).

{
  "type": "object",
  "properties": {
    "item": {
      "type": "string"
    }
  },
  "required": [
    "item"
  ]
}

drive_search

sheets

Search file contents across the drive (GET /drive/root/search(q='...') shape, cell-level hits).

{
  "type": "object",
  "properties": {
    "q": {
      "type": "string"
    }
  },
  "required": [
    "q"
  ]
}

Replay the work

Trajectories

AgentReference oracleDeterministic reference solver (release qualification)

Score—Reference trace

Tool calls11lgr100-001-duplicate-payment-mar

Run cost—

11 events

1
Discoverdocs.search_documentsok
```
{
  "query": "duplicate"
}
```

Investigatedocs.get_documentok

{
  "doc_id": "duplicate-payment-control"
}

Delivererp.api_invoke_actionok

{
  "action": "ContosoPaymentRunPropose",
  "parameters": {
    "bank_account": "USMF-OPER",
    "pay_date": "2026-03-06",
    "vendor_group": "OFFICE"
  }
}

Investigateerp.data_find_entities_sqlok

{
  "sql": "SELECT invoice, po_number, amount, settled, closed, trans_date FROM erp_vend_trans WHERE account IN ('PVEN-21','PVEN-22') AND txn_type='Invoice' ORDER BY po_number, trans_date"
}

Investigateerp.data_find_entities_sqlok

{
  "sql_bytes": 314,
  "sql_sha256": "e3ebb4395ba5f28212582d7d83a4e0d78c82f5306b241bef62747955b6d9806f"
}

6
Discoveremail.messages_listok
```
{
  "q": "statement"
}
```
7
Investigateemail.messages_getok
```
{
  "id": "em-0701"
}
```
8
Investigateemail.attachments_getok
```
{
  "message_id": "em-0701"
}
```

Delivererp.api_invoke_actionok

{
  "action": "ContosoPaymentRunCommit",
  "parameters_bytes": 371,
  "parameters_sha256": "6f1f8b7a6f2a16dd1d9723c2578e22df5513abd8beeb139f879936fd369ef224"
}

Investigateerp.data_find_entitiesok

{
  "entity": "PaymentRunLines",
  "filters": {
    "run_id": "PR-00001"
  }
}

Deliverharness.submit_answerok

{
  "answers": {
    "amount_prevented": 18400,
    "duplicate_found": "yes",
    "duplicate_invoice": "5521-OPS",
    "duplicate_of": "OSINV-5521",
    "run_id": "PR-00001",
    "total_paid": 34450
  }
}

Methodology

How the numbers are made

A shared finance world behind eight MCP servers

Every task runs against one simulated company served through 8 MCP servers exposing 66 tools: a D365-shaped ERP, an Odoo-shaped procure-to-pay and manufacturing surface, a QuickBooks-style subsidiary ledger, a shared drive, email, document management, a reporting harness — and frozen real SEC XBRL filings (46,686 facts across 38 registrants, snapshot-pinned public data). The per-task world is 61 SQLite tables holding 68,098 seeded rows, with the clock frozen at 2026-03-02.

In-fiction persona tasks

Prompts are persona chat messages inside the company's workflow. The graded answer contract is discovered through the harness server's reporting_fields tool — the way a real reporting system's schema is read before filing into it. Oracle walks run 3–104 MCP calls (median 6, 1,515 total) across 22 task families.

Deterministic, binary grading

Reward is binary and fully deterministic: 955 graded checks across the release — 434 answer checks with typed tolerances, 257 trace checks (required servers, reads before submission), and 264 state checks that grade the world the agent leaves behind (committed payment runs, paid/rejected partitions, reason codes) — plus a writes_only anti-hack veto. No LLM judge, no network, no clock in the reward path.

Prompt duplication is deliberate — and disclosed

30 tasks are escalations of a base task that is also in the release. 25 of them (doc_mode = "buried") reuse the base persona message verbatim against a harder world where the governing policy must be found among seeded decoy documents — so those prompt texts appear twice by design, and the release carries 75 distinct prompts. Treat prompt-level dedup accordingly.

Qualification evidence

The release gate executed 600 runs: 100 oracle replays all passing with reward 1.0, 100 byte-identical deterministic replays, and 400 negative-control executions (no-submit, no-op, off-task-write, wrong-submit) with zero false accepts. Eight released task packs were additionally probed end-to-end in Docker through Harbor, 8/8 passing. Full per-task evidence ships in reports/qualification.json.

How to read the deepseek-v4-pro row

The deepseek-v4-pro row comes from a pre-release run of the source world (sim/run_batch.py, 3 trials per task) mapped onto the release by source-task id — it covers only the released tasks that run exercised, and none of the escalated buried-policy variants. It is published as an honest signal that the world separates models, not as a full-benchmark claim; ranked rows over the released Harbor dataset will replace it.

Run it yourself

The benchmark is public. The world is executable.

Download the dataset, replay the oracle walks, inspect all 955 graded checks, and submit the first pinned run over the released tasks.

Open Harbor world ↗Download dataset ↗