Skip to benchmark
Blobfish AI
Talk to usOpen Studio
Blobfish ResearchCounselBench-100 v1.1.0Public

Can an agent finish a legal matter—not just answer a question?

CounselBench-100 measures complete legal work: inventory a production, read every record, reconcile conflicts, preserve custody evidence, and deliver source-grounded advice through real MCP tools.

View leaderboard Browse all 100 tasks
Run on Harbor ↗Dataset on Hugging Face ↗Source ↗
100legal tasks
9,600seeded documents
109+MCP calls per task
10practice areas
0LLM grading calls
v1.1 scoring correction

The original exact-match display collapsed ten completed model runs into the same zero. The underlying runs were real; the public metric was not diagnostic. The current release grades 182 recoverable criteria while retaining strict pass as a separate signal.

See the contract change →

Measured performance

Leaderboard

The headline metric is mean deterministic criteria score. Strict pass remains visible, but a single missed field no longer erases every correct read, fact, and citation.

REF
Oracle referenceHarbor reference solver
100 tasks
100.0%
109 calls avg.Reference ceiling; not a model submission.
1
gpt-5.6-solcodex 0.149.1
10 tasks
98.7%
118 calls avg.$0.84 / task
25%Procedure

Inventory, 96 full reads, custody checks, and MCP-written outputs.

55%Findings

Field-level identity, evidence paths, source-bounded facts, owner, and deadline.

20%Memo

Required sections and complete grounding for every finding.

100 distinct matters

Task browser

Search the full catalog. Ten audit samples—one per practice area—include the complete prompt, 96-file production, rubric, and gold evidence map.

100 shown
cb100-001-helios-forgeCorporate M&A

Project Daybreak battery acquisition

A carve-out manufacturer owns cathode tooling in Nevada while its largest automaker customer can terminate upon an indirect change of control. Prioritize conflicts between signed instruments and later operational records.

Client
Helios Forge, Inc.
Matter
CB-MA-2401
Deadline
2026-09-04
Minimum path
109 MCP calls

Complete task prompt

16 independently graded findings
# Project Daybreak battery acquisition

You are acting as buyer-side transactions counsel for Helios Forge, Inc.. The partner responsible for this matter needs a defensible decision record before the next client call.

Matter: CB-MA-2401
Other party or authority: BluePeak Battery Systems LLC
Jurisdiction: Delaware
Venue or forum: Washoe County, Nevada
Decision deadline: 2026-09-04

## Assignment

A carve-out manufacturer owns cathode tooling in Nevada while its largest automaker customer can terminate upon an indirect change of control. Prioritize conflicts between signed instruments and later operational records.

Reconcile the complete seeded production and prepare an exception tracker plus a concise counsel memo. The data room contains 96 text-native legal and business records in 12 folders. Some facts can be evaluated only by comparing a primary record in one folder with a later or independently maintained record elsewhere. Do not rely on filenames alone, do not assume that a later record controls, and do not use facts outside this production.

Finding-bearing records identify the review question, stable finding ID, record role, severity, remediation owner, and response due date in their record-control metadata. Preserve those values exactly. Records labeled `context only` are responsive background, not independent findings.

Your review scope includes these independently tested legal and control questions:

- charter authorization mismatch
- unrecorded option grant
- change-of-control consent
- anti-assignment restriction
- most-favored-customer exposure
- financial covenant breach
- key-person retention gap
- invention assignment defect
- open-source copyleft exposure
- unreported security incident
- permit transfer restriction
- threatened customer dispute
- facility title exception
- sales-tax nexus gap
- claims-made insurance tail
- revenue recognition exception

## Required review procedure

Use the `io.github.modelcontextprotocol/server-filesystem` MCP tools exposed by the environment.

1. Call `list_allowed_directories` and confirm the available roots.
2. Call `directory_tree` on `/workspace/documents` to inventory the complete production.
3. Call `search_files` on `/workspace/documents` with pattern `**/*.eml`.
4. Call `read_text_file` separately, without `head` or `tail`, for every one of the 96 files. Bulk reads are intentionally unavailable because the work product must have file-level provenance.
5. Call `get_file_info` for each of these eight chain-of-custody sample records:

- `/workspace/documents/01_corporate/001_corporate_control_register.md`
- `/workspace/documents/01_corporate/004_corporate_ledger_export.csv`
- `/workspace/documents/01_corporate/007_corporate_officer_certificate.html`
- `/workspace/documents/02_capitalization/010_capitalization_executed_instrument.txt`
- `/workspace/documents/02_capitalization/013_capitalization_review_memorandum.json`
- `/workspace/documents/02_capitalization/016_capitalization_status_report.md`
- `/workspace/documents/03_material_contracts/019_material_contracts_correspondence.eml`
- `/workspace/documents/03_material_contracts/022_material_contracts_formal_notice.xml`

6. Use `write_file` to create both final deliverables under `/workspace/output`.

An accepted trajectory therefore contains at least 109 successful MCP tool calls. The verifier checks the call trace and final filesystem state; merely stating that records were reviewed is insufficient.

## Deliverables

Create exactly these two files and no others:

- `/workspace/output/findings.json`
- `/workspace/output/advice.md`

`findings.json` must be valid JSON with this exact top-level shape:

```json
{
  "schema_version": "1.0",
  "task_id": "cb100-001-helios-forge",
  "matter_number": "CB-MA-2401",
  "prepared_for": "Helios Forge, Inc.",
  "as_of": "2026-09-04",
  "findings": [
    {
      "id": "F-01",
      "severity": "critical|high|medium|low",
      "issue": "short issue label",
      "primary_source": "/workspace/documents/...",
      "corroborating_source": "/workspace/documents/...",
      "determination": "precise reconciliation of the conflicting facts",
      "recommended_action": "dated, owned next step"
    }
  ]
}
```

Use one row per confirmed exception, stable IDs beginning at `F-01`, exact absolute source paths, and exact dates, amounts, names, statuses, thresholds, or counts from the cited records.

`advice.md` must contain these headings: `Executive assessment`, `Method and record coverage`, `Findings`, `Recommended next actions`, and `Assumptions and limitations`. Address every JSON finding in the memo with its ID, issue, severity, both source paths, determination, and recommended action. Keep confirmed contradictions distinct from assumptions or unresolved legal questions.

All facts in this task are synthetic. The work product is benchmark output, not legal advice.
Open task record on Hugging Face

Seeded evidence

Asset room

Every task starts with 96 immutable, text-native records across 12 workstream folders. Finding-bearing records expose recoverable control metadata; the rest are realistic context and distractors.

001_corporate_control_register.md01_corporateF-01 · primaryMD6.5 KB

# Control Register — CB-DOC-001-001 > Confidential — legal review · reviewed — source conflict identified · native version 2.1 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak batter

charter authorization mismatch · critical
002_corporate_executed_instrument.txt01_corporateContext · contextTXT5.9 KB

CONFIDENTIAL — MATTER TEAM EXECUTED INSTRUMENT PROJECT DAYBREAK BATTERY ACQUISITION DOCUMENT CONTROL: CB-DOC-001-002 MATTER: CB-MA-2401 | Project Daybreak battery acquisition RECORD TYPE: executed instrument DA

Context only
003_corporate_correspondence.eml01_corporateContext · contextEML6.1 KB

From: nora.chen@example.test To: micah.sullivan@example.test Cc: matter-team-helios_forge_inc@example.test Date: 2026-02-11 09:30:00 -0700 Message-ID: In-Reply-To: Subject: CB-MA-2401 — correspondence / 01_corp

Context only
004_corporate_ledger_export.csv01_corporateF-02 · primaryCSV12.3 KB

record_id,matter_number,record_date,row_type,section,key,value,owner,status,reference CB-DOC-001-004,CB-MA-2401,2024-11-03,metadata,document_control,record_id,CB-DOC-001-004,,,CB-DOC-001-061 CB-DOC-001-004,CB-M

unrecorded option grant · high
005_corporate_review_memorandum.json01_corporateContext · contextJSON12.7 KB

{ "document_control": { "record_id": "CB-DOC-001-005", "matter_number": "CB-MA-2401", "record_date": "2024-06-15", "record_type": "review memorandum", "folder": "01_corporate", "workstream": "corporate", "sourc

Context only
006_corporate_formal_notice.xml01_corporateContext · contextXML7.0 KB

CB-DOC-001-006 CB-MA-2401 2024-09-25 formal notice 01_corporate corporate Workday HRIS 1.3 reviewed — no independent exception Attorney work product Elliot Mercer Amara Patel CB-DOC-001-095 464208 context only

Context only
007_corporate_officer_certificate.html01_corporateF-03 · primaryHTML7.3 KB

CB-DOC-001-007 body{font-family:Georgia,serif;max-width:960px;margin:40px auto;line-height:1.5;color:#17242a}table{width:100%;border-collapse:collapse;margin:18px 0 30px}th,td{border:1px solid #aeb8bc;padding:8

change-of-control consent · high
008_corporate_status_report.md01_corporateContext · contextMD6.4 KB

# Status Report — CB-DOC-001-008 > Confidential — legal review · reviewed — no independent exception · native version 4.2 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acq

Context only
009_capitalization_control_register.md02_capitalizationContext · contextMD6.4 KB

# Control Register — CB-DOC-001-009 > Attorney work product · indexed — responsive context · native version 2.8 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquisition |

Context only
010_capitalization_executed_instrument.txt02_capitalizationF-04 · primaryTXT5.9 KB

CONFIDENTIAL — LEGAL REVIEW EXECUTED INSTRUMENT PROJECT DAYBREAK BATTERY ACQUISITION DOCUMENT CONTROL: CB-DOC-001-010 MATTER: CB-MA-2401 | Project Daybreak battery acquisition RECORD TYPE: executed instrument D

anti-assignment restriction · medium
011_capitalization_correspondence.eml02_capitalizationContext · contextEML6.3 KB

From: leona.park@example.test To: dominic.alvarez@example.test Cc: matter-team-helios_forge_inc@example.test Date: 2024-04-08 09:30:00 -0700 Message-ID: In-Reply-To: Subject: CB-MA-2401 — correspondence / 02_ca

Context only
012_capitalization_ledger_export.csv02_capitalizationContext · contextCSV12.5 KB

record_id,matter_number,record_date,row_type,section,key,value,owner,status,reference CB-DOC-001-012,CB-MA-2401,2025-02-12,metadata,document_control,record_id,CB-DOC-001-012,,,CB-DOC-001-005 CB-DOC-001-012,CB-M

Context only
013_capitalization_review_memorandum.json02_capitalizationF-05 · primaryJSON12.6 KB

{ "document_control": { "record_id": "CB-DOC-001-013", "matter_number": "CB-MA-2401", "record_date": "2025-08-31", "record_type": "review memorandum", "folder": "02_capitalization", "workstream": "capitalizatio

most-favored-customer exposure · medium
014_capitalization_formal_notice.xml02_capitalizationContext · contextXML7.1 KB

CB-DOC-001-014 CB-MA-2401 2024-10-28 formal notice 02_capitalization capitalization Carta capitalization ledger 2.8 reviewed — no independent exception Attorney work product Adrian Mensah Maya Ellison CB-DOC-00

Context only
015_capitalization_officer_certificate.html02_capitalizationContext · contextHTML7.3 KB

CB-DOC-001-015 body{font-family:Georgia,serif;max-width:960px;margin:40px auto;line-height:1.5;color:#17242a}table{width:100%;border-collapse:collapse;margin:18px 0 30px}th,td{border:1px solid #aeb8bc;padding:8

Context only
016_capitalization_status_report.md02_capitalizationF-06 · primaryMD6.4 KB

# Status Report — CB-DOC-001-016 > Confidential — legal review · reviewed — source conflict identified · native version 1.9 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery a

financial covenant breach · low
017_material_contracts_control_register.md03_material_contractsContext · contextMD6.5 KB

# Control Register — CB-DOC-001-017 > Restricted — need to know · indexed — responsive context · native version 4.6 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquisiti

Context only
018_material_contracts_executed_instrument.txt03_material_contractsContext · contextTXT5.8 KB

CONFIDENTIAL — MATTER TEAM EXECUTED INSTRUMENT PROJECT DAYBREAK BATTERY ACQUISITION DOCUMENT CONTROL: CB-DOC-001-018 MATTER: CB-MA-2401 | Project Daybreak battery acquisition RECORD TYPE: executed instrument DA

Context only
019_material_contracts_correspondence.eml03_material_contractsF-07 · primaryEML6.2 KB

From: nadine.flores@example.test To: isaac.romero@example.test Cc: matter-team-helios_forge_inc@example.test Date: 2025-08-09 09:30:00 -0700 Message-ID: In-Reply-To: Subject: CB-MA-2401 — correspondence / 03_ma

key-person retention gap · critical
020_material_contracts_ledger_export.csv03_material_contractsContext · contextCSV12.7 KB

record_id,matter_number,record_date,row_type,section,key,value,owner,status,reference CB-DOC-001-020,CB-MA-2401,2025-03-11,metadata,document_control,record_id,CB-DOC-001-020,,,CB-DOC-001-045 CB-DOC-001-020,CB-M

Context only
021_material_contracts_review_memorandum.json03_material_contractsContext · contextJSON12.8 KB

{ "document_control": { "record_id": "CB-DOC-001-021", "matter_number": "CB-MA-2401", "record_date": "2025-06-29", "record_type": "review memorandum", "folder": "03_material_contracts", "workstream": "material

Context only
022_material_contracts_formal_notice.xml03_material_contractsF-08 · primaryXML7.2 KB

CB-DOC-001-022 CB-MA-2401 2024-10-30 formal notice 03_material_contracts material contracts NetSuite close workbook 4.9 reviewed — source conflict identified Restricted — need to know Owen Delgado Mei Whitaker

invention assignment defect · high
023_material_contracts_officer_certificate.html03_material_contractsContext · contextHTML7.3 KB

CB-DOC-001-023 body{font-family:Georgia,serif;max-width:960px;margin:40px auto;line-height:1.5;color:#17242a}table{width:100%;border-collapse:collapse;margin:18px 0 30px}th,td{border:1px solid #aeb8bc;padding:8

Context only
024_material_contracts_status_report.md03_material_contractsContext · contextMD6.4 KB

# Status Report — CB-DOC-001-024 > Confidential — matter team · reviewed — no independent exception · native version 3.1 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acqu

Context only
025_debt_security_control_register.md04_debt_securityF-09 · primaryMD6.5 KB

# Control Register — CB-DOC-001-025 > Attorney work product · reviewed — source conflict identified · native version 1.4 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acqu

open-source copyleft exposure · high
026_debt_security_executed_instrument.txt04_debt_securityContext · contextTXT5.9 KB

CONFIDENTIAL — MATTER TEAM EXECUTED INSTRUMENT PROJECT DAYBREAK BATTERY ACQUISITION DOCUMENT CONTROL: CB-DOC-001-026 MATTER: CB-MA-2401 | Project Daybreak battery acquisition RECORD TYPE: executed instrument DA

Context only
027_debt_security_correspondence.eml04_debt_securityContext · contextEML6.2 KB

From: nora.chen@example.test To: micah.sullivan@example.test Cc: matter-team-helios_forge_inc@example.test Date: 2025-12-03 09:30:00 -0700 Message-ID: In-Reply-To: Subject: CB-MA-2401 — correspondence / 04_debt

Context only
028_debt_security_ledger_export.csv04_debt_securityF-10 · primaryCSV12.3 KB

record_id,matter_number,record_date,row_type,section,key,value,owner,status,reference CB-DOC-001-028,CB-MA-2401,2025-12-22,metadata,document_control,record_id,CB-DOC-001-028,,,CB-DOC-001-085 CB-DOC-001-028,CB-M

unreported security incident · medium
029_debt_security_review_memorandum.json04_debt_securityContext · contextJSON12.7 KB

{ "document_control": { "record_id": "CB-DOC-001-029", "matter_number": "CB-MA-2401", "record_date": "2024-05-17", "record_type": "review memorandum", "folder": "04_debt_security", "workstream": "debt security"

Context only
030_debt_security_formal_notice.xml04_debt_securityContext · contextXML7.0 KB

CB-DOC-001-030 CB-MA-2401 2024-05-30 formal notice 04_debt_security debt security Workday HRIS 3.0 reviewed — no independent exception Confidential — legal review Elliot Mercer Amara Patel CB-DOC-001-023 86810

Context only
031_debt_security_officer_certificate.html04_debt_securityF-11 · primaryHTML7.3 KB

CB-DOC-001-031 body{font-family:Georgia,serif;max-width:960px;margin:40px auto;line-height:1.5;color:#17242a}table{width:100%;border-collapse:collapse;margin:18px 0 30px}th,td{border:1px solid #aeb8bc;padding:8

permit transfer restriction · medium
032_debt_security_status_report.md04_debt_securityContext · contextMD6.5 KB

# Status Report — CB-DOC-001-032 > Confidential — legal review · reviewed — no independent exception · native version 3.1 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acq

Context only
033_employment_control_register.md05_employmentContext · contextMD6.4 KB

# Control Register — CB-DOC-001-033 > Attorney work product · indexed — responsive context · native version 3.5 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquisition |

Context only
034_employment_executed_instrument.txt05_employmentF-12 · primaryTXT5.8 KB

CONFIDENTIAL — LEGAL REVIEW EXECUTED INSTRUMENT PROJECT DAYBREAK BATTERY ACQUISITION DOCUMENT CONTROL: CB-DOC-001-034 MATTER: CB-MA-2401 | Project Daybreak battery acquisition RECORD TYPE: executed instrument D

threatened customer dispute · low
035_employment_correspondence.eml05_employmentContext · contextEML6.2 KB

From: leona.park@example.test To: dominic.alvarez@example.test Cc: matter-team-helios_forge_inc@example.test Date: 2025-12-05 09:30:00 -0700 Message-ID: In-Reply-To: Subject: CB-MA-2401 — correspondence / 05_em

Context only
036_employment_ledger_export.csv05_employmentContext · contextCSV12.4 KB

record_id,matter_number,record_date,row_type,section,key,value,owner,status,reference CB-DOC-001-036,CB-MA-2401,2024-11-24,metadata,document_control,record_id,CB-DOC-001-036,,,CB-DOC-001-029 CB-DOC-001-036,CB-M

Context only
037_employment_review_memorandum.json05_employmentF-13 · primaryJSON12.6 KB

{ "document_control": { "record_id": "CB-DOC-001-037", "matter_number": "CB-MA-2401", "record_date": "2025-12-15", "record_type": "review memorandum", "folder": "05_employment", "workstream": "employment", "sou

facility title exception · critical
038_employment_formal_notice.xml05_employmentContext · contextXML7.2 KB

CB-DOC-001-038 CB-MA-2401 2024-12-11 formal notice 05_employment employment Carta capitalization ledger 2.1 reviewed — no independent exception Confidential — legal review Adrian Mensah Maya Ellison CB-DOC-001-

Context only
039_employment_officer_certificate.html05_employmentContext · contextHTML7.2 KB

CB-DOC-001-039 body{font-family:Georgia,serif;max-width:960px;margin:40px auto;line-height:1.5;color:#17242a}table{width:100%;border-collapse:collapse;margin:18px 0 30px}th,td{border:1px solid #aeb8bc;padding:8

Context only
040_employment_status_report.md05_employmentF-14 · primaryMD6.3 KB

# Status Report — CB-DOC-001-040 > Confidential — matter team · reviewed — source conflict identified · native version 1.0 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery ac

sales-tax nexus gap · high
041_ip_technology_control_register.md06_ip_technologyContext · contextMD6.4 KB

# Control Register — CB-DOC-001-041 > Confidential — legal review · indexed — responsive context · native version 4.8 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquisi

Context only
042_ip_technology_executed_instrument.txt06_ip_technologyContext · contextTXT5.8 KB

RESTRICTED — NEED TO KNOW EXECUTED INSTRUMENT PROJECT DAYBREAK BATTERY ACQUISITION DOCUMENT CONTROL: CB-DOC-001-042 MATTER: CB-MA-2401 | Project Daybreak battery acquisition RECORD TYPE: executed instrument DAT

Context only
043_ip_technology_correspondence.eml06_ip_technologyF-15 · primaryEML6.3 KB

From: nadine.flores@example.test To: isaac.romero@example.test Cc: matter-team-helios_forge_inc@example.test Date: 2025-05-27 09:30:00 -0700 Message-ID: In-Reply-To: Subject: CB-MA-2401 — correspondence / 06_ip

claims-made insurance tail · high
044_ip_technology_ledger_export.csv06_ip_technologyContext · contextCSV12.6 KB

record_id,matter_number,record_date,row_type,section,key,value,owner,status,reference CB-DOC-001-044,CB-MA-2401,2025-11-13,metadata,document_control,record_id,CB-DOC-001-044,,,CB-DOC-001-069 CB-DOC-001-044,CB-M

Context only
045_ip_technology_review_memorandum.json06_ip_technologyContext · contextJSON12.7 KB

{ "document_control": { "record_id": "CB-DOC-001-045", "matter_number": "CB-MA-2401", "record_date": "2026-01-29", "record_type": "review memorandum", "folder": "06_ip_technology", "workstream": "ip technology"

Context only
046_ip_technology_formal_notice.xml06_ip_technologyF-16 · primaryXML7.1 KB

CB-DOC-001-046 CB-MA-2401 2025-10-17 formal notice 06_ip_technology ip technology NetSuite close workbook 4.5 reviewed — source conflict identified Restricted — need to know Owen Delgado Mei Whitaker CB-DOC-001

revenue recognition exception · medium
047_ip_technology_officer_certificate.html06_ip_technologyContext · contextHTML7.3 KB

CB-DOC-001-047 body{font-family:Georgia,serif;max-width:960px;margin:40px auto;line-height:1.5;color:#17242a}table{width:100%;border-collapse:collapse;margin:18px 0 30px}th,td{border:1px solid #aeb8bc;padding:8

Context only
048_ip_technology_status_report.md06_ip_technologyContext · contextMD6.4 KB

# Status Report — CB-DOC-001-048 > Confidential — matter team · reviewed — no independent exception · native version 3.4 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acqu

Context only
049_privacy_security_control_register.md07_privacy_securityF-01 · corroboratingMD6.5 KB

# Control Register — CB-DOC-001-049 > Restricted — need to know · reviewed — variance confirmed · native version 2.4 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquisit

charter authorization mismatch · critical
050_privacy_security_executed_instrument.txt07_privacy_securityContext · contextTXT5.9 KB

ATTORNEY WORK PRODUCT EXECUTED INSTRUMENT PROJECT DAYBREAK BATTERY ACQUISITION DOCUMENT CONTROL: CB-DOC-001-050 MATTER: CB-MA-2401 | Project Daybreak battery acquisition RECORD TYPE: executed instrument DATE: 2

Context only
051_privacy_security_correspondence.eml07_privacy_securityContext · contextEML6.2 KB

From: nora.chen@example.test To: micah.sullivan@example.test Cc: matter-team-helios_forge_inc@example.test Date: 2025-04-18 09:30:00 -0700 Message-ID: In-Reply-To: Subject: CB-MA-2401 — correspondence / 07_priv

Context only
052_privacy_security_ledger_export.csv07_privacy_securityF-02 · corroboratingCSV12.4 KB

record_id,matter_number,record_date,row_type,section,key,value,owner,status,reference CB-DOC-001-052,CB-MA-2401,2025-12-25,metadata,document_control,record_id,CB-DOC-001-052,,,CB-DOC-001-013 CB-DOC-001-052,CB-M

unrecorded option grant · high
053_privacy_security_review_memorandum.json07_privacy_securityContext · contextJSON12.8 KB

{ "document_control": { "record_id": "CB-DOC-001-053", "matter_number": "CB-MA-2401", "record_date": "2025-08-29", "record_type": "review memorandum", "folder": "07_privacy_security", "workstream": "privacy sec

Context only
054_privacy_security_formal_notice.xml07_privacy_securityContext · contextXML7.0 KB

CB-DOC-001-054 CB-MA-2401 2024-03-02 formal notice 07_privacy_security privacy security Workday HRIS 2.8 reviewed — no independent exception Confidential — legal review Elliot Mercer Amara Patel CB-DOC-001-047

Context only
055_privacy_security_officer_certificate.html07_privacy_securityF-03 · corroboratingHTML7.3 KB

CB-DOC-001-055 body{font-family:Georgia,serif;max-width:960px;margin:40px auto;line-height:1.5;color:#17242a}table{width:100%;border-collapse:collapse;margin:18px 0 30px}th,td{border:1px solid #aeb8bc;padding:8

change-of-control consent · high
056_privacy_security_status_report.md07_privacy_securityContext · contextMD6.5 KB

# Status Report — CB-DOC-001-056 > Attorney work product · reviewed — no independent exception · native version 1.2 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquisiti

Context only
057_regulatory_control_register.md08_regulatoryContext · contextMD6.4 KB

# Control Register — CB-DOC-001-057 > Confidential — legal review · indexed — responsive context · native version 4.9 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquisi

Context only
058_regulatory_executed_instrument.txt08_regulatoryF-04 · corroboratingTXT5.9 KB

ATTORNEY WORK PRODUCT EXECUTED INSTRUMENT PROJECT DAYBREAK BATTERY ACQUISITION DOCUMENT CONTROL: CB-DOC-001-058 MATTER: CB-MA-2401 | Project Daybreak battery acquisition RECORD TYPE: executed instrument DATE: 2

anti-assignment restriction · medium
059_regulatory_correspondence.eml08_regulatoryContext · contextEML6.3 KB

From: leona.park@example.test To: dominic.alvarez@example.test Cc: matter-team-helios_forge_inc@example.test Date: 2025-05-30 09:30:00 -0700 Message-ID: In-Reply-To: Subject: CB-MA-2401 — correspondence / 08_re

Context only
060_regulatory_ledger_export.csv08_regulatoryContext · contextCSV12.4 KB

record_id,matter_number,record_date,row_type,section,key,value,owner,status,reference CB-DOC-001-060,CB-MA-2401,2025-01-12,metadata,document_control,record_id,CB-DOC-001-060,,,CB-DOC-001-053 CB-DOC-001-060,CB-M

Context only
061_regulatory_review_memorandum.json08_regulatoryF-05 · corroboratingJSON12.7 KB

{ "document_control": { "record_id": "CB-DOC-001-061", "matter_number": "CB-MA-2401", "record_date": "2025-03-01", "record_type": "review memorandum", "folder": "08_regulatory", "workstream": "regulatory", "sou

most-favored-customer exposure · medium
062_regulatory_formal_notice.xml08_regulatoryContext · contextXML7.1 KB

CB-DOC-001-062 CB-MA-2401 2024-09-27 formal notice 08_regulatory regulatory Carta capitalization ledger 3.7 reviewed — no independent exception Restricted — need to know Adrian Mensah Maya Ellison CB-DOC-001-08

Context only
063_regulatory_officer_certificate.html08_regulatoryContext · contextHTML7.2 KB

CB-DOC-001-063 body{font-family:Georgia,serif;max-width:960px;margin:40px auto;line-height:1.5;color:#17242a}table{width:100%;border-collapse:collapse;margin:18px 0 30px}th,td{border:1px solid #aeb8bc;padding:8

Context only
064_regulatory_status_report.md08_regulatoryF-06 · corroboratingMD6.5 KB

# Status Report — CB-DOC-001-064 > Confidential — legal review · reviewed — variance confirmed · native version 1.7 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquisiti

financial covenant breach · low
065_litigation_control_register.md09_litigationContext · contextMD6.4 KB

# Control Register — CB-DOC-001-065 > Restricted — need to know · indexed — responsive context · native version 3.3 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquisiti

Context only
066_litigation_executed_instrument.txt09_litigationContext · contextTXT5.8 KB

CONFIDENTIAL — MATTER TEAM EXECUTED INSTRUMENT PROJECT DAYBREAK BATTERY ACQUISITION DOCUMENT CONTROL: CB-DOC-001-066 MATTER: CB-MA-2401 | Project Daybreak battery acquisition RECORD TYPE: executed instrument DA

Context only
067_litigation_correspondence.eml09_litigationF-07 · corroboratingEML6.3 KB

From: nadine.flores@example.test To: isaac.romero@example.test Cc: matter-team-helios_forge_inc@example.test Date: 2024-03-19 09:30:00 -0700 Message-ID: In-Reply-To: Subject: CB-MA-2401 — correspondence / 09_li

key-person retention gap · critical
068_litigation_ledger_export.csv09_litigationContext · contextCSV12.5 KB

record_id,matter_number,record_date,row_type,section,key,value,owner,status,reference CB-DOC-001-068,CB-MA-2401,2024-07-29,metadata,document_control,record_id,CB-DOC-001-068,,,CB-DOC-001-093 CB-DOC-001-068,CB-M

Context only
069_litigation_review_memorandum.json09_litigationContext · contextJSON12.7 KB

{ "document_control": { "record_id": "CB-DOC-001-069", "matter_number": "CB-MA-2401", "record_date": "2025-04-01", "record_type": "review memorandum", "folder": "09_litigation", "workstream": "litigation", "sou

Context only
070_litigation_formal_notice.xml09_litigationF-08 · corroboratingXML7.1 KB

CB-DOC-001-070 CB-MA-2401 2025-03-31 formal notice 09_litigation litigation NetSuite close workbook 4.8 reviewed — variance confirmed Attorney work product Owen Delgado Mei Whitaker CB-DOC-001-031 476555 invent

invention assignment defect · high
071_litigation_officer_certificate.html09_litigationContext · contextHTML7.3 KB

CB-DOC-001-071 body{font-family:Georgia,serif;max-width:960px;margin:40px auto;line-height:1.5;color:#17242a}table{width:100%;border-collapse:collapse;margin:18px 0 30px}th,td{border:1px solid #aeb8bc;padding:8

Context only
072_litigation_status_report.md09_litigationContext · contextMD6.3 KB

# Status Report — CB-DOC-001-072 > Restricted — need to know · reviewed — no independent exception · native version 1.8 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acqui

Context only
073_real_property_control_register.md10_real_propertyF-09 · corroboratingMD6.5 KB

# Control Register — CB-DOC-001-073 > Confidential — legal review · reviewed — variance confirmed · native version 3.6 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquis

open-source copyleft exposure · high
074_real_property_executed_instrument.txt10_real_propertyContext · contextTXT5.9 KB

CONFIDENTIAL — MATTER TEAM EXECUTED INSTRUMENT PROJECT DAYBREAK BATTERY ACQUISITION DOCUMENT CONTROL: CB-DOC-001-074 MATTER: CB-MA-2401 | Project Daybreak battery acquisition RECORD TYPE: executed instrument DA

Context only
075_real_property_correspondence.eml10_real_propertyContext · contextEML6.2 KB

From: nora.chen@example.test To: micah.sullivan@example.test Cc: matter-team-helios_forge_inc@example.test Date: 2024-08-29 09:30:00 -0700 Message-ID: In-Reply-To: Subject: CB-MA-2401 — correspondence / 10_real

Context only
076_real_property_ledger_export.csv10_real_propertyF-10 · corroboratingCSV12.4 KB

record_id,matter_number,record_date,row_type,section,key,value,owner,status,reference CB-DOC-001-076,CB-MA-2401,2025-11-16,metadata,document_control,record_id,CB-DOC-001-076,,,CB-DOC-001-037 CB-DOC-001-076,CB-M

unreported security incident · medium
077_real_property_review_memorandum.json10_real_propertyContext · contextJSON12.7 KB

{ "document_control": { "record_id": "CB-DOC-001-077", "matter_number": "CB-MA-2401", "record_date": "2025-11-01", "record_type": "review memorandum", "folder": "10_real_property", "workstream": "real property"

Context only
078_real_property_formal_notice.xml10_real_propertyContext · contextXML7.1 KB

CB-DOC-001-078 CB-MA-2401 2025-10-20 formal notice 10_real_property real property Workday HRIS 3.0 reviewed — no independent exception Confidential — legal review Elliot Mercer Amara Patel CB-DOC-001-071 291318

Context only
079_real_property_officer_certificate.html10_real_propertyF-11 · corroboratingHTML7.3 KB

CB-DOC-001-079 body{font-family:Georgia,serif;max-width:960px;margin:40px auto;line-height:1.5;color:#17242a}table{width:100%;border-collapse:collapse;margin:18px 0 30px}th,td{border:1px solid #aeb8bc;padding:8

permit transfer restriction · medium
080_real_property_status_report.md10_real_propertyContext · contextMD6.5 KB

# Status Report — CB-DOC-001-080 > Attorney work product · reviewed — no independent exception · native version 1.5 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquisiti

Context only
081_tax_insurance_control_register.md11_tax_insuranceContext · contextMD6.5 KB

# Control Register — CB-DOC-001-081 > Confidential — legal review · indexed — responsive context · native version 2.9 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquisi

Context only
082_tax_insurance_executed_instrument.txt11_tax_insuranceF-12 · corroboratingTXT5.9 KB

ATTORNEY WORK PRODUCT EXECUTED INSTRUMENT PROJECT DAYBREAK BATTERY ACQUISITION DOCUMENT CONTROL: CB-DOC-001-082 MATTER: CB-MA-2401 | Project Daybreak battery acquisition RECORD TYPE: executed instrument DATE: 2

threatened customer dispute · low
083_tax_insurance_correspondence.eml11_tax_insuranceContext · contextEML6.3 KB

From: leona.park@example.test To: dominic.alvarez@example.test Cc: matter-team-helios_forge_inc@example.test Date: 2025-03-22 09:30:00 -0700 Message-ID: In-Reply-To: Subject: CB-MA-2401 — correspondence / 11_ta

Context only
084_tax_insurance_ledger_export.csv11_tax_insuranceContext · contextCSV12.4 KB

record_id,matter_number,record_date,row_type,section,key,value,owner,status,reference CB-DOC-001-084,CB-MA-2401,2025-09-01,metadata,document_control,record_id,CB-DOC-001-084,,,CB-DOC-001-077 CB-DOC-001-084,CB-M

Context only
085_tax_insurance_review_memorandum.json11_tax_insuranceF-13 · corroboratingJSON12.7 KB

{ "document_control": { "record_id": "CB-DOC-001-085", "matter_number": "CB-MA-2401", "record_date": "2025-09-21", "record_type": "review memorandum", "folder": "11_tax_insurance", "workstream": "tax insurance"

facility title exception · critical
086_tax_insurance_formal_notice.xml11_tax_insuranceContext · contextXML7.1 KB

CB-DOC-001-086 CB-MA-2401 2025-06-04 formal notice 11_tax_insurance tax insurance Carta capitalization ledger 1.4 reviewed — no independent exception Confidential — legal review Adrian Mensah Maya Ellison CB-DO

Context only
087_tax_insurance_officer_certificate.html11_tax_insuranceContext · contextHTML7.2 KB

CB-DOC-001-087 body{font-family:Georgia,serif;max-width:960px;margin:40px auto;line-height:1.5;color:#17242a}table{width:100%;border-collapse:collapse;margin:18px 0 30px}th,td{border:1px solid #aeb8bc;padding:8

Context only
088_tax_insurance_status_report.md11_tax_insuranceF-14 · corroboratingMD6.5 KB

# Status Report — CB-DOC-001-088 > Attorney work product · reviewed — variance confirmed · native version 1.0 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquisition | |

sales-tax nexus gap · high
089_financials_control_register.md12_financialsContext · contextMD6.5 KB

# Control Register — CB-DOC-001-089 > Confidential — matter team · indexed — responsive context · native version 1.4 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acquisit

Context only
090_financials_executed_instrument.txt12_financialsContext · contextTXT5.7 KB

CONFIDENTIAL — LEGAL REVIEW EXECUTED INSTRUMENT PROJECT DAYBREAK BATTERY ACQUISITION DOCUMENT CONTROL: CB-DOC-001-090 MATTER: CB-MA-2401 | Project Daybreak battery acquisition RECORD TYPE: executed instrument D

Context only
091_financials_correspondence.eml12_financialsF-15 · corroboratingEML6.3 KB

From: nadine.flores@example.test To: isaac.romero@example.test Cc: matter-team-helios_forge_inc@example.test Date: 2024-12-31 09:30:00 -0700 Message-ID: In-Reply-To: Subject: CB-MA-2401 — correspondence / 12_fi

claims-made insurance tail · high
092_financials_ledger_export.csv12_financialsContext · contextCSV12.5 KB

record_id,matter_number,record_date,row_type,section,key,value,owner,status,reference CB-DOC-001-092,CB-MA-2401,2026-02-13,metadata,document_control,record_id,CB-DOC-001-092,,,CB-DOC-001-021 CB-DOC-001-092,CB-M

Context only
093_financials_review_memorandum.json12_financialsContext · contextJSON12.5 KB

{ "document_control": { "record_id": "CB-DOC-001-093", "matter_number": "CB-MA-2401", "record_date": "2025-03-18", "record_type": "review memorandum", "folder": "12_financials", "workstream": "financials", "sou

Context only
094_financials_formal_notice.xml12_financialsF-16 · corroboratingXML7.1 KB

CB-DOC-001-094 CB-MA-2401 2024-12-21 formal notice 12_financials financials NetSuite close workbook 4.9 reviewed — variance confirmed Restricted — need to know Owen Delgado Mei Whitaker CB-DOC-001-055 246215 re

revenue recognition exception · medium
095_financials_officer_certificate.html12_financialsContext · contextHTML7.2 KB

CB-DOC-001-095 body{font-family:Georgia,serif;max-width:960px;margin:40px auto;line-height:1.5;color:#17242a}table{width:100%;border-collapse:collapse;margin:18px 0 30px}th,td{border:1px solid #aeb8bc;padding:8

Context only
096_financials_status_report.md12_financialsContext · contextMD6.4 KB

# Status Report — CB-DOC-001-096 > Confidential — matter team · reviewed — no independent exception · native version 1.6 | Control field | Value | |---|---| | Matter | CB-MA-2401 — Project Daybreak battery acqu

Context only

Runnable world

Environment and tool contract

The published Harbor task is executable, not a folder of questions. The agent enters an isolated container, calls the pinned MCP server, writes deliverables, and receives a reproducible score.

Task + 96 filesImmutable seed
Filesystem MCPJSON-RPC 2.0
Agent harness109+ calls
Verifier182 criteria
Score + traceReplayable
Official upstream pin@modelcontextprotocol/server-filesystem@2026.7.10
MCP protocol2025-06-18
Reviewed commit9a96ea6e5913
read_text_file

Read the complete contents of a file from the file system as text. Handles various text encodings and provides detailed error messages if the file cannot be read. Use this tool when you need to examine the contents of a single file. Use the 'head' parameter to read only the first N lines of a file, or the 'tail' parameter to read only the last N lines of a file. Operates on the file as text regardless of extension. Only works within allowed directories.

{
  "properties": {
    "head": {
      "description": "If provided, returns only the first N lines of the file",
      "type": "number"
    },
    "path": {
      "type": "string"
    },
    "tail": {
      "description": "If provided, returns only the last N lines of the file",
      "type": "number"
    }
  },
  "required": [
    "path"
  ],
  "type": "object"
}
write_file

Create a new file or completely overwrite an existing file with new content. Use with caution as it will overwrite existing files without warning. Handles text content with proper encoding. Only works within allowed directories.

{
  "properties": {
    "content": {
      "type": "string"
    },
    "path": {
      "type": "string"
    }
  },
  "required": [
    "path",
    "content"
  ],
  "type": "object"
}
directory_tree

Get a recursive tree view of files and directories as a JSON structure. Each entry includes 'name', 'type' (file/directory), and 'children' for directories. Files have no children array, while directories always have a children array (which may be empty). The output is formatted with 2-space indentation for readability. Only works within allowed directories.

{
  "properties": {
    "excludePatterns": {
      "default": [],
      "items": {
        "type": "string"
      },
      "type": "array"
    },
    "path": {
      "type": "string"
    }
  },
  "required": [
    "path"
  ],
  "type": "object"
}
search_files

Recursively search for files and directories matching a pattern. The patterns should be glob-style patterns that match paths relative to the working directory. Use pattern like '*.ext' to match files in current directory, and '**/*.ext' to match files in all subdirectories. Returns full paths to all matching items. Great for finding files when you don't know their exact location. Only searches within allowed directories.

{
  "properties": {
    "excludePatterns": {
      "default": [],
      "items": {
        "type": "string"
      },
      "type": "array"
    },
    "path": {
      "type": "string"
    },
    "pattern": {
      "type": "string"
    }
  },
  "required": [
    "path",
    "pattern"
  ],
  "type": "object"
}
get_file_info

Retrieve detailed metadata about a file or directory. Returns comprehensive information including size, creation time, last modified time, permissions, and type. This tool is perfect for understanding file characteristics without reading the actual content. Only works within allowed directories.

{
  "properties": {
    "path": {
      "type": "string"
    }
  },
  "required": [
    "path"
  ],
  "type": "object"
}
list_allowed_directories

Returns the list of directories that this server is allowed to access. Subdirectories within these allowed directories are also accessible. Use this to understand which directories and their nested paths are available before trying to access files.

{
  "properties": {},
  "type": "object"
}

Archipelago comparison

Yes—we have the same core loop, with a different grading choice.

Archipelago packages environment, agents, snapshots, grading, and trajectory output as three services. CounselBench composes Harbor’s runner with Blobfish’s MCP world and verifier. The biggest intentional difference is grading: CounselBench is deterministic; APEX’s published methodology uses criterion-level LM judges.

Inspect Archipelago source
LayerCounselBench / BlobfishArchipelago
World lifecyclePer-task Docker Compose world, immutable seed documents, isolated output volumeContainerized environment service populated from a world snapshot
MCP boundaryPinned Streamable HTTP filesystem MCP with an offline contract mirrorHot-swappable MCP gateway configured through /apps
Agent runnerHarbor adapters; published runs use the Codex adapterRegistry of Loop, ReAct, and other agent implementations
State evidenceAppend-only MCP trace plus final filesystem deliverablesInitial/final snapshots plus artifact diffs
Grading182 deterministic criteria; no model, network, clock, locale, or randomnessComposable verifier registry; APEX commonly uses criterion-level LM judges
TrajectoryHarbor result, model transcript, tool calls, token/cost data, verifier reporttrajectory.json, final snapshot, grades.json, prompts and verifier config

Replay the work

Sample trajectory

Inspect the actual agent messages, arguments, tool results, and verifier outcome. Document bodies are summarized in the browser so the trace stays readable; the full transcript remains in the dataset.

Agentgpt-5.6-solcodex 0.149.1
Criteria score99.3%Strict criteria incomplete
Evidence96/96121 tool calls
Run cost$1.0215,715 output tokens
127 events
Full model transcript Deterministic verifier report
  1. Agent

    I’ll follow the production protocol exactly: inventory the permitted root and full tree, identify all email records, read each file individually with file-level provenance, capture the eight custody samples, reconcile cross-folder conflicts, then write only the two required deliverables.

  2. 1
    Orientlist_allowed_directoriesok
    {}

    Allowed directories: /workspace/documents /workspace/output

  3. 2
    Orientdirectory_treeok
    {
  "path": "/workspace/documents"
}

    96 files across 12 folders

  4. 3
    Orientsearch_filesok
    {
  "path": "/workspace/documents",
  "pattern": "**/*.eml"
}

    12 matching files

  5. Agent

    The allowed roots are exactly `/workspace/documents` and `/workspace/output`. The inventory confirms 96 records—eight in each of 12 folders—and the email search returned 12 `.eml` files. I’m now moving through the production record by record and preserving the control metadata verbatim.

  6. 4
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/001_corporate_control_register.md"
}

    Read 6,610 bytes

  7. 5
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/002_corporate_executed_instrument.txt"
}

    Read 6,027 bytes

  8. 6
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/003_corporate_correspondence.eml"
}

    Read 6,295 bytes

  9. 7
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/004_corporate_ledger_export.csv"
}

    Read 12,561 bytes

  10. 8
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/005_corporate_review_memorandum.json"
}

    Read 12,985 bytes

  11. 9
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/006_corporate_formal_notice.xml"
}

    Read 7,200 bytes

  12. 10
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/007_corporate_officer_certificate.html"
}

    Read 7,435 bytes

  13. 11
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/008_corporate_status_report.md"
}

    Read 6,589 bytes

  14. 12
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/001_corporate_control_register.md"
}

    Read 6,610 bytes

  15. 13
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/002_corporate_executed_instrument.txt"
}

    Read 6,027 bytes

  16. 14
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/003_corporate_correspondence.eml"
}

    Read 6,295 bytes

  17. 15
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/004_corporate_ledger_export.csv"
}

    Read 12,561 bytes

  18. 16
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/005_corporate_review_memorandum.json"
}

    Read 12,985 bytes

  19. 17
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/006_corporate_formal_notice.xml"
}

    Read 7,200 bytes

  20. 18
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/007_corporate_officer_certificate.html"
}

    Read 7,435 bytes

  21. 19
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/01_corporate/008_corporate_status_report.md"
}

    Read 6,589 bytes

  22. 20
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/02_capitalization/009_capitalization_control_register.md"
}

    Read 6,560 bytes

  23. 21
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/02_capitalization/010_capitalization_executed_instrument.txt"
}

    Read 6,057 bytes

  24. 22
    Reviewread_text_fileok
    {
  "path": "/workspace/documents/02_capitalization/011_capitalization_correspondence.eml"
}

    Read 6,459 bytes

Why the score changed

Deterministic without being brittle

v1.0 used full JSON equality and an all-or-nothing memo anchor gate. That made a complete 113-call run indistinguishable from an empty submission. v1.1 keeps strict pass, but exposes every recoverable criterion.

v1.0 audit signal

One mismatch → 0

  • Full-output string equality
  • Arbitrary owner values hidden from the record
  • No diagnostic reward for correct evidence work
v1.1 benchmark metric

182 visible criteria

  • Recoverable control metadata
  • Field-level, source-bounded fact checks
  • Weighted partial score + strict full pass

Run it yourself

The benchmark is public. The world is executable.

Download the evidence, replay the oracle, submit an agent, and inspect every scored criterion.

Open Harbor world ↗Download dataset ↗