Typed Decision Bench · tasks · Safety and quality

`cuad-checklist`

compliance verification · legal · 204 items · primary question type noul. Source nguha/legalbench (CC-BY-4.0); labels are expert labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
decider-2b	95.1	95.1%	204 / 204
JevFish	93.6	93.1%	204 / 204
System One Scorer	92.1	92.7%	204 / 204
Jev	91.6	89.2%	204 / 204
OpenJev	91.2	89.2%	204 / 204
NanoJev	72.7	49.5%	204 / 204

Sample 1 of 6 · safety-and-quality:cuad-checklist-audit_rights-308 · 107 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "document_id": "TELEGLOBEINTERNATIONALHOLDINGSLTD_03_29_2004-EX-10.10-CONSTRUCTION AND …",
  "clause": "The PG shall ensure that the Supply Contract shall afford them or their designated representatives reasonable rights of access to examine, test, and inspect the APCN 2 cable equipment, material, supplies and installation activities.",
  "checklist_item": "Audit rights: lets a party inspect the other's books, records or premises to verify compliance."
}

Question clause_matches_item · noul · primary (ranked)

The state holds one clause extracted from the contract named in document_id and one due-diligence checklist_item. Does the clause contain the kind of provision the checklist item describes?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
decider-2b	`yes`	correct	69.4%	0.906	`yes` 69.4% `no` 30.6%
JevFish	`yes`	correct	66.4%	0.887	`yes` 66.4% `no` 33.6%
System One Scorer	`yes`	correct	78.9%	0.956	`yes` 78.9% `no` 21.1%
Jev	`yes`	correct	84.0%	0.974	`yes` 84.0% `no` 16.0%
OpenJev	`yes`	correct	99.9%	1.000	`yes` 99.9% `no` 0.1%
NanoJev	`yes`	correct	74.0%	0.932	`yes` 74.0% `no` 26.0%

Sample 2 of 6 · safety-and-quality:cuad-checklist-audit_rights-426 · 127 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "document_id": "NEXSTARFINANCEHOLDINGSINC_03_27_2002-EX-10.26-OUTSOURCING AGREEMENT.PDF",
  "clause": "At all times during the Term and for six (6) months following the termination of this Agreement, WYZZ shall have the right, upon prior written request to Nexstar, to review all of the books and records of Nexstar relating to the BCF Report and the Distributions.",
  "checklist_item": "Audit rights: lets a party inspect the other's books, records or premises to verify compliance."
}

Question clause_matches_item · noul · primary (ranked)

The state holds one clause extracted from the contract named in document_id and one due-diligence checklist_item. Does the clause contain the kind of provision the checklist item describes?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
decider-2b	`yes`	correct	90.5%	0.991	`yes` 90.5% `no` 9.4%
JevFish	`yes`	correct	88.2%	0.986	`yes` 88.2% `no` 11.8%
System One Scorer	`yes`	correct	85.5%	0.979	`yes` 85.5% `no` 14.5%
Jev	`yes`	correct	98.0%	1.000	`yes` 98.0% `no` 2.0%
OpenJev	`yes`	correct	99.9%	1.000	`yes` 99.9% `no` 0.1%
NanoJev	`yes`	correct	72.2%	0.923	`yes` 72.2% `no` 27.8%

Sample 3 of 6 · safety-and-quality:cuad-checklist-cap_on_liability-1192 · 126 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "document_id": "InmodeLtd_20190729_F-1A_EX-10.9_11743243_EX-10.9_Manufacturing Agreement.pdf",
  "clause": "The construction, interpretation and performance of this Agreement and all transactions under it shall be governed by the law of the State of Israel, without giving effect to choice of law rules, and both Parties consent to jurisdiction by the courts of the City of Haifa.",
  "checklist_item": "Cap on liability: limits the amount a party can recover from the other for breach, loss or damages."
}

Question clause_matches_item · noul · primary (ranked)

The state holds one clause extracted from the contract named in document_id and one due-diligence checklist_item. Does the clause contain the kind of provision the checklist item describes?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
decider-2b	`no`	correct	97.6%	0.999	`yes` 2.4% `no` 97.6%
JevFish	`no`	correct	97.3%	0.999	`yes` 2.7% `no` 97.3%
System One Scorer	`no`	correct	76.3%	0.944	`yes` 23.7% `no` 76.3%
Jev	`no`	correct	98.0%	1.000	`yes` 2.0% `no` 98.0%
OpenJev	`no`	correct	100.0%	1.000	`yes` 0.0% `no` 100.0%
NanoJev	`yes`	wrong	33.4%	0.556	`yes` 66.6% `no` 33.4%

Sample 4 of 6 · safety-and-quality:cuad-checklist-governing_law-309 · 137 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "document_id": "EmeraldHealthBioceuticalsInc_20200218_1-A_EX1A-6 MAT CTRCT_11987205_EX1A-6 MAT …",
  "clause": "This Agreement shall be governed by and construed and enforced in accordance with the laws of the State of Arizona without regard to conflict of law principles, may not be amended except by a writing signed by both parties, and shall supersede any and all prior discussions and writings between the parties concerning the subject matter.",
  "checklist_item": "Governing law: names the state or country whose law governs the interpretation of the contract."
}

Question clause_matches_item · noul · primary (ranked)

The state holds one clause extracted from the contract named in document_id and one due-diligence checklist_item. Does the clause contain the kind of provision the checklist item describes?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
decider-2b	`yes`	correct	95.4%	0.998	`yes` 95.4% `no` 4.6%
JevFish	`yes`	correct	95.0%	0.998	`yes` 95.0% `no` 5.0%
System One Scorer	`yes`	correct	87.9%	0.985	`yes` 87.9% `no` 12.1%
Jev	`yes`	correct	99.0%	1.000	`yes` 99.0% `no` 1.0%
OpenJev	`yes`	correct	99.9%	1.000	`yes` 99.9% `no` 0.1%
NanoJev	`yes`	correct	70.7%	0.914	`yes` 70.7% `no` 29.3%

Sample 5 of 6 · safety-and-quality:cuad-checklist-governing_law-490 · 109 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "document_id": "INTRICONCORP_03_10_2009-EX-10.22-Strategic Alliance Agreement.PDF",
  "clause": "The initial term of this Agreement shall be five (5) years from the date of execution and may be extended subject to satisfactory agreement on ongoing commercial terms, to be agreed two (2) months prior to the expiration of this Agreement.",
  "checklist_item": "Governing law: names the state or country whose law governs the interpretation of the contract."
}

Question clause_matches_item · noul · primary (ranked)

The state holds one clause extracted from the contract named in document_id and one due-diligence checklist_item. Does the clause contain the kind of provision the checklist item describes?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
decider-2b	`no`	correct	97.2%	0.999	`yes` 2.8% `no` 97.2%
JevFish	`no`	correct	98.5%	1.000	`yes` 1.5% `no` 98.5%
System One Scorer	`no`	correct	76.4%	0.944	`yes` 23.6% `no` 76.4%
Jev	`no`	correct	98.0%	1.000	`yes` 2.0% `no` 98.0%
OpenJev	`no`	correct	100.0%	1.000	`yes` 0.0% `no` 100.0%
NanoJev	`yes`	wrong	32.1%	0.539	`yes` 67.9% `no` 32.1%

Sample 6 of 6 · safety-and-quality:cuad-checklist-governing_law-697 · 149 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "document_id": "WARNINGMANAGEMENTSERVICESINC_12_10_1999-EX-10-ENDORSEMENT AGREEMENT.PDF",
  "clause": "Pey Dirt, or its representatives, shall, upon two weeks' written notice, have the right at all reasonable times (prior to the expiration of two (2) years after the termination of the Contract Period) to inspect and make copies of the books and records of Company insofar as they shall relate to the computation of royalties to be paid to Pey Dirt hereunder and the shipment of Endorsed Products pursuant to this Agreement.",
  "checklist_item": "Governing law: names the state or country whose law governs the interpretation of the contract."
}

Question clause_matches_item · noul · primary (ranked)

The state holds one clause extracted from the contract named in document_id and one due-diligence checklist_item. Does the clause contain the kind of provision the checklist item describes?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
decider-2b	`no`	correct	96.8%	0.999	`yes` 3.2% `no` 96.8%
JevFish	`no`	correct	95.9%	0.998	`yes` 4.1% `no` 95.9%
System One Scorer	`no`	correct	74.6%	0.936	`yes` 25.4% `no` 74.6%
Jev	`no`	correct	98.0%	1.000	`yes` 2.0% `no` 98.0%
OpenJev	`no`	correct	100.0%	1.000	`yes` 0.0% `no` 100.0%
NanoJev	`yes`	wrong	25.8%	0.450	`yes` 74.2% `no` 25.8%