Typed Decision Bench · tasks · Safety and quality

`cuad-checklist`

compliance verification · legal · 204 items · primary question type noul. Source nguha/legalbench (CC-BY-4.0); labels are expert labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
decider-2b	95.1	95.1%	204 / 204
System One Scorer	92.1	92.7%	204 / 204
Jev	91.6	89.2%	204 / 204
OpenJev	91.2	89.2%	204 / 204
JevFish	86.3	80.9%	204 / 204
NanoJev	72.7	49.5%	204 / 204

Sample 1 of 6 · safety-and-quality:cuad-checklist-anti-assignment-532 · 133 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "document_id": "REGANHOLDINGCORP_03_31_2008-EX-10-LICENSE AND HOSTING AGREEMENT.PDF",
  "clause": "This Agreement shall be binding on the parties and their respective successors in interest and assigns, but neither party shall have the power to assign this Agreement without the prior written consent of the other party. LMG may not subcontract or delegate any of its duties or obligations of performance in this Agreement to any third party without the prior written consent of TAG.",
  "checklist_item": "Anti-assignment: requires consent before a party may assign the contract or its rights to a third party."
}

Question clause_matches_item · noul · primary (ranked)

The state holds one clause extracted from the contract named in document_id and one due-diligence checklist_item. Does the clause contain the kind of provision the checklist item describes?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
decider-2b	`yes`	correct	89.7%	0.989	`yes` 89.7% `no` 10.3%
System One Scorer	`yes`	correct	87.6%	0.985	`yes` 87.6% `no` 12.3%
Jev	`yes`	correct	99.0%	1.000	`yes` 99.0% `no` 1.0%
OpenJev	`yes`	correct	99.9%	1.000	`yes` 99.9% `no` 0.1%
JevFish	`yes`	correct	71.3%	0.918	`yes` 71.3% `no` 28.7%
NanoJev	`yes`	correct	61.2%	0.850	`yes` 61.2% `no` 38.8%

Sample 2 of 6 · safety-and-quality:cuad-checklist-audit_rights-863 · 113 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "document_id": "LiquidmetalTechnologiesInc_20200205_8-K_EX-10.1_11968198_EX-10.1_Development …",
  "clause": "New Application Technologies that have been developed by cooperation of the Parties shall be jointly and equally owned by the Parties, and any royalties for the use thereof by third parties shall be shared equitably between the Parties",
  "checklist_item": "Audit rights: lets a party inspect the other's books, records or premises to verify compliance."
}

Question clause_matches_item · noul · primary (ranked)

The state holds one clause extracted from the contract named in document_id and one due-diligence checklist_item. Does the clause contain the kind of provision the checklist item describes?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
decider-2b	`no`	correct	80.9%	0.963	`yes` 19.1% `no` 80.9%
System One Scorer	`no`	correct	77.1%	0.948	`yes` 22.9% `no` 77.1%
Jev	`no`	correct	98.0%	1.000	`yes` 2.0% `no` 98.0%
OpenJev	`no`	correct	100.0%	1.000	`yes` 0.0% `no` 100.0%
JevFish	`no`	correct	83.8%	0.974	`yes` 16.2% `no` 83.8%
NanoJev	`yes`	wrong	35.6%	0.585	`yes` 64.4% `no` 35.6%

Sample 3 of 6 · safety-and-quality:cuad-checklist-cap_on_liability-732 · 103 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "document_id": "MANUFACTURERSSERVICESLTD_06_05_2000-EX-10.14-OUTSOURCING AGREEMENT.PDF",
  "clause": "This Agreement shall become effective on the Effective Date and shall continue for a period of three (3) years unless terminated as provided in Section 5.0.",
  "checklist_item": "Cap on liability: limits the amount a party can recover from the other for breach, loss or damages."
}

Question clause_matches_item · noul · primary (ranked)

The state holds one clause extracted from the contract named in document_id and one due-diligence checklist_item. Does the clause contain the kind of provision the checklist item describes?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
decider-2b	`no`	correct	86.1%	0.981	`yes` 13.9% `no` 86.1%
System One Scorer	`no`	correct	75.6%	0.941	`yes` 24.3% `no` 75.6%
Jev	`no`	correct	98.0%	1.000	`yes` 2.0% `no` 98.0%
OpenJev	`no`	correct	100.0%	1.000	`yes` 0.0% `no` 100.0%
JevFish	`no`	correct	89.6%	0.989	`yes` 10.4% `no` 89.6%
NanoJev	`yes`	wrong	25.1%	0.439	`yes` 74.9% `no` 25.1%

Sample 4 of 6 · safety-and-quality:cuad-checklist-cap_on_liability-755 · 177 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "document_id": "NEXSTARFINANCEHOLDINGSINC_03_27_2002-EX-10.26-OUTSOURCING AGREEMENT.PDF",
  "clause": "The Minimum BCF shall be Three Million Seven Hundred Thousand Dollars ($3,700,000) for calendar year 2002 and shall be increased on January 1, 2003 and on each January 1 thereafter in an amount equal to the percentage increase in the Consumer Price Index (published by the U.S. Department of Labor, Bureau of Labor Statistics, Philadelphia Regional Office - All Urban Consumers for the United States - All Items) (the \"PI\") over the prior year.",
  "checklist_item": "Cap on liability: limits the amount a party can recover from the other for breach, loss or damages."
}

Question clause_matches_item · noul · primary (ranked)

The state holds one clause extracted from the contract named in document_id and one due-diligence checklist_item. Does the clause contain the kind of provision the checklist item describes?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
decider-2b	`no`	correct	68.3%	0.900	`yes` 31.7% `no` 68.3%
System One Scorer	`no`	correct	69.7%	0.908	`yes` 30.3% `no` 69.7%
Jev	`no`	correct	93.0%	0.995	`yes` 7.0% `no` 93.0%
OpenJev	`no`	correct	100.0%	1.000	`yes` 0.0% `no` 100.0%
JevFish	`no`	correct	89.7%	0.989	`yes` 10.3% `no` 89.7%
NanoJev	`yes`	wrong	38.3%	0.619	`yes` 61.7% `no` 38.3%

Sample 5 of 6 · safety-and-quality:cuad-checklist-governing_law-168 · 98 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "document_id": "MANUFACTURERSSERVICESLTD_06_05_2000-EX-10.14-OUTSOURCING AGREEMENT.PDF",
  "clause": "This Agreement and the rights and obligations of the parties hereto shall be construed in accordance with the substantive laws of the State of New York.",
  "checklist_item": "Governing law: names the state or country whose law governs the interpretation of the contract."
}

Question clause_matches_item · noul · primary (ranked)

The state holds one clause extracted from the contract named in document_id and one due-diligence checklist_item. Does the clause contain the kind of provision the checklist item describes?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
decider-2b	`yes`	correct	94.2%	0.997	`yes` 94.2% `no` 5.8%
System One Scorer	`yes`	correct	87.9%	0.985	`yes` 87.9% `no` 12.1%
Jev	`yes`	correct	99.0%	1.000	`yes` 99.0% `no` 1.0%
OpenJev	`yes`	correct	99.9%	1.000	`yes` 99.9% `no` 0.1%
JevFish	`yes`	correct	71.4%	0.918	`yes` 71.4% `no` 28.6%
NanoJev	`yes`	correct	76.9%	0.947	`yes` 76.9% `no` 23.1%

Sample 6 of 6 · safety-and-quality:cuad-checklist-non-compete-62 · 132 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "document_id": "JOINTCORP_09_19_2014-EX-10.15-FRANCHISE AGREEMENT.PDF",
  "clause": "Upon expiration or termination of this Agreement for any reason, you agree not to engage in a competitive business for a period of two (2) years after the termination or expiration and within twenty-five (25) miles of your Franchise Premises or any other The Joint Corp. Location franchise location.",
  "checklist_item": "Non-compete: restricts a party from competing with the counterparty, or from operating in a named line of business, territory or market."
}

Question clause_matches_item · noul · primary (ranked)

The state holds one clause extracted from the contract named in document_id and one due-diligence checklist_item. Does the clause contain the kind of provision the checklist item describes?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
decider-2b	`yes`	correct	94.5%	0.997	`yes` 94.5% `no` 5.5%
System One Scorer	`yes`	correct	88.3%	0.986	`yes` 88.3% `no` 11.7%
Jev	`yes`	correct	99.0%	1.000	`yes` 99.0% `no` 1.0%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.1%
JevFish	`yes`	correct	71.3%	0.917	`yes` 71.3% `no` 28.7%
NanoJev	`yes`	correct	64.6%	0.875	`yes` 64.6% `no` 35.4%