Typed Decision Bench · tasks · Safety and quality

`unfair-tos-clause-type`

compliance verification · consumer legal · 200 items · primary question type choice. Source nguha/legalbench (CC-BY-4.0); labels are expert labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
JevFish	87.3	83.5%	200 / 200
Jev	85.9	82.0%	200 / 200
OpenJev	83.9	80.5%	200 / 200
decider-2b	83.5	80.0%	200 / 200
System One Scorer	79.1	78.0%	200 / 200
NanoJev	60.8	25.5%	200 / 200

Sample 1 of 6 · safety-and-quality:tos-14 · 55 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "clause": "by continuing to access or use the site or services after we have posted a modification on the site or have provided you with notice of a modification , you are indicating that you agree to be bound by the modified terms .",
  "document_type": "consumer terms of service"
}

Question unfair_term_type · choice · primary (ranked)

The state holds one sentence from a consumer terms-of-service document. Which one category of potentially unfair contractual term (EU consumer-protection law) does the clause fall into? Pick other when it is none of the eight.

9 options

arbitration — Requires or allows disputes to be settled by arbitration rather than in court.
unilateral_change — Lets the provider change the terms or the service at will, without the user's consent.
content_removal — Lets the provider remove the user's content or account at its discretion.
jurisdiction — Fixes the courts or place where disputes must be brought.
choice_of_law — Fixes which country's or state's law governs the contract.
limitation_of_liability — Excludes or limits the provider's liability for damages or losses.
unilateral_termination — Lets the provider terminate the contract or account at will.
contract_by_using — States that using the service counts as accepting the terms.
other — None of the eight unfair-term categories.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: contract_by_using

System	Top pick	Grade	P(expected)	Proper score	Distribution
JevFish	`contract_by_using`	correct	79.3%	0.963	`contract_by_using` 79.3% `unilateral_change` 17.3% `other` 3.3% `unilateral_termination` 0.1% `jurisdiction` 0.1% `limitation_of_liability` 0.0%
Jev	`unilateral_change`	wrong	33.0%	0.551	`unilateral_change` 67.0% `contract_by_using` 33.0% `choice_of_law` 0.0% `jurisdiction` 0.0% `content_removal` 0.0% `unilateral_termination` 0.0%
OpenJev	`unilateral_change`	wrong	18.5%	0.343	`unilateral_change` 80.5% `contract_by_using` 18.5% `other` 0.5% `unilateral_termination` 0.2% `arbitration` 0.1% `limitation_of_liability` 0.1%
decider-2b	`contract_by_using`	correct	91.5%	0.995	`contract_by_using` 91.5% `unilateral_change` 4.9% `other` 2.7% `unilateral_termination` 0.2% `jurisdiction` 0.2% `content_removal` 0.2%
System One Scorer	`contract_by_using`	correct	70.6%	0.944	`contract_by_using` 70.6% `other` 12.6% `unilateral_change` 9.1% `unilateral_termination` 2.0% `arbitration` 1.6% `jurisdiction` 1.3%
NanoJev	`contract_by_using`	correct	32.4%	0.737	`contract_by_using` 32.4% `other` 15.8% `choice_of_law` 11.2% `unilateral_change` 10.5% `limitation_of_liability` 8.8% `unilateral_termination` 7.0%

Sample 2 of 6 · safety-and-quality:tos-2204 · 42 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "clause": "crowdtangle may change these terms of service , as described above , notwithstanding any provision to the contrary in any agreement between you and crowdtangle .",
  "document_type": "consumer terms of service"
}

Question unfair_term_type · choice · primary (ranked)

9 options

arbitration — Requires or allows disputes to be settled by arbitration rather than in court.
unilateral_change — Lets the provider change the terms or the service at will, without the user's consent.
content_removal — Lets the provider remove the user's content or account at its discretion.
jurisdiction — Fixes the courts or place where disputes must be brought.
choice_of_law — Fixes which country's or state's law governs the contract.
limitation_of_liability — Excludes or limits the provider's liability for damages or losses.
unilateral_termination — Lets the provider terminate the contract or account at will.
contract_by_using — States that using the service counts as accepting the terms.
other — None of the eight unfair-term categories.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: unilateral_change

System	Top pick	Grade	P(expected)	Proper score	Distribution
JevFish	`unilateral_change`	correct	98.2%	1.000	`unilateral_change` 98.2% `other` 1.1% `contract_by_using` 0.4% `unilateral_termination` 0.2% `limitation_of_liability` 0.1% `content_removal` 0.0%
Jev	`unilateral_change`	correct	100.0%	1.000	`unilateral_change` 100.0% `arbitration` 0.0% `content_removal` 0.0% `unilateral_termination` 0.0% `other` 0.0% `contract_by_using` 0.0%
OpenJev	`unilateral_change`	correct	100.0%	1.000	`unilateral_change` 100.0% `limitation_of_liability` 0.0% `arbitration` 0.0% `contract_by_using` 0.0% `other` 0.0% `choice_of_law` 0.0%
decider-2b	`unilateral_change`	correct	97.0%	0.999	`unilateral_change` 97.0% `other` 1.3% `unilateral_termination` 0.5% `contract_by_using` 0.3% `limitation_of_liability` 0.3% `content_removal` 0.1%
System One Scorer	`unilateral_change`	correct	63.8%	0.919	`unilateral_change` 63.8% `other` 14.7% `contract_by_using` 6.6% `unilateral_termination` 4.7% `choice_of_law` 2.7% `arbitration` 2.1%
NanoJev	`unilateral_change`	correct	21.4%	0.646	`unilateral_change` 21.4% `contract_by_using` 16.7% `other` 13.5% `unilateral_termination` 11.2% `choice_of_law` 10.5% `content_removal` 8.7%

Sample 3 of 6 · safety-and-quality:tos-2708 · 34 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "clause": "headspace will enforce its trademark rights to the fullest extent of the law , including the seeking of criminal prosecution .",
  "document_type": "consumer terms of service"
}

Question unfair_term_type · choice · primary (ranked)

9 options

arbitration — Requires or allows disputes to be settled by arbitration rather than in court.
unilateral_change — Lets the provider change the terms or the service at will, without the user's consent.
content_removal — Lets the provider remove the user's content or account at its discretion.
jurisdiction — Fixes the courts or place where disputes must be brought.
choice_of_law — Fixes which country's or state's law governs the contract.
limitation_of_liability — Excludes or limits the provider's liability for damages or losses.
unilateral_termination — Lets the provider terminate the contract or account at will.
contract_by_using — States that using the service counts as accepting the terms.
other — None of the eight unfair-term categories.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: other

System	Top pick	Grade	P(expected)	Proper score	Distribution
JevFish	`other`	correct	74.4%	0.957	`other` 74.4% `limitation_of_liability` 13.4% `jurisdiction` 4.5% `contract_by_using` 2.0% `choice_of_law` 1.8% `content_removal` 1.6%
Jev	`other`	correct	100.0%	1.000	`other` 100.0% `content_removal` 0.0% `choice_of_law` 0.0% `limitation_of_liability` 0.0% `contract_by_using` 0.0% `jurisdiction` 0.0%
OpenJev	`other`	correct	99.9%	1.000	`other` 99.9% `arbitration` 0.0% `unilateral_change` 0.0% `limitation_of_liability` 0.0% `contract_by_using` 0.0% `jurisdiction` 0.0%
decider-2b	`other`	correct	43.8%	0.783	`other` 43.8% `limitation_of_liability` 32.8% `choice_of_law` 7.0% `unilateral_termination` 5.3% `jurisdiction` 3.8% `content_removal` 2.6%
System One Scorer	`other`	correct	41.7%	0.803	`other` 41.7% `contract_by_using` 15.7% `unilateral_termination` 9.3% `jurisdiction` 7.4% `choice_of_law` 6.9% `arbitration` 6.7%
NanoJev	`contract_by_using`	wrong	15.6%	0.588	`contract_by_using` 22.8% `other` 15.6% `choice_of_law` 13.6% `unilateral_termination` 10.8% `limitation_of_liability` 8.7% `content_removal` 8.3%

Sample 4 of 6 · safety-and-quality:tos-373 · 52 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "clause": "a download speed of at least 25.0 mbps per stream is recommended to receive ultra hd ( defined as a resolution of 1080p or higher ) and hdr content .",
  "document_type": "consumer terms of service"
}

Question unfair_term_type · choice · primary (ranked)

9 options

arbitration — Requires or allows disputes to be settled by arbitration rather than in court.
unilateral_change — Lets the provider change the terms or the service at will, without the user's consent.
content_removal — Lets the provider remove the user's content or account at its discretion.
jurisdiction — Fixes the courts or place where disputes must be brought.
choice_of_law — Fixes which country's or state's law governs the contract.
limitation_of_liability — Excludes or limits the provider's liability for damages or losses.
unilateral_termination — Lets the provider terminate the contract or account at will.
contract_by_using — States that using the service counts as accepting the terms.
other — None of the eight unfair-term categories.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: other

System	Top pick	Grade	P(expected)	Proper score	Distribution
JevFish	`other`	correct	95.3%	0.998	`other` 95.3% `contract_by_using` 3.3% `limitation_of_liability` 1.0% `unilateral_change` 0.1% `unilateral_termination` 0.1% `jurisdiction` 0.1%
Jev	`other`	correct	100.0%	1.000	`other` 100.0% `jurisdiction` 0.0% `arbitration` 0.0% `unilateral_change` 0.0% `content_removal` 0.0% `contract_by_using` 0.0%
OpenJev	`other`	correct	100.0%	1.000	`other` 100.0% `unilateral_change` 0.0% `arbitration` 0.0% `limitation_of_liability` 0.0% `choice_of_law` 0.0% `contract_by_using` 0.0%
decider-2b	`other`	correct	89.8%	0.993	`other` 89.8% `contract_by_using` 4.0% `limitation_of_liability` 3.6% `choice_of_law` 0.7% `unilateral_change` 0.6% `unilateral_termination` 0.5%
System One Scorer	`other`	correct	63.2%	0.921	`other` 63.2% `contract_by_using` 11.4% `arbitration` 5.6% `jurisdiction` 4.4% `limitation_of_liability` 4.0% `choice_of_law` 3.9%
NanoJev	`contract_by_using`	wrong	12.9%	0.565	`contract_by_using` 20.7% `choice_of_law` 13.1% `other` 12.9% `unilateral_termination` 11.4% `content_removal` 10.0% `limitation_of_liability` 9.7%

Sample 5 of 6 · safety-and-quality:tos-613 · 43 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "clause": "nothing in the terms gives you a right to use the twitter name or any of the twitter trademarks , logos , domain names , and other distinctive brand features .",
  "document_type": "consumer terms of service"
}

Question unfair_term_type · choice · primary (ranked)

9 options

arbitration — Requires or allows disputes to be settled by arbitration rather than in court.
unilateral_change — Lets the provider change the terms or the service at will, without the user's consent.
content_removal — Lets the provider remove the user's content or account at its discretion.
jurisdiction — Fixes the courts or place where disputes must be brought.
choice_of_law — Fixes which country's or state's law governs the contract.
limitation_of_liability — Excludes or limits the provider's liability for damages or losses.
unilateral_termination — Lets the provider terminate the contract or account at will.
contract_by_using — States that using the service counts as accepting the terms.
other — None of the eight unfair-term categories.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: other

System	Top pick	Grade	P(expected)	Proper score	Distribution
JevFish	`other`	correct	78.6%	0.968	`other` 78.6% `contract_by_using` 10.5% `limitation_of_liability` 8.8% `unilateral_termination` 1.1% `unilateral_change` 0.3% `jurisdiction` 0.3%
Jev	`other`	correct	100.0%	1.000	`other` 100.0% `content_removal` 0.0% `unilateral_change` 0.0% `jurisdiction` 0.0% `arbitration` 0.0% `unilateral_termination` 0.0%
OpenJev	`other`	correct	99.9%	1.000	`other` 99.9% `limitation_of_liability` 0.0% `arbitration` 0.0% `unilateral_change` 0.0% `contract_by_using` 0.0% `unilateral_termination` 0.0%
decider-2b	`other`	correct	43.9%	0.777	`other` 43.9% `limitation_of_liability` 32.9% `contract_by_using` 14.5% `unilateral_termination` 3.0% `choice_of_law` 1.5% `unilateral_change` 1.4%
System One Scorer	`other`	correct	49.0%	0.833	`other` 49.0% `contract_by_using` 25.3% `unilateral_termination` 5.4% `arbitration` 4.9% `limitation_of_liability` 4.3% `jurisdiction` 3.7%
NanoJev	`other`	correct	29.8%	0.717	`other` 29.8% `contract_by_using` 16.7% `unilateral_termination` 11.5% `content_removal` 9.5% `choice_of_law` 8.8% `limitation_of_liability` 7.9%

Sample 6 of 6 · safety-and-quality:tos-618 · 108 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "clause": "we may suspend or terminate your account or cease providing you with all or part of the services at any time for any or no reason , including , but not limited to , if we reasonably believe : ( i ) you have violated these terms or the twitter rules , ( ii ) you create risk or possible legal exposure for us ; ( iii ) your account should be removed due to prolonged inactivity ; or ( iv ) our provision of the services to you is no longer commercially viable .",
  "document_type": "consumer terms of service"
}

Question unfair_term_type · choice · primary (ranked)

9 options

arbitration — Requires or allows disputes to be settled by arbitration rather than in court.
unilateral_change — Lets the provider change the terms or the service at will, without the user's consent.
content_removal — Lets the provider remove the user's content or account at its discretion.
jurisdiction — Fixes the courts or place where disputes must be brought.
choice_of_law — Fixes which country's or state's law governs the contract.
limitation_of_liability — Excludes or limits the provider's liability for damages or losses.
unilateral_termination — Lets the provider terminate the contract or account at will.
contract_by_using — States that using the service counts as accepting the terms.
other — None of the eight unfair-term categories.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: unilateral_termination

System	Top pick	Grade	P(expected)	Proper score	Distribution
JevFish	`unilateral_termination`	correct	88.3%	0.990	`unilateral_termination` 88.3% `content_removal` 7.5% `other` 2.5% `unilateral_change` 1.5% `limitation_of_liability` 0.1% `jurisdiction` 0.1%
Jev	`unilateral_termination`	correct	100.0%	1.000	`unilateral_termination` 100.0% `arbitration` 0.0% `other` 0.0% `limitation_of_liability` 0.0% `unilateral_change` 0.0% `jurisdiction` 0.0%
OpenJev	`unilateral_termination`	correct	99.8%	1.000	`unilateral_termination` 99.8% `unilateral_change` 0.2% `content_removal` 0.0% `limitation_of_liability` 0.0% `other` 0.0% `arbitration` 0.0%
decider-2b	`unilateral_termination`	correct	80.5%	0.973	`unilateral_termination` 80.5% `content_removal` 11.8% `unilateral_change` 5.0% `other` 1.9% `limitation_of_liability` 0.4% `jurisdiction` 0.2%
System One Scorer	`unilateral_termination`	correct	83.8%	0.984	`unilateral_termination` 83.8% `other` 4.2% `unilateral_change` 3.8% `content_removal` 3.5% `contract_by_using` 2.2% `limitation_of_liability` 0.8%
NanoJev	`unilateral_termination`	correct	29.7%	0.712	`unilateral_termination` 29.7% `other` 17.8% `content_removal` 14.6% `contract_by_using` 9.8% `limitation_of_liability` 9.1% `choice_of_law` 7.7%