Typed Decision Bench · tasks · Safety and quality

`unfair-tos-clause-type`

compliance verification · consumer legal · 200 items · primary question type choice. Source nguha/legalbench (CC-BY-4.0); labels are expert labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
JevFish	90.3	89.5%	200 / 200
Jev	85.9	82.0%	200 / 200
OpenJev	83.9	80.5%	200 / 200
decider-2b	83.5	80.0%	200 / 200
System One Scorer	79.1	78.0%	200 / 200
NanoJev	60.8	25.5%	200 / 200

Sample 1 of 6 · safety-and-quality:tos-14 · 55 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "clause": "by continuing to access or use the site or services after we have posted a modification on the site or have provided you with notice of a modification , you are indicating that you agree to be bound by the modified terms .",
  "document_type": "consumer terms of service"
}

Question unfair_term_type · choice · primary (ranked)

The state holds one sentence from a consumer terms-of-service document. Which one category of potentially unfair contractual term (EU consumer-protection law) does the clause fall into? Pick other when it is none of the eight.

9 options

arbitration — Requires or allows disputes to be settled by arbitration rather than in court.
unilateral_change — Lets the provider change the terms or the service at will, without the user's consent.
content_removal — Lets the provider remove the user's content or account at its discretion.
jurisdiction — Fixes the courts or place where disputes must be brought.
choice_of_law — Fixes which country's or state's law governs the contract.
limitation_of_liability — Excludes or limits the provider's liability for damages or losses.
unilateral_termination — Lets the provider terminate the contract or account at will.
contract_by_using — States that using the service counts as accepting the terms.
other — None of the eight unfair-term categories.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: contract_by_using

System	Top pick	Grade	P(expected)	Proper score	Distribution
JevFish	`contract_by_using`	correct	48.5%	0.783	`contract_by_using` 48.5% `other` 40.5% `unilateral_change` 7.4% `unilateral_termination` 1.1% `arbitration` 0.6% `content_removal` 0.6%
Jev	`unilateral_change`	wrong	33.0%	0.551	`unilateral_change` 67.0% `contract_by_using` 33.0% `choice_of_law` 0.0% `jurisdiction` 0.0% `content_removal` 0.0% `unilateral_termination` 0.0%
OpenJev	`unilateral_change`	wrong	18.5%	0.343	`unilateral_change` 80.5% `contract_by_using` 18.5% `other` 0.5% `unilateral_termination` 0.2% `arbitration` 0.1% `limitation_of_liability` 0.1%
decider-2b	`contract_by_using`	correct	91.5%	0.995	`contract_by_using` 91.5% `unilateral_change` 4.9% `other` 2.7% `unilateral_termination` 0.2% `jurisdiction` 0.2% `content_removal` 0.2%
System One Scorer	`contract_by_using`	correct	70.6%	0.944	`contract_by_using` 70.6% `other` 12.6% `unilateral_change` 9.1% `unilateral_termination` 2.0% `arbitration` 1.6% `jurisdiction` 1.3%
NanoJev	`contract_by_using`	correct	32.4%	0.737	`contract_by_using` 32.4% `other` 15.8% `choice_of_law` 11.2% `unilateral_change` 10.5% `limitation_of_liability` 8.8% `unilateral_termination` 7.0%

Sample 2 of 6 · safety-and-quality:tos-2204 · 42 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "clause": "crowdtangle may change these terms of service , as described above , notwithstanding any provision to the contrary in any agreement between you and crowdtangle .",
  "document_type": "consumer terms of service"
}

Question unfair_term_type · choice · primary (ranked)

9 options

arbitration — Requires or allows disputes to be settled by arbitration rather than in court.
unilateral_change — Lets the provider change the terms or the service at will, without the user's consent.
content_removal — Lets the provider remove the user's content or account at its discretion.
jurisdiction — Fixes the courts or place where disputes must be brought.
choice_of_law — Fixes which country's or state's law governs the contract.
limitation_of_liability — Excludes or limits the provider's liability for damages or losses.
unilateral_termination — Lets the provider terminate the contract or account at will.
contract_by_using — States that using the service counts as accepting the terms.
other — None of the eight unfair-term categories.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: unilateral_change

System	Top pick	Grade	P(expected)	Proper score	Distribution
JevFish	`unilateral_change`	correct	58.7%	0.848	`unilateral_change` 58.7% `other` 36.4% `unilateral_termination` 2.3% `arbitration` 0.8% `content_removal` 0.5% `limitation_of_liability` 0.4%
Jev	`unilateral_change`	correct	100.0%	1.000	`unilateral_change` 100.0% `arbitration` 0.0% `content_removal` 0.0% `unilateral_termination` 0.0% `other` 0.0% `contract_by_using` 0.0%
OpenJev	`unilateral_change`	correct	100.0%	1.000	`unilateral_change` 100.0% `limitation_of_liability` 0.0% `arbitration` 0.0% `contract_by_using` 0.0% `other` 0.0% `choice_of_law` 0.0%
decider-2b	`unilateral_change`	correct	97.0%	0.999	`unilateral_change` 97.0% `other` 1.3% `unilateral_termination` 0.5% `contract_by_using` 0.3% `limitation_of_liability` 0.3% `content_removal` 0.1%
System One Scorer	`unilateral_change`	correct	63.8%	0.919	`unilateral_change` 63.8% `other` 14.7% `contract_by_using` 6.6% `unilateral_termination` 4.7% `choice_of_law` 2.7% `arbitration` 2.1%
NanoJev	`unilateral_change`	correct	21.4%	0.646	`unilateral_change` 21.4% `contract_by_using` 16.7% `other` 13.5% `unilateral_termination` 11.2% `choice_of_law` 10.5% `content_removal` 8.7%

Sample 3 of 6 · safety-and-quality:tos-2708 · 34 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "clause": "headspace will enforce its trademark rights to the fullest extent of the law , including the seeking of criminal prosecution .",
  "document_type": "consumer terms of service"
}

Question unfair_term_type · choice · primary (ranked)

9 options

arbitration — Requires or allows disputes to be settled by arbitration rather than in court.
unilateral_change — Lets the provider change the terms or the service at will, without the user's consent.
content_removal — Lets the provider remove the user's content or account at its discretion.
jurisdiction — Fixes the courts or place where disputes must be brought.
choice_of_law — Fixes which country's or state's law governs the contract.
limitation_of_liability — Excludes or limits the provider's liability for damages or losses.
unilateral_termination — Lets the provider terminate the contract or account at will.
contract_by_using — States that using the service counts as accepting the terms.
other — None of the eight unfair-term categories.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: other

System	Top pick	Grade	P(expected)	Proper score	Distribution
JevFish	`other`	correct	83.5%	0.981	`other` 83.5% `unilateral_change` 10.3% `unilateral_termination` 1.8% `limitation_of_liability` 1.2% `content_removal` 1.1% `arbitration` 0.9%
Jev	`other`	correct	100.0%	1.000	`other` 100.0% `content_removal` 0.0% `choice_of_law` 0.0% `limitation_of_liability` 0.0% `contract_by_using` 0.0% `jurisdiction` 0.0%
OpenJev	`other`	correct	99.9%	1.000	`other` 99.9% `arbitration` 0.0% `unilateral_change` 0.0% `limitation_of_liability` 0.0% `contract_by_using` 0.0% `jurisdiction` 0.0%
decider-2b	`other`	correct	43.8%	0.783	`other` 43.8% `limitation_of_liability` 32.8% `choice_of_law` 7.0% `unilateral_termination` 5.3% `jurisdiction` 3.8% `content_removal` 2.6%
System One Scorer	`other`	correct	41.7%	0.803	`other` 41.7% `contract_by_using` 15.7% `unilateral_termination` 9.3% `jurisdiction` 7.4% `choice_of_law` 6.9% `arbitration` 6.7%
NanoJev	`contract_by_using`	wrong	15.6%	0.588	`contract_by_using` 22.8% `other` 15.6% `choice_of_law` 13.6% `unilateral_termination` 10.8% `limitation_of_liability` 8.7% `content_removal` 8.3%

Sample 4 of 6 · safety-and-quality:tos-3714 · 76 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "clause": "you acknowledge that your participation in the service , including your creation or uploading of content in the service , does not make you a linden lab employee and that you do not expect to be , and will not be , compensated by linden lab for such activities , and you will make no claim inconsistent with these acknowledgements .",
  "document_type": "consumer terms of service"
}

Question unfair_term_type · choice · primary (ranked)

9 options

arbitration — Requires or allows disputes to be settled by arbitration rather than in court.
unilateral_change — Lets the provider change the terms or the service at will, without the user's consent.
content_removal — Lets the provider remove the user's content or account at its discretion.
jurisdiction — Fixes the courts or place where disputes must be brought.
choice_of_law — Fixes which country's or state's law governs the contract.
limitation_of_liability — Excludes or limits the provider's liability for damages or losses.
unilateral_termination — Lets the provider terminate the contract or account at will.
contract_by_using — States that using the service counts as accepting the terms.
other — None of the eight unfair-term categories.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: other

System	Top pick	Grade	P(expected)	Proper score	Distribution
JevFish	`other`	correct	74.1%	0.948	`other` 74.1% `limitation_of_liability` 19.2% `unilateral_change` 2.1% `contract_by_using` 2.1% `unilateral_termination` 0.7% `arbitration` 0.7%
Jev	`other`	correct	82.0%	0.968	`other` 82.0% `limitation_of_liability` 18.0% `unilateral_change` 0.0% `content_removal` 0.0% `choice_of_law` 0.0% `unilateral_termination` 0.0%
OpenJev	`other`	correct	99.7%	1.000	`other` 99.7% `limitation_of_liability` 0.3% `unilateral_change` 0.0% `choice_of_law` 0.0% `arbitration` 0.0% `unilateral_termination` 0.0%
decider-2b	`limitation_of_liability`	wrong	18.4%	0.517	`limitation_of_liability` 39.7% `contract_by_using` 37.8% `other` 18.4% `choice_of_law` 1.1% `unilateral_termination` 0.8% `jurisdiction` 0.7%
System One Scorer	`other`	correct	42.3%	0.791	`other` 42.3% `contract_by_using` 26.3% `unilateral_termination` 7.2% `arbitration` 6.0% `limitation_of_liability` 5.1% `jurisdiction` 4.4%
NanoJev	`contract_by_using`	wrong	17.2%	0.568	`contract_by_using` 38.0% `other` 17.2% `limitation_of_liability` 9.4% `choice_of_law` 8.9% `unilateral_termination` 8.3% `content_removal` 7.0%

Sample 5 of 6 · safety-and-quality:tos-618 · 108 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "clause": "we may suspend or terminate your account or cease providing you with all or part of the services at any time for any or no reason , including , but not limited to , if we reasonably believe : ( i ) you have violated these terms or the twitter rules , ( ii ) you create risk or possible legal exposure for us ; ( iii ) your account should be removed due to prolonged inactivity ; or ( iv ) our provision of the services to you is no longer commercially viable .",
  "document_type": "consumer terms of service"
}

Question unfair_term_type · choice · primary (ranked)

9 options

arbitration — Requires or allows disputes to be settled by arbitration rather than in court.
unilateral_change — Lets the provider change the terms or the service at will, without the user's consent.
content_removal — Lets the provider remove the user's content or account at its discretion.
jurisdiction — Fixes the courts or place where disputes must be brought.
choice_of_law — Fixes which country's or state's law governs the contract.
limitation_of_liability — Excludes or limits the provider's liability for damages or losses.
unilateral_termination — Lets the provider terminate the contract or account at will.
contract_by_using — States that using the service counts as accepting the terms.
other — None of the eight unfair-term categories.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: unilateral_termination

System	Top pick	Grade	P(expected)	Proper score	Distribution
JevFish	`unilateral_termination`	correct	76.0%	0.959	`unilateral_termination` 76.0% `unilateral_change` 13.9% `content_removal` 7.1% `other` 1.8% `arbitration` 0.4% `limitation_of_liability` 0.3%
Jev	`unilateral_termination`	correct	100.0%	1.000	`unilateral_termination` 100.0% `arbitration` 0.0% `other` 0.0% `limitation_of_liability` 0.0% `unilateral_change` 0.0% `jurisdiction` 0.0%
OpenJev	`unilateral_termination`	correct	99.8%	1.000	`unilateral_termination` 99.8% `unilateral_change` 0.2% `content_removal` 0.0% `limitation_of_liability` 0.0% `other` 0.0% `arbitration` 0.0%
decider-2b	`unilateral_termination`	correct	80.5%	0.973	`unilateral_termination` 80.5% `content_removal` 11.8% `unilateral_change` 5.0% `other` 1.9% `limitation_of_liability` 0.4% `jurisdiction` 0.2%
System One Scorer	`unilateral_termination`	correct	83.8%	0.984	`unilateral_termination` 83.8% `other` 4.2% `unilateral_change` 3.8% `content_removal` 3.5% `contract_by_using` 2.2% `limitation_of_liability` 0.8%
NanoJev	`unilateral_termination`	correct	29.7%	0.712	`unilateral_termination` 29.7% `other` 17.8% `content_removal` 14.6% `contract_by_using` 9.8% `limitation_of_liability` 9.1% `choice_of_law` 7.7%

Sample 6 of 6 · safety-and-quality:tos-803 · 90 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "clause": "violate the intellectual property or other rights of linkedin , including , without limitation , ( i ) copying or distributing our learning videos or other materials or ( ii ) copying or distributing our technology , unless it is released under open source licenses ; ( iii ) using the word `` linkedin '' or our logos in any business name , email , or url except as provided in the brand guidelines ;",
  "document_type": "consumer terms of service"
}

Question unfair_term_type · choice · primary (ranked)

9 options

arbitration — Requires or allows disputes to be settled by arbitration rather than in court.
unilateral_change — Lets the provider change the terms or the service at will, without the user's consent.
content_removal — Lets the provider remove the user's content or account at its discretion.
jurisdiction — Fixes the courts or place where disputes must be brought.
choice_of_law — Fixes which country's or state's law governs the contract.
limitation_of_liability — Excludes or limits the provider's liability for damages or losses.
unilateral_termination — Lets the provider terminate the contract or account at will.
contract_by_using — States that using the service counts as accepting the terms.
other — None of the eight unfair-term categories.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: other

System	Top pick	Grade	P(expected)	Proper score	Distribution
JevFish	`other`	correct	72.0%	0.939	`other` 72.0% `unilateral_change` 20.6% `content_removal` 3.2% `unilateral_termination` 1.1% `limitation_of_liability` 0.9% `contract_by_using` 0.8%
Jev	`other`	correct	100.0%	1.000	`other` 100.0% `choice_of_law` 0.0% `contract_by_using` 0.0% `arbitration` 0.0% `content_removal` 0.0% `unilateral_change` 0.0%
OpenJev	`other`	correct	100.0%	1.000	`other` 100.0% `limitation_of_liability` 0.0% `unilateral_change` 0.0% `arbitration` 0.0% `content_removal` 0.0% `unilateral_termination` 0.0%
decider-2b	`other`	correct	58.1%	0.881	`other` 58.1% `contract_by_using` 22.2% `limitation_of_liability` 10.8% `choice_of_law` 2.5% `unilateral_change` 1.9% `unilateral_termination` 1.8%
System One Scorer	`other`	correct	42.2%	0.790	`other` 42.2% `contract_by_using` 26.3% `unilateral_termination` 7.3% `arbitration` 6.7% `jurisdiction` 4.4% `limitation_of_liability` 4.3%
NanoJev	`other`	correct	19.4%	0.628	`other` 19.4% `contract_by_using` 18.6% `content_removal` 11.9% `unilateral_termination` 11.2% `choice_of_law` 9.7% `unilateral_change` 9.3%