Typed Decision Bench · tasks · Workflow control

`unfair-tos-publish-gate`

confidence gated actions · consumer legal · 198 items · primary question type noul. Source coastalcph/lex_glue (CC-BY-4.0); labels are expert labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
Jev	87.1	84.3%	198 / 198
System One Scorer	83.9	82.3%	198 / 198
OpenJev	83.9	81.8%	198 / 198
decider-2b	83.3	75.8%	198 / 198
JevFish	79.7	67.2%	198 / 198
NanoJev	76.0	62.6%	198 / 198

Sample 1 of 6 · workflow-control:tosgate-018f93aa6d06 · 30 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "sentence": "if this specific paragraph is held unenforceable , then the entirety of this `` dispute resolution '' section will be deemed void ."
}

Question potentially_unfair · noul · primary (ranked)

Read `sentence`, one sentence from an online platform's Terms of Service (text is lower-cased and tokenised). Answer yes if the sentence is a potentially unfair contractual term for consumers under EU consumer law: it limits the provider's liability, lets the provider terminate the contract unilaterally, lets the provider change the terms unilaterally, lets the provider remove user content at its discretion, binds the user merely by use of the service, imposes a choice of governing law, imposes a jurisdiction for disputes, or imposes arbitration. Answer no for any other sentence, including neutral descriptions of the service and sentences about the user's own obligations.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	correct	65.0%	0.877	`yes` 35.0% `no` 65.0%
System One Scorer	`no`	correct	52.6%	0.775	`yes` 47.4% `no` 52.6%
OpenJev	`no`	correct	97.5%	0.999	`yes` 2.5% `no` 97.5%
decider-2b	`no`	correct	53.6%	0.785	`yes` 46.4% `no` 53.6%
JevFish	`no`	correct	57.4%	0.818	`yes` 42.6% `no` 57.4%
NanoJev	`no`	correct	65.0%	0.878	`yes` 34.9% `no` 65.0%

Sample 2 of 6 · workflow-control:tosgate-123eac3c8c49 · 49 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "sentence": "we may change our seller fees from time to time by posting the changes on the ebay site 14 days in advance , but with no advance notice required for temporary promotions or any changes that result in the reduction of fees ."
}

Question potentially_unfair · noul · primary (ranked)

Read `sentence`, one sentence from an online platform's Terms of Service (text is lower-cased and tokenised). Answer yes if the sentence is a potentially unfair contractual term for consumers under EU consumer law: it limits the provider's liability, lets the provider terminate the contract unilaterally, lets the provider change the terms unilaterally, lets the provider remove user content at its discretion, binds the user merely by use of the service, imposes a choice of governing law, imposes a jurisdiction for disputes, or imposes arbitration. Answer no for any other sentence, including neutral descriptions of the service and sentences about the user's own obligations.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	88.0%	0.986	`yes` 88.0% `no` 12.0%
System One Scorer	`yes`	correct	69.9%	0.910	`yes` 69.9% `no` 30.1%
OpenJev	`yes`	correct	99.4%	1.000	`yes` 99.4% `no` 0.6%
decider-2b	`no`	wrong	23.1%	0.409	`yes` 23.1% `no` 76.9%
JevFish	`no`	wrong	18.6%	0.338	`yes` 18.6% `no` 81.3%
NanoJev	`no`	wrong	46.5%	0.714	`yes` 46.5% `no` 53.5%

Sample 3 of 6 · workflow-control:tosgate-710c7d661bd1 · 20 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "sentence": "please review our commercial terms for information about additional contract terms relating to such purchases ."
}

Question potentially_unfair · noul · primary (ranked)

Read `sentence`, one sentence from an online platform's Terms of Service (text is lower-cased and tokenised). Answer yes if the sentence is a potentially unfair contractual term for consumers under EU consumer law: it limits the provider's liability, lets the provider terminate the contract unilaterally, lets the provider change the terms unilaterally, lets the provider remove user content at its discretion, binds the user merely by use of the service, imposes a choice of governing law, imposes a jurisdiction for disputes, or imposes arbitration. Answer no for any other sentence, including neutral descriptions of the service and sentences about the user's own obligations.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	correct	94.0%	0.996	`yes` 6.0% `no` 94.0%
System One Scorer	`no`	correct	65.7%	0.882	`yes` 34.3% `no` 65.7%
OpenJev	`no`	correct	100.0%	1.000	`yes` 0.1% `no` 100.0%
decider-2b	`no`	correct	98.2%	1.000	`yes` 1.8% `no` 98.2%
JevFish	`no`	correct	81.9%	0.967	`yes` 18.1% `no` 81.9%
NanoJev	`no`	correct	55.9%	0.806	`yes` 44.1% `no` 55.9%

Sample 4 of 6 · workflow-control:tosgate-af8e0d59f4ed · 76 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "sentence": "in the event that duolingo suspends or terminates your use of the service or these terms and conditions or you close your account voluntarily , you understand and agree that you will receive no refund or exchange of any kind , including for any unused virtual currency or other virtual item , any content or data associated with your use of the service , or for anything else ."
}

Question potentially_unfair · noul · primary (ranked)

Read `sentence`, one sentence from an online platform's Terms of Service (text is lower-cased and tokenised). Answer yes if the sentence is a potentially unfair contractual term for consumers under EU consumer law: it limits the provider's liability, lets the provider terminate the contract unilaterally, lets the provider change the terms unilaterally, lets the provider remove user content at its discretion, binds the user merely by use of the service, imposes a choice of governing law, imposes a jurisdiction for disputes, or imposes arbitration. Answer no for any other sentence, including neutral descriptions of the service and sentences about the user's own obligations.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	88.0%	0.986	`yes` 88.0% `no` 12.0%
System One Scorer	`yes`	correct	61.4%	0.851	`yes` 61.4% `no` 38.6%
OpenJev	`yes`	correct	99.0%	1.000	`yes` 99.0% `no` 1.0%
decider-2b	`yes`	correct	57.2%	0.817	`yes` 57.2% `no` 42.8%
JevFish	`no`	wrong	50.0%	0.750	`yes` 50.0% `no` 50.0%
NanoJev	`yes`	correct	59.1%	0.833	`yes` 59.1% `no` 40.9%

Sample 5 of 6 · workflow-control:tosgate-c34f00fee30d · 103 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "sentence": "you also agree not to : circumvent , remove , alter , deactivate , degrade or thwart any of the content protections in the netflix service ; use any robot , spider , scraper or other automated means to access the netflix service ; decompile , reverse engineer or disassemble any software or other products or processes accessible through the netflix service ; insert any code or product or manipulate the content of the netflix service in any way ; or use any data mining , data gathering or extraction method ."
}

Question potentially_unfair · noul · primary (ranked)

Read `sentence`, one sentence from an online platform's Terms of Service (text is lower-cased and tokenised). Answer yes if the sentence is a potentially unfair contractual term for consumers under EU consumer law: it limits the provider's liability, lets the provider terminate the contract unilaterally, lets the provider change the terms unilaterally, lets the provider remove user content at its discretion, binds the user merely by use of the service, imposes a choice of governing law, imposes a jurisdiction for disputes, or imposes arbitration. Answer no for any other sentence, including neutral descriptions of the service and sentences about the user's own obligations.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	correct	94.0%	0.996	`yes` 6.0% `no` 94.0%
System One Scorer	`no`	correct	77.3%	0.948	`yes` 22.7% `no` 77.3%
OpenJev	`no`	correct	100.0%	1.000	`yes` 0.0% `no` 100.0%
decider-2b	`no`	correct	85.0%	0.977	`yes` 15.0% `no` 85.0%
JevFish	`no`	correct	55.5%	0.802	`yes` 44.5% `no` 55.5%
NanoJev	`no`	correct	54.5%	0.793	`yes` 45.5% `no` 54.5%

Sample 6 of 6 · workflow-control:tosgate-f3dd463d69ac · 47 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "sentence": "by continuing to access or use the site or services after we have posted a modification on the site or have provided you with notice of a modification , you are indicating that you agree to be bound by the modified terms ."
}

Question potentially_unfair · noul · primary (ranked)

Read `sentence`, one sentence from an online platform's Terms of Service (text is lower-cased and tokenised). Answer yes if the sentence is a potentially unfair contractual term for consumers under EU consumer law: it limits the provider's liability, lets the provider terminate the contract unilaterally, lets the provider change the terms unilaterally, lets the provider remove user content at its discretion, binds the user merely by use of the service, imposes a choice of governing law, imposes a jurisdiction for disputes, or imposes arbitration. Answer no for any other sentence, including neutral descriptions of the service and sentences about the user's own obligations.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	94.0%	0.996	`yes` 94.0% `no` 6.0%
System One Scorer	`yes`	correct	63.8%	0.869	`yes` 63.8% `no` 36.2%
OpenJev	`yes`	correct	99.9%	1.000	`yes` 99.9% `no` 0.1%
decider-2b	`no`	wrong	28.6%	0.491	`yes` 28.6% `no` 71.4%
JevFish	`no`	wrong	40.8%	0.650	`yes` 40.8% `no` 59.2%
NanoJev	`no`	wrong	48.5%	0.735	`yes` 48.5% `no` 51.5%