Typed Decision Bench · tasks · Workflow control

`unfair-tos-publish-gate`

confidence gated actions · consumer legal · 198 items · primary question type noul. Source coastalcph/lex_glue (CC-BY-4.0); labels are expert labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
Jev	87.1	84.3%	198 / 198
System One Scorer	83.9	82.3%	198 / 198
OpenJev	83.9	81.8%	198 / 198
decider-2b	83.3	75.8%	198 / 198
JevFish	83.1	70.7%	198 / 198
NanoJev	76.0	62.6%	198 / 198

Sample 1 of 6 · workflow-control:tosgate-018f93aa6d06 · 30 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "sentence": "if this specific paragraph is held unenforceable , then the entirety of this `` dispute resolution '' section will be deemed void ."
}

Question potentially_unfair · noul · primary (ranked)

Read `sentence`, one sentence from an online platform's Terms of Service (text is lower-cased and tokenised). Answer yes if the sentence is a potentially unfair contractual term for consumers under EU consumer law: it limits the provider's liability, lets the provider terminate the contract unilaterally, lets the provider change the terms unilaterally, lets the provider remove user content at its discretion, binds the user merely by use of the service, imposes a choice of governing law, imposes a jurisdiction for disputes, or imposes arbitration. Answer no for any other sentence, including neutral descriptions of the service and sentences about the user's own obligations.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	correct	65.0%	0.877	`yes` 35.0% `no` 65.0%
System One Scorer	`no`	correct	52.6%	0.775	`yes` 47.4% `no` 52.6%
OpenJev	`no`	correct	97.5%	0.999	`yes` 2.5% `no` 97.5%
decider-2b	`no`	correct	53.6%	0.785	`yes` 46.4% `no` 53.6%
JevFish	`no`	correct	60.7%	0.846	`yes` 39.3% `no` 60.7%
NanoJev	`no`	correct	65.0%	0.878	`yes` 34.9% `no` 65.0%

Sample 2 of 6 · workflow-control:tosgate-0cd4f97afd50 · 32 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "sentence": "while we hope you remain a lifelong snapchatter , you can terminate these terms at any time and for any reason by deleting your account ."
}

Question potentially_unfair · noul · primary (ranked)

Read `sentence`, one sentence from an online platform's Terms of Service (text is lower-cased and tokenised). Answer yes if the sentence is a potentially unfair contractual term for consumers under EU consumer law: it limits the provider's liability, lets the provider terminate the contract unilaterally, lets the provider change the terms unilaterally, lets the provider remove user content at its discretion, binds the user merely by use of the service, imposes a choice of governing law, imposes a jurisdiction for disputes, or imposes arbitration. Answer no for any other sentence, including neutral descriptions of the service and sentences about the user's own obligations.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	correct	90.0%	0.990	`yes` 10.0% `no` 90.0%
System One Scorer	`yes`	wrong	47.1%	0.720	`yes` 52.9% `no` 47.1%
OpenJev	`no`	correct	97.2%	0.999	`yes` 2.8% `no` 97.2%
decider-2b	`no`	correct	58.3%	0.826	`yes` 41.7% `no` 58.3%
JevFish	`no`	correct	60.1%	0.841	`yes` 39.9% `no` 60.1%
NanoJev	`no`	correct	51.0%	0.760	`yes` 49.0% `no` 51.0%

Sample 3 of 6 · workflow-control:tosgate-68201d1ed880 · 10 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "sentence": "international buying and selling ; translation"
}

Question potentially_unfair · noul · primary (ranked)

Read `sentence`, one sentence from an online platform's Terms of Service (text is lower-cased and tokenised). Answer yes if the sentence is a potentially unfair contractual term for consumers under EU consumer law: it limits the provider's liability, lets the provider terminate the contract unilaterally, lets the provider change the terms unilaterally, lets the provider remove user content at its discretion, binds the user merely by use of the service, imposes a choice of governing law, imposes a jurisdiction for disputes, or imposes arbitration. Answer no for any other sentence, including neutral descriptions of the service and sentences about the user's own obligations.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	correct	93.0%	0.995	`yes` 7.0% `no` 93.0%
System One Scorer	`no`	correct	66.3%	0.886	`yes` 33.7% `no` 66.3%
OpenJev	`no`	correct	100.0%	1.000	`yes` 0.0% `no` 100.0%
decider-2b	`no`	correct	97.8%	1.000	`yes` 2.2% `no` 97.8%
JevFish	`no`	correct	97.3%	0.999	`yes` 2.7% `no` 97.3%
NanoJev	`no`	correct	54.6%	0.794	`yes` 45.4% `no` 54.6%

Sample 4 of 6 · workflow-control:tosgate-7f33ed1314b8 · 123 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "sentence": "the existence and content of the mediation and arbitration proceedings , including documents and briefs submitted by the parties , correspondence from and to the international chamber of commerce , correspondence from the mediator , and correspondence , orders and awards issued by the sole arbitrator , shall remain strictly confidential and shall not be disclosed to any third party without the express written consent from the other party unless : ( i ) the disclosure to the third party is reasonably required in the context of conducting the mediation or arbitration proceedings ; and ( ii ) the third party agrees unconditionally in writing to be bound by the confidentiality obligation stipulated herein ."
}

Question potentially_unfair · noul · primary (ranked)

Read `sentence`, one sentence from an online platform's Terms of Service (text is lower-cased and tokenised). Answer yes if the sentence is a potentially unfair contractual term for consumers under EU consumer law: it limits the provider's liability, lets the provider terminate the contract unilaterally, lets the provider change the terms unilaterally, lets the provider remove user content at its discretion, binds the user merely by use of the service, imposes a choice of governing law, imposes a jurisdiction for disputes, or imposes arbitration. Answer no for any other sentence, including neutral descriptions of the service and sentences about the user's own obligations.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	wrong	21.0%	0.376	`yes` 79.0% `no` 21.0%
System One Scorer	`yes`	wrong	39.1%	0.629	`yes` 60.9% `no` 39.1%
OpenJev	`no`	correct	76.1%	0.943	`yes` 23.9% `no` 76.1%
decider-2b	`no`	correct	95.4%	0.998	`yes` 4.6% `no` 95.4%
JevFish	`no`	correct	89.0%	0.988	`yes` 11.0% `no` 89.0%
NanoJev	`no`	correct	56.3%	0.809	`yes` 43.7% `no` 56.3%

Sample 5 of 6 · workflow-control:tosgate-917d685fe4c4 · 45 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "sentence": "you understand that academia.edu does not screen or inquire into the background of any users of the site or services , nor does academia.edu make any attempt to verify the statements of users of the site or services ."
}

Question potentially_unfair · noul · primary (ranked)

Read `sentence`, one sentence from an online platform's Terms of Service (text is lower-cased and tokenised). Answer yes if the sentence is a potentially unfair contractual term for consumers under EU consumer law: it limits the provider's liability, lets the provider terminate the contract unilaterally, lets the provider change the terms unilaterally, lets the provider remove user content at its discretion, binds the user merely by use of the service, imposes a choice of governing law, imposes a jurisdiction for disputes, or imposes arbitration. Answer no for any other sentence, including neutral descriptions of the service and sentences about the user's own obligations.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	correct	62.0%	0.856	`yes` 38.0% `no` 62.0%
System One Scorer	`no`	correct	64.8%	0.876	`yes` 35.2% `no` 64.8%
OpenJev	`no`	correct	100.0%	1.000	`yes` 0.0% `no` 100.0%
decider-2b	`no`	correct	71.4%	0.918	`yes` 28.6% `no` 71.4%
JevFish	`no`	correct	70.4%	0.913	`yes` 29.6% `no` 70.4%
NanoJev	`no`	correct	55.4%	0.801	`yes` 44.6% `no` 55.4%

Sample 6 of 6 · workflow-control:tosgate-9e0021b308a7 · 51 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "sentence": "you acknowledge and agree that academia.edu is not responsible or liable for : ( i ) the availability or accuracy of such websites or resources ; or ( ii ) the content , products , or services on or available from such websites or resources ."
}

Question potentially_unfair · noul · primary (ranked)

Read `sentence`, one sentence from an online platform's Terms of Service (text is lower-cased and tokenised). Answer yes if the sentence is a potentially unfair contractual term for consumers under EU consumer law: it limits the provider's liability, lets the provider terminate the contract unilaterally, lets the provider change the terms unilaterally, lets the provider remove user content at its discretion, binds the user merely by use of the service, imposes a choice of governing law, imposes a jurisdiction for disputes, or imposes arbitration. Answer no for any other sentence, including neutral descriptions of the service and sentences about the user's own obligations.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	wrong	9.0%	0.172	`yes` 91.0% `no` 9.0%
System One Scorer	`yes`	wrong	39.0%	0.628	`yes` 61.0% `no` 39.0%
OpenJev	`yes`	wrong	0.1%	0.001	`yes` 99.9% `no` 0.1%
decider-2b	`no`	correct	59.5%	0.836	`yes` 40.5% `no` 59.5%
JevFish	`no`	correct	56.1%	0.807	`yes` 43.9% `no` 56.1%
NanoJev	`no`	correct	59.3%	0.835	`yes` 40.7% `no` 59.3%