Typed Decision Bench · tasks · Safety and quality

`vitaminc-claim-consistency`

self consistency checks · media · 200 items · primary question type choice. Source tals/vitaminc (CC-BY-SA-3.0); labels are crowd labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
Jev	88.1	83.0%	200 / 200
JevFish	83.3	73.0%	200 / 200
OpenJev	82.8	77.0%	200 / 200
decider-2b	82.4	71.5%	200 / 200
System One Scorer	82.3	76.0%	200 / 200
NanoJev	60.8	50.5%	200 / 200

Sample 1 of 6 · safety-and-quality:vitaminc-19058 · 94 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "claim": "Since 2015 , the Global Financial Centres Index has been published twice a year by Z/Yen Group in London and the China Development Institute in Shenzhen .",
  "evidence": "It has been jointly published twice per year by Z/Yen Group in London and the China Development Institute in Shenzhen since 2015 ,   and is widely quoted as a source for ranking financial centres .",
  "page": "Global Financial Centres Index"
}

Question verdict · choice · primary (ranked)

Using only the evidence (one sentence from the Wikipedia article named in page), what is the verdict on the claim?

3 options

supports — The evidence establishes that the claim is true.
refutes — The evidence establishes that the claim is false.
not_enough_info — The evidence does not settle whether the claim is true or false.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: supports

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`supports`	correct	100.0%	1.000	`supports` 100.0% `refutes` 0.0% `not_enough_info` 0.0%
JevFish	`supports`	correct	93.0%	0.996	`supports` 93.0% `not_enough_info` 6.2% `refutes` 0.9%
OpenJev	`supports`	correct	100.0%	1.000	`supports` 100.0% `refutes` 0.0% `not_enough_info` 0.0%
decider-2b	`supports`	correct	95.0%	0.998	`supports` 95.0% `not_enough_info` 4.4% `refutes` 0.6%
System One Scorer	`supports`	correct	92.7%	0.996	`supports` 92.7% `not_enough_info` 4.2% `refutes` 3.0%
NanoJev	`supports`	correct	91.3%	0.994	`supports` 91.3% `refutes` 4.6% `not_enough_info` 4.0%

Question supported · noul

Using only the evidence, is the claim true?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	97.0%	0.999	`yes` 97.0% `no` 3.0%
JevFish	`yes`	correct	94.0%	0.996	`yes` 94.0% `no` 6.0%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%
decider-2b	`yes`	correct	92.7%	0.995	`yes` 92.7% `no` 7.3%
System One Scorer	`yes`	correct	86.1%	0.981	`yes` 86.1% `no` 13.9%
NanoJev	`yes`	correct	77.3%	0.949	`yes` 77.3% `no` 22.7%

Sample 2 of 6 · safety-and-quality:vitaminc-20590 · 106 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "claim": "Men in Black : International has grossed more than $ 79.9 million in the United States and Canada and a worldwide total of more than $ 253.8 million .",
  "evidence": "Men in Black : International has grossed $ 79.9 million in the United States and Canada , and $ 173.9 million in other territories , for a worldwide total of $ 253.7 million .",
  "page": "Men in Black-COLON- International"
}

Question verdict · choice · primary (ranked)

Using only the evidence (one sentence from the Wikipedia article named in page), what is the verdict on the claim?

3 options

supports — The evidence establishes that the claim is true.
refutes — The evidence establishes that the claim is false.
not_enough_info — The evidence does not settle whether the claim is true or false.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: refutes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`refutes`	correct	99.0%	1.000	`refutes` 99.0% `not_enough_info` 1.0% `supports` 0.0%
JevFish	`refutes`	correct	48.6%	0.797	`refutes` 48.6% `supports` 32.3% `not_enough_info` 19.1%
OpenJev	`refutes`	correct	99.5%	1.000	`refutes` 99.5% `supports` 0.4% `not_enough_info` 0.1%
decider-2b	`supports`	wrong	37.0%	0.670	`supports` 49.4% `refutes` 37.0% `not_enough_info` 13.5%
System One Scorer	`supports`	wrong	40.2%	0.712	`supports` 44.0% `refutes` 40.2% `not_enough_info` 15.8%
NanoJev	`supports`	wrong	12.7%	0.330	`supports` 75.0% `refutes` 12.7% `not_enough_info` 12.4%

Question supported · noul

Using only the evidence, is the claim true?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	correct	95.0%	0.998	`yes` 5.0% `no` 95.0%
JevFish	`no`	correct	58.1%	0.824	`yes` 41.9% `no` 58.1%
OpenJev	`no`	correct	100.0%	1.000	`yes` 0.0% `no` 100.0%
decider-2b	`yes`	wrong	42.8%	0.673	`yes` 57.2% `no` 42.8%
System One Scorer	`yes`	wrong	43.9%	0.685	`yes` 56.1% `no` 43.9%
NanoJev	`yes`	wrong	43.9%	0.685	`yes` 56.1% `no` 43.9%

Sample 3 of 6 · safety-and-quality:vitaminc-27304 · 132 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "claim": "Pubs and restaurants were locked in London because of coronavirus regulations .",
  "evidence": "The act received Royal assent on 25 March 2020.In England , closures to pubs , restaurants and indoor sports and leisure facilities , and restrictions on movements except for four specified purposes , were imposed on 26 March by the [ [ Health Protection ( Coronavirus , Business Closure ) ( England ) Regulations 2020.The Health Protection ( Coronavirus , Business Closure ) ( England ) Regulations 2020 UK Statutory Instruments 2020 No .",
  "page": "2020 coronavirus pandemic in the United Kingdom"
}

Question verdict · choice · primary (ranked)

Using only the evidence (one sentence from the Wikipedia article named in page), what is the verdict on the claim?

3 options

supports — The evidence establishes that the claim is true.
refutes — The evidence establishes that the claim is false.
not_enough_info — The evidence does not settle whether the claim is true or false.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: supports

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`supports`	correct	89.0%	0.988	`supports` 89.0% `not_enough_info` 11.0% `refutes` 0.0%
JevFish	`supports`	correct	77.2%	0.953	`supports` 77.2% `not_enough_info` 20.1% `refutes` 2.7%
OpenJev	`supports`	correct	99.9%	1.000	`supports` 99.9% `refutes` 0.1% `not_enough_info` 0.0%
decider-2b	`supports`	correct	88.0%	0.987	`supports` 88.0% `not_enough_info` 10.6% `refutes` 1.4%
System One Scorer	`supports`	correct	82.5%	0.976	`supports` 82.5% `not_enough_info` 11.6% `refutes` 5.9%
NanoJev	`supports`	correct	86.8%	0.987	`supports` 86.8% `not_enough_info` 7.9% `refutes` 5.3%

Question supported · noul

Using only the evidence, is the claim true?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	91.0%	0.992	`yes` 91.0% `no` 9.0%
JevFish	`yes`	correct	79.9%	0.960	`yes` 79.9% `no` 20.1%
OpenJev	`yes`	correct	99.3%	1.000	`yes` 99.3% `no` 0.8%
decider-2b	`yes`	correct	86.7%	0.982	`yes` 86.7% `no` 13.3%
System One Scorer	`yes`	correct	85.5%	0.979	`yes` 85.5% `no` 14.5%
NanoJev	`yes`	correct	59.4%	0.835	`yes` 59.4% `no` 40.6%

Sample 4 of 6 · safety-and-quality:vitaminc-31166 · 72 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "claim": "More than three Celebrity Rehab patients died .",
  "evidence": "On August 17 , 2012 , Joey Kovar was found dead at the age of 29 , making him the third Celebrity Rehab patient to die , after Mike Starr , Jeff Conaway and Rodney King.",
  "page": "Celebrity Rehab with Dr. Drew"
}

Question verdict · choice · primary (ranked)

Using only the evidence (one sentence from the Wikipedia article named in page), what is the verdict on the claim?

3 options

supports — The evidence establishes that the claim is true.
refutes — The evidence establishes that the claim is false.
not_enough_info — The evidence does not settle whether the claim is true or false.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: refutes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`supports`	wrong	32.0%	0.600	`supports` 57.0% `refutes` 32.0% `not_enough_info` 11.0%
JevFish	`supports`	wrong	1.7%	0.113	`supports` 89.5% `not_enough_info` 8.8% `refutes` 1.7%
OpenJev	`refutes`	correct	86.4%	0.982	`refutes` 86.4% `supports` 12.9% `not_enough_info` 0.7%
decider-2b	`supports`	wrong	0.5%	0.032	`supports` 97.2% `not_enough_info` 2.3% `refutes` 0.5%
System One Scorer	`refutes`	correct	40.2%	0.727	`refutes` 40.2% `supports` 36.6% `not_enough_info` 23.2%
NanoJev	`supports`	wrong	9.6%	0.264	`supports` 80.3% `not_enough_info` 10.0% `refutes` 9.6%

Question supported · noul

Using only the evidence, is the claim true?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	wrong	42.0%	0.664	`yes` 58.0% `no` 42.0%
JevFish	`yes`	wrong	13.8%	0.257	`yes` 86.2% `no` 13.8%
OpenJev	`no`	correct	84.1%	0.975	`yes` 15.9% `no` 84.1%
decider-2b	`yes`	wrong	6.3%	0.123	`yes` 93.7% `no` 6.3%
System One Scorer	`no`	correct	57.8%	0.822	`yes` 42.3% `no` 57.8%
NanoJev	`yes`	wrong	48.0%	0.730	`yes` 52.0% `no` 48.0%

Sample 5 of 6 · safety-and-quality:vitaminc-31180 · 47 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "claim": "Lake Wylie and Mountain Island Lake are close to the White House .",
  "evidence": "Lake Wylie and Mountain Island Lake are two smaller man-made lakes located near the White House.",
  "page": "Charlotte, North Carolina"
}

Question verdict · choice · primary (ranked)

Using only the evidence (one sentence from the Wikipedia article named in page), what is the verdict on the claim?

3 options

supports — The evidence establishes that the claim is true.
refutes — The evidence establishes that the claim is false.
not_enough_info — The evidence does not settle whether the claim is true or false.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: supports

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`supports`	correct	81.0%	0.972	`supports` 81.0% `refutes` 13.0% `not_enough_info` 6.0%
JevFish	`supports`	correct	92.2%	0.995	`supports` 92.2% `not_enough_info` 7.1% `refutes` 0.7%
OpenJev	`refutes`	wrong	20.1%	0.386	`refutes` 76.7% `supports` 20.1% `not_enough_info` 3.2%
decider-2b	`supports`	correct	94.7%	0.997	`supports` 94.7% `not_enough_info` 4.8% `refutes` 0.5%
System One Scorer	`supports`	correct	86.3%	0.985	`supports` 86.3% `not_enough_info` 9.4% `refutes` 4.3%
NanoJev	`supports`	correct	91.3%	0.994	`supports` 91.3% `not_enough_info` 4.5% `refutes` 4.2%

Question supported · noul

Using only the evidence, is the claim true?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	59.0%	0.832	`yes` 59.0% `no` 41.0%
JevFish	`yes`	correct	94.2%	0.997	`yes` 94.2% `no` 5.8%
OpenJev	`yes`	correct	51.4%	0.764	`yes` 51.4% `no` 48.6%
decider-2b	`yes`	correct	95.2%	0.998	`yes` 95.2% `no` 4.8%
System One Scorer	`yes`	correct	86.8%	0.983	`yes` 86.8% `no` 13.2%
NanoJev	`no`	wrong	48.8%	0.737	`yes` 48.8% `no` 51.2%

Sample 6 of 6 · safety-and-quality:vitaminc-9618 · 30 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "claim": "Draft Day has diverse reviews from critics .",
  "evidence": "Draft Day has received mixed reviews from critics .",
  "page": "Draft Day"
}

Question verdict · choice · primary (ranked)

Using only the evidence (one sentence from the Wikipedia article named in page), what is the verdict on the claim?

3 options

supports — The evidence establishes that the claim is true.
refutes — The evidence establishes that the claim is false.
not_enough_info — The evidence does not settle whether the claim is true or false.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: supports

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`supports`	correct	83.0%	0.974	`supports` 83.0% `not_enough_info` 15.0% `refutes` 2.0%
JevFish	`supports`	correct	74.5%	0.946	`supports` 74.5% `not_enough_info` 20.0% `refutes` 5.5%
OpenJev	`supports`	correct	99.9%	1.000	`supports` 99.9% `refutes` 0.1% `not_enough_info` 0.0%
decider-2b	`supports`	correct	90.7%	0.993	`supports` 90.7% `not_enough_info` 7.8% `refutes` 1.5%
System One Scorer	`supports`	correct	60.9%	0.884	`supports` 60.9% `not_enough_info` 23.8% `refutes` 15.2%
NanoJev	`supports`	correct	92.6%	0.996	`supports` 92.6% `not_enough_info` 3.9% `refutes` 3.5%

Question supported · noul

Using only the evidence, is the claim true?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	69.0%	0.904	`yes` 69.0% `no` 31.0%
JevFish	`yes`	correct	77.3%	0.948	`yes` 77.3% `no` 22.7%
OpenJev	`yes`	correct	98.6%	1.000	`yes` 98.6% `no` 1.4%
decider-2b	`yes`	correct	83.0%	0.971	`yes` 83.0% `no` 17.0%
System One Scorer	`yes`	correct	71.7%	0.920	`yes` 71.7% `no` 28.3%
NanoJev	`yes`	correct	71.6%	0.919	`yes` 71.6% `no` 28.4%