Typed Decision Bench · tasks · Safety and quality

`vitaminc-claim-consistency`

self consistency checks · media · 200 items · primary question type choice. Source tals/vitaminc (CC-BY-SA-3.0); labels are crowd labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
Jev	88.1	83.0%	200 / 200
OpenJev	82.8	77.0%	200 / 200
decider-2b	82.4	71.5%	200 / 200
System One Scorer	82.3	76.0%	200 / 200
JevFish	78.1	63.0%	200 / 200
NanoJev	60.8	50.5%	200 / 200

Sample 1 of 6 · safety-and-quality:vitaminc-20151 · 42 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "claim": "Lee Byung-hun 's place of birth is Seongnam .",
  "evidence": "Lee was born in Seongnam , South Korea .",
  "page": "Lee Byung-hun"
}

Question verdict · choice · primary (ranked)

Using only the evidence (one sentence from the Wikipedia article named in page), what is the verdict on the claim?

3 options

supports — The evidence establishes that the claim is true.
refutes — The evidence establishes that the claim is false.
not_enough_info — The evidence does not settle whether the claim is true or false.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: supports

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`supports`	correct	100.0%	1.000	`supports` 100.0% `not_enough_info` 0.0% `refutes` 0.0%
OpenJev	`supports`	correct	99.5%	1.000	`supports` 99.5% `refutes` 0.4% `not_enough_info` 0.1%
decider-2b	`supports`	correct	96.2%	0.999	`supports` 96.2% `not_enough_info` 3.5% `refutes` 0.3%
System One Scorer	`supports`	correct	96.8%	0.999	`supports` 96.8% `not_enough_info` 1.8% `refutes` 1.4%
JevFish	`supports`	correct	98.1%	1.000	`supports` 98.1% `not_enough_info` 1.1% `refutes` 0.8%
NanoJev	`supports`	correct	92.7%	0.996	`supports` 92.7% `refutes` 3.8% `not_enough_info` 3.5%

Question supported · noul

Using only the evidence, is the claim true?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	98.0%	1.000	`yes` 98.0% `no` 2.0%
OpenJev	`yes`	correct	96.6%	0.999	`yes` 96.6% `no` 3.4%
decider-2b	`yes`	correct	96.2%	0.999	`yes` 96.2% `no` 3.8%
System One Scorer	`yes`	correct	90.6%	0.991	`yes` 90.6% `no` 9.4%
JevFish	`yes`	correct	96.6%	0.999	`yes` 96.6% `no` 3.4%
NanoJev	`yes`	correct	83.3%	0.972	`yes` 83.3% `no` 16.7%

Sample 2 of 6 · safety-and-quality:vitaminc-26876 · 71 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "claim": "There were 266 confirmed cases of coronavirus in Palestine during the pandemic .",
  "evidence": "10 April*Ibrahim Milhem confirmed 3 new cases , one from Halhul and two cases from Artas for a total of 266 .",
  "page": "2020 coronavirus pandemic in the State of Palestine"
}

Question verdict · choice · primary (ranked)

Using only the evidence (one sentence from the Wikipedia article named in page), what is the verdict on the claim?

3 options

supports — The evidence establishes that the claim is true.
refutes — The evidence establishes that the claim is false.
not_enough_info — The evidence does not settle whether the claim is true or false.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: supports

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`supports`	correct	92.0%	0.994	`supports` 92.0% `not_enough_info` 7.0% `refutes` 1.0%
OpenJev	`supports`	correct	99.2%	1.000	`supports` 99.2% `refutes` 0.8% `not_enough_info` 0.0%
decider-2b	`supports`	correct	79.1%	0.960	`supports` 79.1% `not_enough_info` 18.7% `refutes` 2.3%
System One Scorer	`supports`	correct	84.5%	0.981	`supports` 84.5% `not_enough_info` 10.1% `refutes` 5.5%
JevFish	`supports`	correct	82.7%	0.977	`supports` 82.7% `not_enough_info` 11.3% `refutes` 5.9%
NanoJev	`supports`	correct	85.0%	0.983	`supports` 85.0% `not_enough_info` 7.6% `refutes` 7.3%

Question supported · noul

Using only the evidence, is the claim true?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	86.0%	0.980	`yes` 86.0% `no` 14.0%
OpenJev	`yes`	correct	95.7%	0.998	`yes` 95.7% `no` 4.3%
decider-2b	`yes`	correct	80.1%	0.961	`yes` 80.1% `no` 19.9%
System One Scorer	`yes`	correct	79.6%	0.958	`yes` 79.6% `no` 20.4%
JevFish	`yes`	correct	71.4%	0.918	`yes` 71.4% `no` 28.6%
NanoJev	`yes`	correct	64.7%	0.875	`yes` 64.7% `no` 35.3%

Sample 3 of 6 · safety-and-quality:vitaminc-28695 · 99 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "claim": "John Wick : Chapter 3 - Parabellum grossed more than $ 255.5 million worldwide .",
  "evidence": "It has grossed $ 255 million worldwide , surpassing the entire gross of the second film in just 10 days , and received positive reviews from critics , with praise for the fight choreography , visual style , and Reeves ' performance.",
  "page": "John Wick-COLON- Chapter 3 – Parabellum"
}

Question verdict · choice · primary (ranked)

Using only the evidence (one sentence from the Wikipedia article named in page), what is the verdict on the claim?

3 options

supports — The evidence establishes that the claim is true.
refutes — The evidence establishes that the claim is false.
not_enough_info — The evidence does not settle whether the claim is true or false.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: refutes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`refutes`	correct	93.0%	0.996	`refutes` 93.0% `not_enough_info` 6.0% `supports` 1.0%
OpenJev	`refutes`	correct	52.9%	0.808	`refutes` 52.9% `not_enough_info` 39.6% `supports` 7.6%
decider-2b	`supports`	wrong	9.0%	0.373	`supports` 53.1% `not_enough_info` 37.9% `refutes` 9.0%
System One Scorer	`supports`	wrong	31.2%	0.623	`supports` 49.0% `refutes` 31.2% `not_enough_info` 19.8%
JevFish	`supports`	wrong	2.4%	0.088	`supports` 93.2% `not_enough_info` 4.5% `refutes` 2.4%
NanoJev	`supports`	wrong	4.2%	0.107	`supports` 93.1% `refutes` 4.2% `not_enough_info` 2.7%

Question supported · noul

Using only the evidence, is the claim true?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	correct	91.0%	0.992	`yes` 9.0% `no` 91.0%
OpenJev	`no`	correct	96.0%	0.998	`yes` 4.0% `no` 96.0%
decider-2b	`no`	correct	51.2%	0.762	`yes` 48.8% `no` 51.2%
System One Scorer	`yes`	wrong	40.8%	0.649	`yes` 59.2% `no` 40.8%
JevFish	`yes`	wrong	13.0%	0.242	`yes` 87.1% `no` 13.0%
NanoJev	`yes`	wrong	29.4%	0.502	`yes` 70.6% `no` 29.4%

Sample 4 of 6 · safety-and-quality:vitaminc-28810 · 101 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "claim": "Nanabhai Haridas & family reside in Malabar Hill .",
  "evidence": "Notable residents include Adi Godrej , the Birla family , Shashi Ruia & family , Pallonji Mistry , Mahesh Jethmalani , the Jindal family , Nanabhai Haridas & family , the Lal family , the Thakkar family , the Shah family , the Karnavat family , among many more.",
  "page": "Malabar Hill"
}

Question verdict · choice · primary (ranked)

Using only the evidence (one sentence from the Wikipedia article named in page), what is the verdict on the claim?

3 options

supports — The evidence establishes that the claim is true.
refutes — The evidence establishes that the claim is false.
not_enough_info — The evidence does not settle whether the claim is true or false.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: not_enough_info

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`supports`	wrong	1.0%	0.020	`supports` 99.0% `not_enough_info` 1.0% `refutes` 0.0%
OpenJev	`supports`	wrong	0.0%	0.000	`supports` 100.0% `refutes` 0.0% `not_enough_info` 0.0%
decider-2b	`not_enough_info`	correct	50.0%	0.761	`not_enough_info` 50.0% `supports` 47.6% `refutes` 2.4%
System One Scorer	`supports`	wrong	13.5%	0.272	`supports` 84.1% `not_enough_info` 13.5% `refutes` 2.5%
JevFish	`not_enough_info`	correct	76.6%	0.958	`not_enough_info` 76.6% `refutes` 15.1% `supports` 8.3%
NanoJev	`supports`	wrong	7.5%	0.242	`supports` 80.3% `refutes` 12.2% `not_enough_info` 7.5%

Question supported · noul

Using only the evidence, is the claim true?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: not graded (no label recorded for this item)

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	answered (ungraded)
OpenJev	answered (ungraded)
decider-2b	answered (ungraded)
System One Scorer	answered (ungraded)
JevFish	answered (ungraded)
NanoJev	answered (ungraded)

Sample 5 of 6 · safety-and-quality:vitaminc-29724 · 94 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "claim": "Following the confirmation of multiple COVID-19 cases in North Rhine-Westphalia , the district of Heinsberg initiated the closure of schools .",
  "evidence": "On 26 February , following the confirmation of multiple COVID-19 cases in North Rhine-Westphalia , Heinsberg initiated closure of schools , swimming pools , libraries and the town hall until 2 March .",
  "page": "2020 coronavirus pandemic in Germany"
}

Question verdict · choice · primary (ranked)

Using only the evidence (one sentence from the Wikipedia article named in page), what is the verdict on the claim?

3 options

supports — The evidence establishes that the claim is true.
refutes — The evidence establishes that the claim is false.
not_enough_info — The evidence does not settle whether the claim is true or false.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: not_enough_info

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`supports`	wrong	0.0%	0.000	`supports` 100.0% `not_enough_info` 0.0% `refutes` 0.0%
OpenJev	`supports`	wrong	0.0%	0.000	`supports` 100.0% `refutes` 0.0% `not_enough_info` 0.0%
decider-2b	`supports`	wrong	1.7%	0.036	`supports` 98.0% `not_enough_info` 1.7% `refutes` 0.4%
System One Scorer	`supports`	wrong	3.2%	0.083	`supports` 94.6% `not_enough_info` 3.2% `refutes` 2.2%
JevFish	`supports`	wrong	38.0%	0.652	`supports` 55.4% `not_enough_info` 38.0% `refutes` 6.6%
NanoJev	`supports`	wrong	4.3%	0.119	`supports` 91.8% `not_enough_info` 4.3% `refutes` 3.9%

Question supported · noul

Using only the evidence, is the claim true?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: not graded (no label recorded for this item)

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	answered (ungraded)
OpenJev	answered (ungraded)
decider-2b	answered (ungraded)
System One Scorer	answered (ungraded)
JevFish	answered (ungraded)
NanoJev	answered (ungraded)

Sample 6 of 6 · safety-and-quality:vitaminc-8120 · 54 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "claim": "Angel Olsen sings and plays folk and indie rock music .",
  "evidence": "Angel Olsen is an American indie pop singer and guitarist , who was raised in St. Louis , Missouri , and is now based in Chicago , Illinois .",
  "page": "Angel Olsen"
}

Question verdict · choice · primary (ranked)

Using only the evidence (one sentence from the Wikipedia article named in page), what is the verdict on the claim?

3 options

supports — The evidence establishes that the claim is true.
refutes — The evidence establishes that the claim is false.
not_enough_info — The evidence does not settle whether the claim is true or false.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: refutes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`not_enough_info`	wrong	7.0%	0.171	`not_enough_info` 89.0% `refutes` 7.0% `supports` 4.0%
OpenJev	`not_enough_info`	wrong	2.7%	0.065	`not_enough_info` 96.1% `refutes` 2.7% `supports` 1.2%
decider-2b	`not_enough_info`	wrong	2.4%	0.118	`not_enough_info` 89.8% `supports` 7.7% `refutes` 2.4%
System One Scorer	`not_enough_info`	wrong	33.6%	0.617	`not_enough_info` 56.1% `refutes` 33.6% `supports` 10.3%
JevFish	`not_enough_info`	wrong	21.5%	0.401	`not_enough_info` 76.3% `refutes` 21.5% `supports` 2.2%
NanoJev	`supports`	wrong	2.8%	0.078	`supports` 94.8% `refutes` 2.8% `not_enough_info` 2.4%

Question supported · noul

Using only the evidence, is the claim true?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	correct	75.0%	0.938	`yes` 25.0% `no` 75.0%
OpenJev	`no`	correct	96.8%	0.999	`yes` 3.2% `no` 96.8%
decider-2b	`yes`	wrong	48.8%	0.738	`yes` 51.2% `no` 48.8%
System One Scorer	`no`	correct	64.1%	0.871	`yes` 35.9% `no` 64.1%
JevFish	`no`	correct	89.2%	0.988	`yes` 10.8% `no` 89.2%
NanoJev	`yes`	wrong	21.5%	0.383	`yes` 78.5% `no` 21.5%