Typed Decision Bench · tasks · Safety and quality

`humanevalpack-spec-match`

semantic linting · software engineering · 164 items · primary question type noul. Source bigcode/humanevalpack (MIT); labels are verifier labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
Jev	82.8	89.0%	164 / 164
OpenJev	80.5	88.4%	164 / 164
System One Scorer	73.2	65.8%	164 / 164
JevFish	71.6	62.8%	164 / 164
decider-2b	71.4	65.2%	164 / 164
NanoJev	68.1	50.3%	163 / 164

Sample 1 of 6 · safety-and-quality:heval-Python-163-ok · 175 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "file": "solution.py",
  "function": "def generate_integers(a, b):\n    \"\"\"\n    Given two positive integers a and b, return the even digits between a\n    and b, in ascending order.\n\n    For example:\n    generate_integers(2, 8) => [2, 4, 6, 8]\n    generate_integers(8, 2) => [2, 4, 6, 8]\n    generate_integers(10, 14) => []\n    \"\"\"\n    lower = max(2, min(a, b))\n    upper = min(8, max(a, b))\n\n    return [i for i in range(lower, upper+1) if i % 2 == 0]"
}

Question implements_spec · noul · primary (ranked)

The state holds a Python function whose docstring states the required behaviour, followed by its body. Does the body correctly implement the docstring for all valid inputs (it would pass a hidden test suite written from the docstring)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	wrong	29.0%	0.496	`yes` 29.0% `no` 71.0%
OpenJev	`no`	wrong	23.8%	0.419	`yes` 23.8% `no` 76.2%
System One Scorer	`yes`	correct	65.3%	0.879	`yes` 65.3% `no` 34.7%
JevFish	`yes`	correct	62.0%	0.856	`yes` 62.0% `no` 38.0%
decider-2b	`yes`	correct	62.9%	0.863	`yes` 62.9% `no` 37.1%
NanoJev	`yes`	correct	64.8%	0.876	`yes` 64.8% `no` 35.2%

Question defect_type · choice

Which single kind of defect does the body have relative to its docstring? Pick no_bug if it is correct.

7 options

no_bug — The body implements the docstring correctly.
missing_logic — A needed step, condition or case is absent.
excess_logic — An extra step or condition changes the result.
value_misuse — A wrong literal, constant or initial value is used.
operator_misuse — A wrong operator or comparison is used.
variable_misuse — The wrong variable is read or written.
function_misuse — The wrong function or method is called.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no_bug

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`value_misuse`	wrong	22.0%	0.597	`value_misuse` 30.0% `missing_logic` 28.0% `no_bug` 22.0% `excess_logic` 17.0% `variable_misuse` 1.0% `operator_misuse` 1.0%
OpenJev	`missing_logic`	wrong	10.8%	0.481	`missing_logic` 33.9% `excess_logic` 32.8% `value_misuse` 13.3% `no_bug` 10.8% `function_misuse` 4.0% `operator_misuse` 3.9%
System One Scorer	`no_bug`	correct	69.5%	0.942	`no_bug` 69.5% `missing_logic` 10.7% `excess_logic` 8.5% `operator_misuse` 4.3% `value_misuse` 4.2% `variable_misuse` 2.3%
JevFish	`no_bug`	correct	51.9%	0.856	`no_bug` 51.9% `excess_logic` 17.9% `value_misuse` 12.1% `missing_logic` 8.2% `operator_misuse` 4.7% `variable_misuse` 3.8%
decider-2b	`no_bug`	correct	41.5%	0.786	`no_bug` 41.5% `excess_logic` 22.2% `value_misuse` 13.8% `missing_logic` 11.3% `operator_misuse` 7.0% `variable_misuse` 2.6%
NanoJev	`no_bug`	correct	33.5%	0.739	`no_bug` 33.5% `excess_logic` 18.9% `value_misuse` 9.9% `missing_logic` 9.8% `operator_misuse` 9.7% `function_misuse` 9.2%

Sample 2 of 6 · safety-and-quality:heval-Python-27-ok · 67 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "file": "solution.py",
  "function": "def flip_case(string: str) -> str:\n    \"\"\" For a given string, flip lowercase characters to uppercase and uppercase to lowercase.\n    >>> flip_case('Hello')\n    'hELLO'\n    \"\"\"\n    return string.swapcase()"
}

Question implements_spec · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	98.0%	1.000	`yes` 98.0% `no` 2.0%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%
System One Scorer	`yes`	correct	85.4%	0.979	`yes` 85.4% `no` 14.6%
JevFish	`yes`	correct	85.7%	0.980	`yes` 85.7% `no` 14.3%
decider-2b	`yes`	correct	78.5%	0.954	`yes` 78.5% `no` 21.4%
NanoJev	`yes`	correct	60.9%	0.847	`yes` 60.9% `no` 39.1%

Question defect_type · choice

Which single kind of defect does the body have relative to its docstring? Pick no_bug if it is correct.

7 options

no_bug — The body implements the docstring correctly.
missing_logic — A needed step, condition or case is absent.
excess_logic — An extra step or condition changes the result.
value_misuse — A wrong literal, constant or initial value is used.
operator_misuse — A wrong operator or comparison is used.
variable_misuse — The wrong variable is read or written.
function_misuse — The wrong function or method is called.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no_bug

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no_bug`	correct	100.0%	1.000	`no_bug` 100.0% `variable_misuse` 0.0% `excess_logic` 0.0% `operator_misuse` 0.0% `value_misuse` 0.0% `function_misuse` 0.0%
OpenJev	`no_bug`	correct	100.0%	1.000	`no_bug` 100.0% `missing_logic` 0.0% `value_misuse` 0.0% `function_misuse` 0.0% `excess_logic` 0.0% `variable_misuse` 0.0%
System One Scorer	`no_bug`	correct	94.8%	0.998	`no_bug` 94.8% `missing_logic` 1.5% `excess_logic` 1.2% `operator_misuse` 0.8% `variable_misuse` 0.8% `value_misuse` 0.6%
JevFish	`no_bug`	correct	84.4%	0.984	`no_bug` 84.4% `function_misuse` 6.7% `excess_logic` 4.3% `operator_misuse` 2.0% `missing_logic` 1.5% `variable_misuse` 0.8%
decider-2b	`no_bug`	correct	74.6%	0.959	`no_bug` 74.6% `function_misuse` 10.9% `excess_logic` 5.1% `missing_logic` 4.0% `value_misuse` 2.7% `operator_misuse` 1.9%
NanoJev	`no_bug`	correct	36.0%	0.759	`no_bug` 36.0% `excess_logic` 16.4% `missing_logic` 10.5% `value_misuse` 10.1% `operator_misuse` 9.9% `variable_misuse` 9.0%

Sample 3 of 6 · safety-and-quality:heval-Python-68-ok · 329 state tokens (long tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "file": "solution.py",
  "function": "def pluck(arr):\n    \"\"\"\n    \"Given an array representing a branch of a tree that has non-negative integer nodes\n    your task is to pluck one of the nodes and return it.\n    The plucked node should be the node with the smallest even value.\n    If multiple nodes with the same smallest even value are found return the node that has smallest index.\n\n    The plucked node should be returned in a list, [ smalest_value, its index ],\n    If there are no even values or the given array is empty, return [].\n\n    Example 1:\n        Input: [4,2,3]\n        Output: [2, 1]\n        Explanation: 2 has the smallest even value, and 2 has the smallest index.\n\n    Example 2:\n        Input: [1,2,3]\n        Output: [2, 1]\n        Explanation: 2 has the smallest even value, and 2 has the smallest index. \n\n    Example 3:\n        Input: []\n        Output: []\n    \n    Example 4:\n        Input: [5, 0, 3, 0, 4, 2]\n        Output: [0, 1]\n        Explanation: 0 is the smallest value, but  there are two zeros,\n                     so we will choose the first zero, which has the smallest index.\n\n    Constraints: …"
}

Question implements_spec · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	wrong	34.0%	0.564	`yes` 34.0% `no` 66.0%
OpenJev	`no`	wrong	5.2%	0.101	`yes` 5.2% `no` 94.8%
System One Scorer	`yes` input truncated	correct	67.7%	0.895	`yes` 67.7% `no` 32.3%
JevFish	`no`	wrong	48.6%	0.736	`yes` 48.6% `no` 51.4%
decider-2b	`yes`	correct	62.9%	0.863	`yes` 62.9% `no` 37.1%
NanoJev	`yes`	correct	61.4%	0.851	`yes` 61.4% `no` 38.6%

Question defect_type · choice

Which single kind of defect does the body have relative to its docstring? Pick no_bug if it is correct.

7 options

no_bug — The body implements the docstring correctly.
missing_logic — A needed step, condition or case is absent.
excess_logic — An extra step or condition changes the result.
value_misuse — A wrong literal, constant or initial value is used.
operator_misuse — A wrong operator or comparison is used.
variable_misuse — The wrong variable is read or written.
function_misuse — The wrong function or method is called.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no_bug

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`missing_logic`	wrong	33.0%	0.551	`missing_logic` 67.0% `no_bug` 33.0% `variable_misuse` 0.0% `operator_misuse` 0.0% `value_misuse` 0.0% `excess_logic` 0.0%
OpenJev	`missing_logic`	wrong	11.7%	0.394	`missing_logic` 63.9% `function_misuse` 13.3% `no_bug` 11.7% `variable_misuse` 7.5% `excess_logic` 1.6% `value_misuse` 1.6%
System One Scorer	`no_bug` input truncated	correct	66.4%	0.928	`no_bug` 66.4% `missing_logic` 14.6% `excess_logic` 5.6% `value_misuse` 5.2% `operator_misuse` 3.8% `variable_misuse` 3.7%
JevFish	`no_bug`	correct	48.1%	0.824	`no_bug` 48.1% `excess_logic` 21.3% `missing_logic` 18.3% `variable_misuse` 4.8% `operator_misuse` 4.3% `function_misuse` 1.6%
decider-2b	`no_bug`	correct	40.4%	0.773	`no_bug` 40.4% `missing_logic` 24.9% `excess_logic` 16.2% `operator_misuse` 5.9% `variable_misuse` 5.1% `value_misuse` 4.6%
NanoJev	`no_bug`	correct	43.1%	0.809	`no_bug` 43.1% `excess_logic` 14.8% `missing_logic` 9.6% `value_misuse` 9.0% `operator_misuse` 8.7% `variable_misuse` 7.4%

Sample 4 of 6 · safety-and-quality:heval-Python-69-buggy · 254 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "file": "solution.py",
  "function": "def search(lst):\n    '''\n    You are given a non-empty list of positive integers. Return the greatest integer that is greater than \n    zero, and has a frequency greater than or equal to the value of the integer itself. \n    The frequency of an integer is the number of times it appears in the list.\n    If no such a value exist, return -1.\n    Examples:\n        search([4, 1, 2, 2, 3, 1]) == 2\n        search([1, 2, 2, 3, 3, 3, 4, 4, 4]) == 3\n        search([5, 5, 4, 4, 4]) == -1\n    '''\n    frq = [0] * (max(lst) + 1)\n    for i in lst:\n        frq[i] += 1;\n\n    ans = 0\n    for i in range(1, len(frq)):\n        if frq[i] >= i:\n            ans = i\n    \n    return ans"
}

Question implements_spec · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	wrong	46.0%	0.708	`yes` 54.0% `no` 46.0%
OpenJev	`no`	correct	99.5%	1.000	`yes` 0.5% `no` 99.5%
System One Scorer	`yes`	wrong	21.0%	0.376	`yes` 79.0% `no` 21.0%
JevFish	`yes`	wrong	14.7%	0.273	`yes` 85.3% `no` 14.7%
decider-2b	`yes`	wrong	12.2%	0.230	`yes` 87.8% `no` 12.2%
NanoJev	`yes`	wrong	37.1%	0.605	`yes` 62.9% `no` 37.1%

Question defect_type · choice

Which single kind of defect does the body have relative to its docstring? Pick no_bug if it is correct.

7 options

no_bug — The body implements the docstring correctly.
missing_logic — A needed step, condition or case is absent.
excess_logic — An extra step or condition changes the result.
value_misuse — A wrong literal, constant or initial value is used.
operator_misuse — A wrong operator or comparison is used.
variable_misuse — The wrong variable is read or written.
function_misuse — The wrong function or method is called.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: value_misuse

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`value_misuse`	correct	63.0%	0.899	`value_misuse` 63.0% `no_bug` 19.0% `missing_logic` 17.0% `variable_misuse` 1.0% `operator_misuse` 0.0% `excess_logic` 0.0%
OpenJev	`value_misuse`	correct	88.6%	0.992	`value_misuse` 88.6% `operator_misuse` 3.7% `missing_logic` 2.8% `no_bug` 2.1% `excess_logic` 1.6% `variable_misuse` 0.7%
System One Scorer	`no_bug`	wrong	2.1%	0.168	`no_bug` 83.7% `missing_logic` 5.9% `variable_misuse` 3.0% `excess_logic` 2.7% `operator_misuse` 2.2% `value_misuse` 2.1%
JevFish	`no_bug`	wrong	2.7%	0.184	`no_bug` 82.5% `excess_logic` 5.4% `missing_logic` 4.4% `value_misuse` 2.7% `variable_misuse` 2.3% `operator_misuse` 2.3%
decider-2b	`no_bug`	wrong	3.5%	0.238	`no_bug` 76.3% `missing_logic` 8.4% `excess_logic` 5.4% `value_misuse` 3.5% `operator_misuse` 3.5% `variable_misuse` 1.9%
NanoJev	`no_bug`	wrong	10.3%	0.503	`no_bug` 35.2% `excess_logic` 17.7% `value_misuse` 10.3% `missing_logic` 10.0% `operator_misuse` 9.6% `variable_misuse` 8.8%

Sample 5 of 6 · safety-and-quality:heval-Python-80-ok · 206 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "file": "solution.py",
  "function": "def is_happy(s):\n    \"\"\"You are given a string s.\n    Your task is to check if the string is happy or not.\n    A string is happy if its length is at least 3 and every 3 consecutive letters are distinct\n    For example:\n    is_happy(a) => False\n    is_happy(aa) => False\n    is_happy(abcd) => True\n    is_happy(aabb) => False\n    is_happy(adb) => True\n    is_happy(xyy) => False\n    \"\"\"\n    if len(s) < 3:\n      return False\n\n    for i in range(len(s) - 2):\n      \n      if s[i] == s[i+1] or s[i+1] == s[i+2] or s[i] == s[i+2]:\n        return False\n    return True"
}

Question implements_spec · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	94.0%	0.996	`yes` 94.0% `no` 6.0%
OpenJev	`yes`	correct	98.9%	1.000	`yes` 98.9% `no` 1.1%
System One Scorer	`yes`	correct	64.5%	0.874	`yes` 64.5% `no` 35.5%
JevFish	`yes`	correct	70.5%	0.913	`yes` 70.5% `no` 29.5%
decider-2b	`yes`	correct	82.3%	0.969	`yes` 82.3% `no` 17.7%
NanoJev	`yes`	correct	64.1%	0.872	`yes` 64.1% `no` 35.9%

Question defect_type · choice

Which single kind of defect does the body have relative to its docstring? Pick no_bug if it is correct.

7 options

no_bug — The body implements the docstring correctly.
missing_logic — A needed step, condition or case is absent.
excess_logic — An extra step or condition changes the result.
value_misuse — A wrong literal, constant or initial value is used.
operator_misuse — A wrong operator or comparison is used.
variable_misuse — The wrong variable is read or written.
function_misuse — The wrong function or method is called.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no_bug

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no_bug`	correct	100.0%	1.000	`no_bug` 100.0% `value_misuse` 0.0% `function_misuse` 0.0% `operator_misuse` 0.0% `variable_misuse` 0.0% `excess_logic` 0.0%
OpenJev	`no_bug`	correct	99.5%	1.000	`no_bug` 99.5% `missing_logic` 0.4% `function_misuse` 0.1% `variable_misuse` 0.1% `value_misuse` 0.0% `operator_misuse` 0.0%
System One Scorer	`no_bug`	correct	85.5%	0.987	`no_bug` 85.5% `missing_logic` 6.0% `excess_logic` 2.8% `operator_misuse` 2.1% `value_misuse` 1.7% `variable_misuse` 1.5%
JevFish	`no_bug`	correct	72.2%	0.951	`no_bug` 72.2% `excess_logic` 10.1% `missing_logic` 8.2% `operator_misuse` 6.5% `variable_misuse` 1.5% `value_misuse` 0.9%
decider-2b	`no_bug`	correct	66.0%	0.928	`no_bug` 66.0% `missing_logic` 11.7% `excess_logic` 9.2% `operator_misuse` 6.9% `value_misuse` 3.2% `variable_misuse` 1.9%
NanoJev	`no_bug`	correct	35.0%	0.749	`no_bug` 35.0% `excess_logic` 19.5% `missing_logic` 10.9% `value_misuse` 9.4% `variable_misuse` 8.7% `operator_misuse` 8.6%

Sample 6 of 6 · safety-and-quality:heval-Python-98-ok · 127 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "file": "solution.py",
  "function": "def count_upper(s):\n    \"\"\"\n    Given a string s, count the number of uppercase vowels in even indices.\n    \n    For example:\n    count_upper('aBCdEf') returns 1\n    count_upper('abcdefg') returns 0\n    count_upper('dBBE') returns 0\n    \"\"\"\n    count = 0\n    for i in range(0,len(s),2):\n        if s[i] in \"AEIOU\":\n            count += 1\n    return count"
}

Question implements_spec · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	90.0%	0.990	`yes` 90.0% `no` 10.0%
OpenJev	`yes`	correct	98.6%	1.000	`yes` 98.6% `no` 1.4%
System One Scorer	`yes`	correct	75.6%	0.940	`yes` 75.6% `no` 24.4%
JevFish	`yes`	correct	68.8%	0.902	`yes` 68.8% `no` 31.3%
decider-2b	`yes`	correct	81.6%	0.966	`yes` 81.6% `no` 18.4%
NanoJev	`yes`	correct	67.9%	0.897	`yes` 67.9% `no` 32.1%

Question defect_type · choice

Which single kind of defect does the body have relative to its docstring? Pick no_bug if it is correct.

7 options

no_bug — The body implements the docstring correctly.
missing_logic — A needed step, condition or case is absent.
excess_logic — An extra step or condition changes the result.
value_misuse — A wrong literal, constant or initial value is used.
operator_misuse — A wrong operator or comparison is used.
variable_misuse — The wrong variable is read or written.
function_misuse — The wrong function or method is called.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no_bug

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no_bug`	correct	98.0%	1.000	`no_bug` 98.0% `missing_logic` 1.0% `value_misuse` 1.0% `function_misuse` 0.0% `excess_logic` 0.0% `variable_misuse` 0.0%
OpenJev	`no_bug`	correct	99.9%	1.000	`no_bug` 99.9% `missing_logic` 0.1% `operator_misuse` 0.0% `function_misuse` 0.0% `value_misuse` 0.0% `variable_misuse` 0.0%
System One Scorer	`no_bug`	correct	89.7%	0.993	`no_bug` 89.7% `missing_logic` 4.7% `excess_logic` 1.7% `variable_misuse` 1.4% `value_misuse` 1.3% `operator_misuse` 1.0%
JevFish	`no_bug`	correct	70.5%	0.944	`no_bug` 70.5% `excess_logic` 10.9% `missing_logic` 10.2% `operator_misuse` 3.7% `variable_misuse` 2.4% `value_misuse` 1.5%
decider-2b	`no_bug`	correct	67.8%	0.935	`no_bug` 67.8% `missing_logic` 10.9% `excess_logic` 10.4% `value_misuse` 4.0% `operator_misuse` 3.6% `variable_misuse` 1.9%
NanoJev	`no_bug`	correct	43.6%	0.812	`no_bug` 43.6% `excess_logic` 14.9% `missing_logic` 9.5% `value_misuse` 9.2% `operator_misuse` 8.4% `variable_misuse` 7.4%