Typed Decision Bench · tasks · Safety and quality

`humanevalpack-spec-match`

semantic linting · software engineering · 164 items · primary question type noul. Source bigcode/humanevalpack (MIT); labels are verifier labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
Jev	82.8	89.0%	164 / 164
OpenJev	80.5	88.4%	164 / 164
System One Scorer	73.2	65.8%	164 / 164
decider-2b	71.4	65.2%	164 / 164
JevFish	69.4	62.8%	164 / 164
NanoJev	68.1	50.3%	163 / 164

Sample 1 of 6 · safety-and-quality:heval-Python-112-ok · 207 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "file": "solution.py",
  "function": "def reverse_delete(s,c):\n    \"\"\"Task\n    We are given two strings s and c, you have to deleted all the characters in s that are equal to any character in c\n    then check if the result string is palindrome.\n    A string is called palindrome if it reads the same backward as forward.\n    You should return a tuple containing the result string and True/False for the check.\n    Example\n    For s = \"abcde\", c = \"ae\", the result should be ('bcd',False)\n    For s = \"abcdef\", c = \"b\"  the result should be ('acdef',False)\n    For s = \"abcdedcba\", c = \"ab\", the result should be ('cdedc',True)\n    \"\"\"\n    s = ''.join([char for char in s if char not in c])\n    return (s,s[::-1] == s)"
}

Question implements_spec · noul · primary (ranked)

The state holds a Python function whose docstring states the required behaviour, followed by its body. Does the body correctly implement the docstring for all valid inputs (it would pass a hidden test suite written from the docstring)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	91.0%	0.992	`yes` 91.0% `no` 9.0%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%
System One Scorer	`yes`	correct	78.5%	0.954	`yes` 78.5% `no` 21.5%
decider-2b	`yes`	correct	77.7%	0.950	`yes` 77.7% `no` 22.3%
JevFish	`yes`	correct	71.3%	0.917	`yes` 71.3% `no` 28.7%
NanoJev	`yes`	correct	56.1%	0.807	`yes` 56.1% `no` 43.9%

Question defect_type · choice

Which single kind of defect does the body have relative to its docstring? Pick no_bug if it is correct.

7 options

no_bug — The body implements the docstring correctly.
missing_logic — A needed step, condition or case is absent.
excess_logic — An extra step or condition changes the result.
value_misuse — A wrong literal, constant or initial value is used.
operator_misuse — A wrong operator or comparison is used.
variable_misuse — The wrong variable is read or written.
function_misuse — The wrong function or method is called.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no_bug

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no_bug`	correct	100.0%	1.000	`no_bug` 100.0% `value_misuse` 0.0% `function_misuse` 0.0% `excess_logic` 0.0% `variable_misuse` 0.0% `missing_logic` 0.0%
OpenJev	`no_bug`	correct	100.0%	1.000	`no_bug` 100.0% `missing_logic` 0.0% `value_misuse` 0.0% `function_misuse` 0.0% `excess_logic` 0.0% `operator_misuse` 0.0%
System One Scorer	`no_bug`	correct	83.2%	0.983	`no_bug` 83.2% `missing_logic` 5.7% `operator_misuse` 3.7% `variable_misuse` 2.9% `value_misuse` 2.1% `excess_logic` 1.8%
decider-2b	`no_bug`	correct	82.4%	0.981	`no_bug` 82.4% `excess_logic` 5.1% `missing_logic` 4.6% `operator_misuse` 2.7% `value_misuse` 2.5% `variable_misuse` 1.5%
JevFish	`no_bug`	correct	85.2%	0.987	`no_bug` 85.2% `missing_logic` 5.7% `excess_logic` 2.6% `operator_misuse` 2.1% `variable_misuse` 1.8% `function_misuse` 1.6%
NanoJev	`no_bug`	correct	30.8%	0.719	`no_bug` 30.8% `excess_logic` 17.0% `missing_logic` 12.6% `value_misuse` 11.2% `variable_misuse` 9.8% `operator_misuse` 9.6%

Sample 2 of 6 · safety-and-quality:heval-Python-159-buggy · 356 state tokens (long tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "file": "solution.py",
  "function": "def eat(number, need, remaining):\n    \"\"\"\n    You're a hungry rabbit, and you already have eaten a certain number of carrots,\n    but now you need to eat more carrots to complete the day's meals.\n    you should return an array of [ total number of eaten carrots after your meals,\n                                    the number of carrots left after your meals ]\n    if there are not enough remaining carrots, you will eat all remaining carrots, but will still be hungry.\n    \n    Example:\n    * eat(5, 6, 10) -> [11, 4]\n    * eat(4, 8, 9) -> [12, 1]\n    * eat(1, 10, 10) -> [11, 0]\n    * eat(2, 11, 5) -> [7, 0]\n    \n    Variables:\n    @number : integer\n        the number of carrots that you have eaten.\n    @need : integer\n        the number of carrots that you need to eat.\n    @remaining : integer\n        the number of remaining carrots thet exist in stock\n    \n    Constrain:\n    * 0 <= number <= 1000\n    * 0 <= need <= 1000\n    * 0 <= remaining <= 1000\n\n    Have fun :)\n    \"\"\"\n    if(need <= remaining):\n        return [ number + need , number + remaining-need ]\n    else:\n        return [ …"
}

Question implements_spec · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	correct	94.0%	0.996	`yes` 6.0% `no` 94.0%
OpenJev	`no`	correct	99.0%	1.000	`yes` 1.0% `no` 99.0%
System One Scorer	`yes` input truncated	wrong	32.0%	0.537	`yes` 68.0% `no` 32.0%
decider-2b	`no`	correct	53.6%	0.785	`yes` 46.4% `no` 53.6%
JevFish	`no`	correct	55.3%	0.800	`yes` 44.7% `no` 55.3%
NanoJev	`yes`	wrong	41.5%	0.658	`yes` 58.5% `no` 41.5%

Question defect_type · choice

Which single kind of defect does the body have relative to its docstring? Pick no_bug if it is correct.

7 options

no_bug — The body implements the docstring correctly.
missing_logic — A needed step, condition or case is absent.
excess_logic — An extra step or condition changes the result.
value_misuse — A wrong literal, constant or initial value is used.
operator_misuse — A wrong operator or comparison is used.
variable_misuse — The wrong variable is read or written.
function_misuse — The wrong function or method is called.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: excess_logic

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`missing_logic`	wrong	0.0%	0.177	`missing_logic` 79.0% `variable_misuse` 14.0% `operator_misuse` 4.0% `no_bug` 1.0% `value_misuse` 1.0% `function_misuse` 0.0% `excess_logic` 0.0%
OpenJev	`missing_logic`	wrong	0.1%	0.039	`missing_logic` 96.1% `no_bug` 1.5% `function_misuse` 0.9% `variable_misuse` 0.6% `operator_misuse` 0.5% `value_misuse` 0.3% `excess_logic` 0.1%
System One Scorer	`no_bug` input truncated	wrong	2.8%	0.338	`no_bug` 47.4% `missing_logic` 39.1% `operator_misuse` 4.1% `variable_misuse` 3.1% `value_misuse` 3.1% `excess_logic` 2.8%
decider-2b	`missing_logic`	wrong	19.5%	0.592	`missing_logic` 26.1% `no_bug` 24.8% `excess_logic` 19.5% `operator_misuse` 18.6% `variable_misuse` 5.3% `value_misuse` 4.0%
JevFish	`no_bug`	wrong	28.4%	0.648	`no_bug` 32.1% `missing_logic` 29.2% `excess_logic` 28.4% `variable_misuse` 4.1% `operator_misuse` 2.8% `function_misuse` 2.3%
NanoJev	`no_bug`	wrong	19.7%	0.613	`no_bug` 26.8% `excess_logic` 19.7% `value_misuse` 11.3% `operator_misuse` 10.9% `missing_logic` 10.6% `function_misuse` 10.4%

Sample 3 of 6 · safety-and-quality:heval-Python-27-ok · 67 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "file": "solution.py",
  "function": "def flip_case(string: str) -> str:\n    \"\"\" For a given string, flip lowercase characters to uppercase and uppercase to lowercase.\n    >>> flip_case('Hello')\n    'hELLO'\n    \"\"\"\n    return string.swapcase()"
}

Question implements_spec · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	98.0%	1.000	`yes` 98.0% `no` 2.0%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%
System One Scorer	`yes`	correct	85.4%	0.979	`yes` 85.4% `no` 14.6%
decider-2b	`yes`	correct	78.5%	0.954	`yes` 78.5% `no` 21.4%
JevFish	`yes`	correct	89.4%	0.989	`yes` 89.4% `no` 10.6%
NanoJev	`yes`	correct	60.9%	0.847	`yes` 60.9% `no` 39.1%

Question defect_type · choice

Which single kind of defect does the body have relative to its docstring? Pick no_bug if it is correct.

7 options

no_bug — The body implements the docstring correctly.
missing_logic — A needed step, condition or case is absent.
excess_logic — An extra step or condition changes the result.
value_misuse — A wrong literal, constant or initial value is used.
operator_misuse — A wrong operator or comparison is used.
variable_misuse — The wrong variable is read or written.
function_misuse — The wrong function or method is called.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no_bug

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no_bug`	correct	100.0%	1.000	`no_bug` 100.0% `variable_misuse` 0.0% `excess_logic` 0.0% `operator_misuse` 0.0% `value_misuse` 0.0% `function_misuse` 0.0%
OpenJev	`no_bug`	correct	100.0%	1.000	`no_bug` 100.0% `missing_logic` 0.0% `value_misuse` 0.0% `function_misuse` 0.0% `excess_logic` 0.0% `variable_misuse` 0.0%
System One Scorer	`no_bug`	correct	94.8%	0.998	`no_bug` 94.8% `missing_logic` 1.5% `excess_logic` 1.2% `operator_misuse` 0.8% `variable_misuse` 0.8% `value_misuse` 0.6%
decider-2b	`no_bug`	correct	74.6%	0.959	`no_bug` 74.6% `function_misuse` 10.9% `excess_logic` 5.1% `missing_logic` 4.0% `value_misuse` 2.7% `operator_misuse` 1.9%
JevFish	`no_bug`	correct	98.9%	1.000	`no_bug` 98.9% `missing_logic` 0.3% `operator_misuse` 0.2% `excess_logic` 0.2% `function_misuse` 0.1% `variable_misuse` 0.1%
NanoJev	`no_bug`	correct	36.0%	0.759	`no_bug` 36.0% `excess_logic` 16.4% `missing_logic` 10.5% `value_misuse` 10.1% `operator_misuse` 9.9% `variable_misuse` 9.0%

Sample 4 of 6 · safety-and-quality:heval-Python-35-ok · 131 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "file": "solution.py",
  "function": "def max_element(l: list):\n    \"\"\"Return maximum element in the list.\n    >>> max_element([1, 2, 3])\n    3\n    >>> max_element([5, 3, -5, 2, -3, 3, 9, 0, 123, 1, -10])\n    123\n    \"\"\"\n    m = l[0]\n    for e in l:\n        if e > m:\n            m = e\n    return m"
}

Question implements_spec · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`yes`	correct	77.0%	0.947	`yes` 77.0% `no` 23.0%
OpenJev	`yes`	correct	96.3%	0.999	`yes` 96.3% `no` 3.7%
System One Scorer	`yes`	correct	76.2%	0.943	`yes` 76.2% `no` 23.8%
decider-2b	`yes`	correct	90.1%	0.990	`yes` 90.1% `no` 9.9%
JevFish	`yes`	correct	76.4%	0.944	`yes` 76.4% `no` 23.6%
NanoJev	`yes`	correct	57.9%	0.823	`yes` 57.9% `no` 42.1%

Question defect_type · choice

Which single kind of defect does the body have relative to its docstring? Pick no_bug if it is correct.

7 options

no_bug — The body implements the docstring correctly.
missing_logic — A needed step, condition or case is absent.
excess_logic — An extra step or condition changes the result.
value_misuse — A wrong literal, constant or initial value is used.
operator_misuse — A wrong operator or comparison is used.
variable_misuse — The wrong variable is read or written.
function_misuse — The wrong function or method is called.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no_bug

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no_bug`	correct	61.0%	0.852	`no_bug` 61.0% `missing_logic` 38.0% `value_misuse` 1.0% `function_misuse` 0.0% `excess_logic` 0.0% `variable_misuse` 0.0%
OpenJev	`no_bug`	correct	98.7%	1.000	`no_bug` 98.7% `missing_logic` 0.8% `function_misuse` 0.2% `operator_misuse` 0.1% `variable_misuse` 0.1% `value_misuse` 0.1%
System One Scorer	`no_bug`	correct	86.9%	0.990	`no_bug` 86.9% `missing_logic` 3.7% `value_misuse` 2.8% `operator_misuse` 2.3% `variable_misuse` 2.1% `excess_logic` 1.9%
decider-2b	`no_bug`	correct	76.3%	0.966	`no_bug` 76.3% `missing_logic` 6.3% `excess_logic` 6.0% `value_misuse` 4.1% `operator_misuse` 3.4% `variable_misuse` 3.0%
JevFish	`no_bug`	correct	94.0%	0.998	`no_bug` 94.0% `missing_logic` 2.1% `operator_misuse` 1.4% `excess_logic` 0.9% `variable_misuse` 0.7% `value_misuse` 0.5%
NanoJev	`no_bug`	correct	24.5%	0.663	`no_bug` 24.5% `excess_logic` 20.1% `value_misuse` 12.1% `operator_misuse` 11.7% `missing_logic` 11.3% `variable_misuse` 10.5%

Sample 5 of 6 · safety-and-quality:heval-Python-60-buggy · 125 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "file": "solution.py",
  "function": "def sum_to_n(n: int):\n    \"\"\"sum_to_n is a function that sums numbers from 1 to n.\n    >>> sum_to_n(30)\n    465\n    >>> sum_to_n(100)\n    5050\n    >>> sum_to_n(5)\n    15\n    >>> sum_to_n(10)\n    55\n    >>> sum_to_n(1)\n    1\n    \"\"\"\n    return sum(range(n))"
}

Question implements_spec · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	correct	95.0%	0.998	`yes` 5.0% `no` 95.0%
OpenJev	`no`	correct	99.9%	1.000	`yes` 0.1% `no` 99.9%
System One Scorer	`yes`	wrong	17.4%	0.318	`yes` 82.6% `no` 17.4%
decider-2b	`yes`	wrong	9.9%	0.188	`yes` 90.1% `no` 9.9%
JevFish	`yes`	wrong	28.7%	0.492	`yes` 71.3% `no` 28.7%
NanoJev	`yes`	wrong	39.0%	0.628	`yes` 61.0% `no` 39.0%

Question defect_type · choice

Which single kind of defect does the body have relative to its docstring? Pick no_bug if it is correct.

7 options

no_bug — The body implements the docstring correctly.
missing_logic — A needed step, condition or case is absent.
excess_logic — An extra step or condition changes the result.
value_misuse — A wrong literal, constant or initial value is used.
operator_misuse — A wrong operator or comparison is used.
variable_misuse — The wrong variable is read or written.
function_misuse — The wrong function or method is called.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: value_misuse

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`missing_logic`	wrong	14.0%	0.381	`missing_logic` 70.0% `value_misuse` 14.0% `no_bug` 7.0% `function_misuse` 5.0% `operator_misuse` 2.0% `variable_misuse` 1.0%
OpenJev	`operator_misuse`	wrong	1.0%	0.124	`operator_misuse` 87.4% `missing_logic` 8.7% `excess_logic` 1.3% `value_misuse` 1.0% `variable_misuse` 0.6% `no_bug` 0.5%
System One Scorer	`no_bug`	wrong	1.4%	0.102	`no_bug` 90.7% `missing_logic` 3.0% `excess_logic` 1.7% `operator_misuse` 1.5% `value_misuse` 1.4% `variable_misuse` 1.4%
decider-2b	`no_bug`	wrong	4.0%	0.167	`no_bug` 86.2% `value_misuse` 4.0% `missing_logic` 3.8% `excess_logic` 3.0% `operator_misuse` 1.3% `function_misuse` 1.2%
JevFish	`no_bug`	wrong	0.8%	0.077	`no_bug` 92.7% `missing_logic` 2.8% `excess_logic` 1.0% `function_misuse` 1.0% `operator_misuse` 0.8% `variable_misuse` 0.8% `value_misuse` 0.8%
NanoJev	`no_bug`	wrong	12.1%	0.538	`no_bug` 26.4% `excess_logic` 18.9% `value_misuse` 12.1% `missing_logic` 11.2% `operator_misuse` 10.9% `function_misuse` 10.4%

Sample 6 of 6 · safety-and-quality:heval-Python-61-buggy · 171 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "file": "solution.py",
  "function": "def correct_bracketing(brackets: str):\n    \"\"\" brackets is a string of \"(\" and \")\".\n    return True if every opening bracket has a corresponding closing bracket.\n\n    >>> correct_bracketing(\"(\")\n    False\n    >>> correct_bracketing(\"()\")\n    True\n    >>> correct_bracketing(\"(()())\")\n    True\n    >>> correct_bracketing(\")(()\")\n    False\n    \"\"\"\n    depth = 0\n    for b in brackets:\n        if b == \"(\":\n            depth += 1\n        else:\n            depth -= 1\n        if depth < 0:\n            return True\n    return depth == 0"
}

Question implements_spec · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`no`	correct	97.0%	0.999	`yes` 3.0% `no` 97.0%
OpenJev	`no`	correct	100.0%	1.000	`yes` 0.0% `no` 100.0%
System One Scorer	`yes`	wrong	50.0%	0.750	`yes` 50.0% `no` 50.0%
decider-2b	`yes`	wrong	34.9%	0.576	`yes` 65.1% `no` 34.9%
JevFish	`yes`	wrong	25.9%	0.451	`yes` 74.1% `no` 25.9%
NanoJev	`yes`	wrong	35.9%	0.590	`yes` 64.1% `no` 35.9%

Question defect_type · choice

Which single kind of defect does the body have relative to its docstring? Pick no_bug if it is correct.

7 options

no_bug — The body implements the docstring correctly.
missing_logic — A needed step, condition or case is absent.
excess_logic — An extra step or condition changes the result.
value_misuse — A wrong literal, constant or initial value is used.
operator_misuse — A wrong operator or comparison is used.
variable_misuse — The wrong variable is read or written.
function_misuse — The wrong function or method is called.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: operator_misuse

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`value_misuse`	wrong	13.0%	0.469	`value_misuse` 47.0% `excess_logic` 26.0% `missing_logic` 13.0% `operator_misuse` 13.0% `variable_misuse` 1.0% `no_bug` 0.0%
OpenJev	`operator_misuse`	correct	91.7%	0.996	`operator_misuse` 91.7% `no_bug` 2.8% `excess_logic` 1.8% `missing_logic` 1.3% `value_misuse` 1.1% `function_misuse` 1.0%
System One Scorer	`no_bug`	wrong	4.4%	0.358	`no_bug` 56.2% `missing_logic` 16.9% `excess_logic` 15.6% `operator_misuse` 4.4% `variable_misuse` 3.3% `value_misuse` 2.9%
decider-2b	`no_bug`	wrong	8.8%	0.435	`no_bug` 47.5% `excess_logic` 22.0% `missing_logic` 15.0% `operator_misuse` 8.8% `value_misuse` 2.8% `variable_misuse` 2.7%
JevFish	`no_bug`	wrong	0.9%	0.067	`no_bug` 94.0% `missing_logic` 2.8% `excess_logic` 1.2% `operator_misuse` 0.9% `variable_misuse` 0.5% `function_misuse` 0.4%
NanoJev	`no_bug`	wrong	8.5%	0.464	`no_bug` 42.9% `excess_logic` 14.6% `missing_logic` 9.5% `value_misuse` 8.9% `operator_misuse` 8.5% `variable_misuse` 8.2%