Typed Decision Bench · tasks · Real-time and agents

`k8s-issue-kind-routing`

agent routing and skill selection · devops · 200 items · primary question type choice. Source open-index/open-github-issues (ODC-BY-1.0); labels are outcome labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
Jev	77.5	72.5%	200 / 200
JevFish	75.5	64.5%	200 / 200
decider-2b	75.4	66.0%	200 / 200
OpenJev	73.8	71.0%	200 / 200
System One Scorer	73.5	61.0%	200 / 200
NanoJev	61.5	34.0%	200 / 200

Sample 1 of 6 · real-time-and-agents:k8s-115483 · 181 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "repo": "kubernetes/kubernetes",
  "title": "Secrets does not creating",
  "body": "### What happened? I am writing to report an issue with the latest version of Kubernetes in Azure, version v1.24.6. Before this version, when creating a Kubernetes cluster, the service account and secrets were automatically created and everything worked smoothly. However, with version v1.24.6, I have encountered an error when attempting to connect the cluster to Azure DevOps. Also during execution command in version v1.23.8 \"kubectl create serviceaccount some_name\", security token generating automatically. But not in v1.24.6. I found only 2 secrets in new cluster v1.24.6 comparing with v1.23.8 where was more than 30 secrets. I am checking all namespaces with command \"kubectl get secrets …"
}

Question kind · choice · primary (ranked)

Read `repo`, `title` and `body`. Which kubernetes issue kind will maintainers assign to this issue?

7 options

bug — Something is broken or behaves contrary to documentation (kind/bug)
feature — Request for new functionality or an enhancement (kind/feature)
cleanup — Refactor, tech-debt removal, deprecation of old code, no user-facing change (kind/cleanup)
documentation — Docs are missing, wrong or unclear (kind/documentation)
support — A usage question or help request rather than a defect (kind/support)
failing_test — A test fails consistently in CI (kind/failing-test)
flake — A test fails intermittently / flaky CI job (kind/flake)

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: support

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`bug`	wrong	23.0%	0.407	`bug` 77.0% `support` 23.0% `cleanup` 0.0% `documentation` 0.0% `failing_test` 0.0% `flake` 0.0%
JevFish	`bug`	wrong	16.6%	0.358	`bug` 76.7% `support` 16.6% `documentation` 2.2% `feature` 1.8% `cleanup` 1.8% `failing_test` 0.7%
decider-2b	`bug`	wrong	7.9%	0.258	`bug` 79.5% `support` 7.9% `feature` 4.2% `documentation` 3.7% `failing_test` 2.4% `cleanup` 1.7%
OpenJev	`bug`	wrong	0.0%	0.016	`bug` 98.4% `feature` 1.6% `flake` 0.0% `cleanup` 0.0% `support` 0.0% `documentation` 0.0%
System One Scorer	`support`	correct	38.6%	0.737	`support` 38.6% `bug` 36.9% `cleanup` 8.4% `documentation` 4.9% `failing_test` 4.3% `flake` 3.7%
NanoJev	`failing_test`	wrong	8.2%	0.481	`failing_test` 28.9% `bug` 23.3% `feature` 18.4% `flake` 14.2% `support` 8.2% `cleanup` 4.9%

Sample 2 of 6 · real-time-and-agents:k8s-115556 · 253 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "repo": "kubernetes/kubernetes",
  "title": "[Flaky Test] Conformance - GCE - master - kubetest2",
  "body": "### Which jobs were failing? Conformance - GCE - master - kubetest2 ### Which tests failed? It looks like the overall build may be failing ### When has it been failing? From 1:35pm - 3:35pm ### Testgrid link https://k8s-testgrid.appspot.com/sig-release-master-blocking#Conformance%20-%20GCE%20-%20master%20-%20kubetest2 ### Reason for failure (if possible) It looks like build/release.sh script blew up: ``` DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --load -t kube-build:build-6454b08013-5-v1.27.0-go1.20-bullseye.0 --pull=false --build-arg=KUBE_CROSS_IMAGE=registry.k8s.io/build-image/kube-cross --build-arg=KUBE_CROSS_VERSION=v1.27.0-go1.20-bullseye.0 …"
}

Question kind · choice · primary (ranked)

Read `repo`, `title` and `body`. Which kubernetes issue kind will maintainers assign to this issue?

7 options

bug — Something is broken or behaves contrary to documentation (kind/bug)
feature — Request for new functionality or an enhancement (kind/feature)
cleanup — Refactor, tech-debt removal, deprecation of old code, no user-facing change (kind/cleanup)
documentation — Docs are missing, wrong or unclear (kind/documentation)
support — A usage question or help request rather than a defect (kind/support)
failing_test — A test fails consistently in CI (kind/failing-test)
flake — A test fails intermittently / flaky CI job (kind/flake)

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: failing_test

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`flake`	wrong	1.0%	0.020	`flake` 99.0% `failing_test` 1.0% `cleanup` 0.0% `support` 0.0% `documentation` 0.0% `feature` 0.0%
JevFish	`flake`	wrong	6.8%	0.154	`flake` 90.7% `failing_test` 6.8% `bug` 2.3% `support` 0.1% `cleanup` 0.0% `feature` 0.0%
decider-2b	`flake`	wrong	10.6%	0.214	`flake` 87.9% `failing_test` 10.6% `bug` 1.1% `cleanup` 0.1% `feature` 0.1% `documentation` 0.1%
OpenJev	`flake`	wrong	0.2%	0.007	`flake` 99.6% `failing_test` 0.2% `feature` 0.1% `bug` 0.1% `documentation` 0.0% `cleanup` 0.0%
System One Scorer	`flake`	wrong	9.3%	0.409	`flake` 57.7% `cleanup` 10.5% `bug` 9.3% `failing_test` 9.3% `support` 6.9% `documentation` 3.5%
NanoJev	`flake`	wrong	25.6%	0.656	`flake` 28.5% `failing_test` 25.6% `bug` 17.0% `feature` 11.7% `support` 7.0% `cleanup` 5.4%

Sample 3 of 6 · real-time-and-agents:k8s-122673 · 244 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "repo": "kubernetes/kubernetes",
  "title": "kubernetes v1.24: systemd[14798]: Failed to set up mount unit: Invalid argument",
  "body": "### What happened? After upgrading the kubernetes version from 1.23 to 1.24, the OS /var/log/syslog log prompts: systemd[14798]: Failed to set up mount unit: Invalid argument, The pod on this host has some rbd volumes mounted. ![err](https://github.com/kubernetes/kubernetes/assets/35327558/0190cc72-64a3-4e11-8ac8-27ed74993b17) ### What did you expect to happen? Mounting the rbd volume is normal and such logs should not appear, The ceph rbd volume used by my cluster ### How can we reproduce it (as minimally and precisely as possible)? My operation was to upgrade kubernetes version v 1.23 to v1.24, and these logs appeared. ### Anything else we need to know? _No response_ ### Kubernetes …"
}

Question kind · choice · primary (ranked)

Read `repo`, `title` and `body`. Which kubernetes issue kind will maintainers assign to this issue?

7 options

bug — Something is broken or behaves contrary to documentation (kind/bug)
feature — Request for new functionality or an enhancement (kind/feature)
cleanup — Refactor, tech-debt removal, deprecation of old code, no user-facing change (kind/cleanup)
documentation — Docs are missing, wrong or unclear (kind/documentation)
support — A usage question or help request rather than a defect (kind/support)
failing_test — A test fails consistently in CI (kind/failing-test)
flake — A test fails intermittently / flaky CI job (kind/flake)

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: bug

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`bug`	correct	94.0%	0.996	`bug` 94.0% `support` 6.0% `failing_test` 0.0% `feature` 0.0% `documentation` 0.0% `flake` 0.0%
JevFish	`bug`	correct	82.1%	0.978	`bug` 82.1% `support` 10.1% `flake` 2.4% `failing_test` 2.3% `cleanup` 2.1% `documentation` 0.7%
decider-2b	`bug`	correct	78.5%	0.972	`bug` 78.5% `cleanup` 5.9% `failing_test` 5.1% `support` 4.8% `documentation` 2.7% `feature` 1.6%
OpenJev	`bug`	correct	98.2%	1.000	`bug` 98.2% `feature` 1.8% `flake` 0.0% `documentation` 0.0% `support` 0.0% `cleanup` 0.0%
System One Scorer	`support`	wrong	33.5%	0.677	`support` 43.6% `bug` 33.5% `cleanup` 8.7% `flake` 4.4% `documentation` 4.2% `failing_test` 3.5%
NanoJev	`failing_test`	wrong	20.2%	0.601	`failing_test` 32.0% `bug` 20.2% `feature` 18.0% `flake` 11.8% `support` 9.9% `cleanup` 5.3%

Sample 4 of 6 · real-time-and-agents:k8s-126206 · 166 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "repo": "kubernetes/kubernetes",
  "title": "WatchList failed when Accept content-type is Table",
  "body": "### What happened? When perform a List or WatchList with `Accept: application/json;as=Table;v=v1;g=meta.k8s.io`, api will convert every objects of response into `metav1.Table`. And as per the specification of the WatchList, a Bookmark event with `k8s.io/initial-events-end` should be sent after all initial events have been dispatched. Upon receiving this event, the client should close the watch and return the result. However, when response in Table format is returned, the annotations are placed on the `metav1.Table.Rows` instead of the top level of the object. This causes misbehavior in `client-go`. ``` # curl …"
}

Question kind · choice · primary (ranked)

Read `repo`, `title` and `body`. Which kubernetes issue kind will maintainers assign to this issue?

7 options

bug — Something is broken or behaves contrary to documentation (kind/bug)
feature — Request for new functionality or an enhancement (kind/feature)
cleanup — Refactor, tech-debt removal, deprecation of old code, no user-facing change (kind/cleanup)
documentation — Docs are missing, wrong or unclear (kind/documentation)
support — A usage question or help request rather than a defect (kind/support)
failing_test — A test fails consistently in CI (kind/failing-test)
flake — A test fails intermittently / flaky CI job (kind/flake)

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: bug

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`bug`	correct	100.0%	1.000	`bug` 100.0% `feature` 0.0% `documentation` 0.0% `flake` 0.0% `cleanup` 0.0% `failing_test` 0.0%
JevFish	`bug`	correct	77.3%	0.966	`bug` 77.3% `cleanup` 10.9% `failing_test` 4.3% `documentation` 4.3% `support` 1.3% `feature` 1.2%
decider-2b	`bug`	correct	76.5%	0.966	`bug` 76.5% `cleanup` 8.8% `documentation` 5.4% `failing_test` 4.5% `feature` 3.0% `support` 0.9%
OpenJev	`bug`	correct	95.3%	0.998	`bug` 95.3% `feature` 4.5% `flake` 0.1% `support` 0.0% `documentation` 0.0% `cleanup` 0.0%
System One Scorer	`bug`	correct	61.8%	0.912	`bug` 61.8% `cleanup` 12.6% `support` 7.9% `documentation` 5.7% `failing_test` 4.9% `feature` 4.0%
NanoJev	`failing_test`	wrong	16.6%	0.457	`failing_test` 61.4% `bug` 16.6% `flake` 9.3% `feature` 5.9% `support` 3.6% `cleanup` 2.1%

Sample 5 of 6 · real-time-and-agents:k8s-129663 · 222 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "repo": "kubernetes/kubernetes",
  "title": "E2eNode Suite.[It] [sig-node] Pod InPlace Resize Container [Feature:InPlacePodVerticalScaling] * is flaking",
  "body": "### Which jobs are flaking? ci-crio-cgroupv1-node-e2e-features pull-kubernetes-node-crio-cgrpv1-evented-pleg-e2e-kubetest2 pull-kubernetes-node-crio-cgrpv1-evented-pleg-e2e ### Which tests are flaking? All tests for `[sig-node] Pod InPlace Resize Container [Feature:InPlacePodVerticalScaling]` have failed. ### Since when has it been flaking? 01-16 07:40 CST ### Testgrid link https://testgrid.k8s.io/sig-node-cri-o#ci-crio-cgroupv1-node-e2e-features ### Reason for failure (if possible) _No response_ ### Anything else we need to know? _No response_ ### Relevant SIG(s)"
}

Question kind · choice · primary (ranked)

Read `repo`, `title` and `body`. Which kubernetes issue kind will maintainers assign to this issue?

7 options

bug — Something is broken or behaves contrary to documentation (kind/bug)
feature — Request for new functionality or an enhancement (kind/feature)
cleanup — Refactor, tech-debt removal, deprecation of old code, no user-facing change (kind/cleanup)
documentation — Docs are missing, wrong or unclear (kind/documentation)
support — A usage question or help request rather than a defect (kind/support)
failing_test — A test fails consistently in CI (kind/failing-test)
flake — A test fails intermittently / flaky CI job (kind/flake)

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: flake

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`flake`	correct	97.0%	0.999	`flake` 97.0% `failing_test` 3.0% `bug` 0.0% `documentation` 0.0% `cleanup` 0.0% `support` 0.0%
JevFish	`flake`	correct	74.2%	0.947	`flake` 74.2% `failing_test` 18.8% `bug` 6.5% `feature` 0.2% `cleanup` 0.1% `support` 0.1%
decider-2b	`flake`	correct	74.6%	0.948	`flake` 74.6% `failing_test` 19.4% `bug` 4.8% `cleanup` 0.4% `feature` 0.3% `documentation` 0.2%
OpenJev	`flake`	correct	99.9%	1.000	`flake` 99.9% `failing_test` 0.1% `feature` 0.0% `bug` 0.0% `cleanup` 0.0% `documentation` 0.0%
System One Scorer	`flake`	correct	54.0%	0.870	`flake` 54.0% `failing_test` 15.6% `bug` 12.6% `feature` 6.0% `cleanup` 5.5% `support` 3.8%
NanoJev	`flake`	correct	31.4%	0.712	`flake` 31.4% `failing_test` 22.3% `feature` 14.8% `bug` 14.8% `support` 9.2% `cleanup` 4.8%

Sample 6 of 6 · real-time-and-agents:k8s-129753 · 206 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "repo": "kubernetes/kubernetes",
  "title": "[Failing Test] ci-kubernetes-kind-network-dual: endpoints related",
  "body": "### Which jobs are failing? https://testgrid.k8s.io/sig-network-kind#sig-network-kind,%20dual,%20master ### Which tests are failing? * [It] [sig-network] [Feature:IPv6DualStack] Granular Checks: Services Secondary IP Family [LinuxOnly] should be able to handle large requests: httpChanges * [It] [sig-network] [Feature:IPv6DualStack] Granular Checks: Services Secondary IP Family [LinuxOnly] should be able to handle large requests: udpChanges * [It] [sig-network] [Feature:IPv6DualStack] Granular Checks: Services Secondary IP Family [LinuxOnly] should function for client IP based session affinity: http [LinuxOnly]Changes * [It] [sig-network] [Feature:IPv6DualStack] Granular Checks: Services …"
}

Question kind · choice · primary (ranked)

Read `repo`, `title` and `body`. Which kubernetes issue kind will maintainers assign to this issue?

7 options

bug — Something is broken or behaves contrary to documentation (kind/bug)
feature — Request for new functionality or an enhancement (kind/feature)
cleanup — Refactor, tech-debt removal, deprecation of old code, no user-facing change (kind/cleanup)
documentation — Docs are missing, wrong or unclear (kind/documentation)
support — A usage question or help request rather than a defect (kind/support)
failing_test — A test fails consistently in CI (kind/failing-test)
flake — A test fails intermittently / flaky CI job (kind/flake)

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: failing_test

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`failing_test`	correct	98.0%	1.000	`failing_test` 98.0% `flake` 2.0% `cleanup` 0.0% `bug` 0.0% `documentation` 0.0% `support` 0.0%
JevFish	`failing_test`	correct	50.8%	0.818	`failing_test` 50.8% `flake` 30.9% `bug` 16.5% `feature` 0.6% `support` 0.5% `documentation` 0.4%
decider-2b	`failing_test`	correct	64.0%	0.909	`failing_test` 64.0% `bug` 18.4% `flake` 13.8% `feature` 1.1% `cleanup` 1.1% `documentation` 0.9%
OpenJev	`failing_test`	correct	93.9%	0.997	`failing_test` 93.9% `flake` 5.8% `bug` 0.2% `documentation` 0.1% `feature` 0.1% `support` 0.0%
System One Scorer	`failing_test`	correct	29.2%	0.679	`failing_test` 29.2% `flake` 27.4% `bug` 23.3% `feature` 7.8% `cleanup` 5.9% `support` 4.5%
NanoJev	`failing_test`	correct	43.3%	0.805	`failing_test` 43.3% `bug` 17.8% `flake` 13.7% `feature` 10.4% `support` 6.2% `cleanup` 4.6%