{
 "schemaVersion": "blobfish.typed-decision-bench-task.v1",
 "version": "0.3.0",
 "task": "k8s-issue-kind-routing",
 "family": "real-time-and-agents",
 "use_case": "agent-routing-and-skill-selection",
 "vertical": "devops",
 "dataset": "open-index/open-github-issues",
 "licence": "ODC-BY-1.0",
 "provenance": "outcome",
 "state_kind": "record",
 "primitive": "choice",
 "items": 200,
 "scores": {
  "decider-2b": {
   "decisionScore": 75.36,
   "accuracy": 0.66,
   "answered": 200,
   "items": 200
  },
  "Jev": {
   "decisionScore": 77.46,
   "accuracy": 0.725,
   "answered": 200,
   "items": 200
  },
  "JevFish": {
   "decisionScore": 73.62,
   "accuracy": 0.62,
   "answered": 200,
   "items": 200
  },
  "NanoJev": {
   "decisionScore": 61.47,
   "accuracy": 0.34,
   "answered": 200,
   "items": 200
  },
  "OpenJev": {
   "decisionScore": 73.84,
   "accuracy": 0.71,
   "answered": 200,
   "items": 200
  },
  "System One Scorer": {
   "decisionScore": 73.46,
   "accuracy": 0.61,
   "answered": 200,
   "items": 200
  }
 },
 "samples": [
  {
   "id": "real-time-and-agents:k8s-115483",
   "state": {
    "repo": "kubernetes/kubernetes",
    "title": "Secrets does not creating",
    "body": "### What happened? I am writing to report an issue with the latest version of Kubernetes in Azure, version v1.24.6. Before this version, when creating a Kubernetes cluster, the service account and secrets were automatically created and everything worked smoothly. However, with version v1.24.6, I have encountered an error when attempting to connect the cluster to Azure DevOps. Also during execution command in version v1.23.8 \"kubectl create serviceaccount some_name\", security token generating automatically. But not in v1.24.6. I found only 2 secrets in new cluster v1.24.6 comparing with v1.23.8 where was more than 30 secrets. I am checking all namespaces with command \"kubectl get secrets …"
   },
   "state_tokens": 181,
   "tier": "common",
   "primary": "kind",
   "questions": [
    {
     "id": "kind",
     "type": "choice",
     "instructions": "Read `repo`, `title` and `body`. Which kubernetes issue kind will maintainers assign to this issue?",
     "options": [
      [
       "bug",
       "Something is broken or behaves contrary to documentation (kind/bug)"
      ],
      [
       "feature",
       "Request for new functionality or an enhancement (kind/feature)"
      ],
      [
       "cleanup",
       "Refactor, tech-debt removal, deprecation of old code, no user-facing change (kind/cleanup)"
      ],
      [
       "documentation",
       "Docs are missing, wrong or unclear (kind/documentation)"
      ],
      [
       "support",
       "A usage question or help request rather than a defect (kind/support)"
      ],
      [
       "failing_test",
       "A test fails consistently in CI (kind/failing-test)"
      ],
      [
       "flake",
       "A test fails intermittently / flaky CI job (kind/flake)"
      ]
     ],
     "graded": true,
     "expected": "support",
     "responses": {
      "decider-2b": {
       "answered": true,
       "pick": "bug",
       "correct": false,
       "probabilities": {
        "bug": 0.7952,
        "support": 0.0791,
        "feature": 0.0423,
        "documentation": 0.0367,
        "failing_test": 0.0238,
        "cleanup": 0.017
       },
       "probability_of_expected": 0.0791,
       "proper_score": 0.2578
      },
      "Jev": {
       "answered": true,
       "pick": "bug",
       "correct": false,
       "probabilities": {
        "bug": 0.77,
        "support": 0.23,
        "cleanup": 0.0,
        "documentation": 0.0,
        "failing_test": 0.0,
        "flake": 0.0
       },
       "probability_of_expected": 0.23,
       "proper_score": 0.4071
      },
      "JevFish": {
       "answered": true,
       "pick": "bug",
       "correct": false,
       "probabilities": {
        "bug": 0.8175,
        "support": 0.0989,
        "feature": 0.0269,
        "cleanup": 0.0266,
        "documentation": 0.0205,
        "flake": 0.0072
       },
       "probability_of_expected": 0.0989,
       "proper_score": 0.2589
      },
      "NanoJev": {
       "answered": true,
       "pick": "failing_test",
       "correct": false,
       "probabilities": {
        "failing_test": 0.2893,
        "bug": 0.2334,
        "feature": 0.1839,
        "flake": 0.1416,
        "support": 0.0819,
        "cleanup": 0.0487
       },
       "probability_of_expected": 0.0819,
       "proper_score": 0.4811
      },
      "OpenJev": {
       "answered": true,
       "pick": "bug",
       "correct": false,
       "probabilities": {
        "bug": 0.9835,
        "feature": 0.0159,
        "flake": 0.0003,
        "cleanup": 0.0001,
        "support": 0.0001,
        "documentation": 0.0001
       },
       "probability_of_expected": 0.0001,
       "proper_score": 0.0163
      },
      "System One Scorer": {
       "answered": true,
       "pick": "support",
       "correct": true,
       "probabilities": {
        "support": 0.3865,
        "bug": 0.3685,
        "cleanup": 0.0844,
        "documentation": 0.0486,
        "failing_test": 0.0426,
        "flake": 0.0373
       },
       "probability_of_expected": 0.3865,
       "proper_score": 0.7371
      }
     }
    }
   ],
   "latency_ms": {
    "decider-2b": 2035.6,
    "Jev": 188.8,
    "JevFish": 1516.3,
    "NanoJev": 693.8,
    "OpenJev": 340.7,
    "System One Scorer": 3236.0
   }
  },
  {
   "id": "real-time-and-agents:k8s-115556",
   "state": {
    "repo": "kubernetes/kubernetes",
    "title": "[Flaky Test] Conformance - GCE - master - kubetest2",
    "body": "### Which jobs were failing? Conformance - GCE - master - kubetest2 ### Which tests failed? It looks like the overall build may be failing ### When has it been failing? From 1:35pm - 3:35pm ### Testgrid link https://k8s-testgrid.appspot.com/sig-release-master-blocking#Conformance%20-%20GCE%20-%20master%20-%20kubetest2 ### Reason for failure (if possible) It looks like build/release.sh script blew up: ``` DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --load -t kube-build:build-6454b08013-5-v1.27.0-go1.20-bullseye.0 --pull=false --build-arg=KUBE_CROSS_IMAGE=registry.k8s.io/build-image/kube-cross --build-arg=KUBE_CROSS_VERSION=v1.27.0-go1.20-bullseye.0 …"
   },
   "state_tokens": 253,
   "tier": "common",
   "primary": "kind",
   "questions": [
    {
     "id": "kind",
     "type": "choice",
     "instructions": "Read `repo`, `title` and `body`. Which kubernetes issue kind will maintainers assign to this issue?",
     "options": [
      [
       "bug",
       "Something is broken or behaves contrary to documentation (kind/bug)"
      ],
      [
       "feature",
       "Request for new functionality or an enhancement (kind/feature)"
      ],
      [
       "cleanup",
       "Refactor, tech-debt removal, deprecation of old code, no user-facing change (kind/cleanup)"
      ],
      [
       "documentation",
       "Docs are missing, wrong or unclear (kind/documentation)"
      ],
      [
       "support",
       "A usage question or help request rather than a defect (kind/support)"
      ],
      [
       "failing_test",
       "A test fails consistently in CI (kind/failing-test)"
      ],
      [
       "flake",
       "A test fails intermittently / flaky CI job (kind/flake)"
      ]
     ],
     "graded": true,
     "expected": "failing_test",
     "responses": {
      "decider-2b": {
       "answered": true,
       "pick": "flake",
       "correct": false,
       "probabilities": {
        "flake": 0.8786,
        "failing_test": 0.106,
        "bug": 0.0111,
        "cleanup": 0.0015,
        "feature": 0.001,
        "documentation": 0.0009
       },
       "probability_of_expected": 0.106,
       "proper_score": 0.2143
      },
      "Jev": {
       "answered": true,
       "pick": "flake",
       "correct": false,
       "probabilities": {
        "flake": 0.99,
        "failing_test": 0.01,
        "cleanup": 0.0,
        "support": 0.0,
        "documentation": 0.0,
        "feature": 0.0
       },
       "probability_of_expected": 0.01,
       "proper_score": 0.0199
      },
      "JevFish": {
       "answered": true,
       "pick": "flake",
       "correct": false,
       "probabilities": {
        "flake": 0.61,
        "failing_test": 0.2672,
        "cleanup": 0.0711,
        "bug": 0.0383,
        "support": 0.0065,
        "feature": 0.0054
       },
       "probability_of_expected": 0.2672,
       "proper_score": 0.5422
      },
      "NanoJev": {
       "answered": true,
       "pick": "flake",
       "correct": false,
       "probabilities": {
        "flake": 0.2852,
        "failing_test": 0.2558,
        "bug": 0.1696,
        "feature": 0.117,
        "support": 0.0701,
        "cleanup": 0.0536
       },
       "probability_of_expected": 0.2558,
       "proper_score": 0.6561
      },
      "OpenJev": {
       "answered": true,
       "pick": "flake",
       "correct": false,
       "probabilities": {
        "flake": 0.9956,
        "failing_test": 0.0024,
        "feature": 0.001,
        "bug": 0.0006,
        "documentation": 0.0003,
        "cleanup": 0.0001
       },
       "probability_of_expected": 0.0024,
       "proper_score": 0.0068
      },
      "System One Scorer": {
       "answered": true,
       "pick": "flake",
       "correct": false,
       "probabilities": {
        "flake": 0.577,
        "cleanup": 0.105,
        "bug": 0.0932,
        "failing_test": 0.0932,
        "support": 0.0686,
        "documentation": 0.0348
       },
       "probability_of_expected": 0.0932,
       "proper_score": 0.4091
      }
     }
    }
   ],
   "latency_ms": {
    "decider-2b": 2253.2,
    "Jev": 188.6,
    "JevFish": 1544.3,
    "NanoJev": 721.1,
    "OpenJev": 366.1,
    "System One Scorer": 3521.0
   }
  },
  {
   "id": "real-time-and-agents:k8s-122585",
   "state": {
    "repo": "kubernetes/kubernetes",
    "title": "kubelet creates two duplicate containers",
    "body": "### What happened? In our k8s cluster, I created a pod, but kubelet pulled up two containers，as shown below: ![pod](https://github.com/kubernetes/kubernetes/assets/62041640/be6c78cf-9241-4273-8896-588c84f04664) ![container](https://github.com/kubernetes/kubernetes/assets/62041640/628f4ad5-ca48-4cbf-afdb-de55eb40a63e) Judging from the kubelet log, a timeout occurred when kubelet created the first container.kubelet may think container creation failed, then kubelet created a second container. When the second container is created successfully, the first container is also created successfully, so container duplication occurs. ### What did you expect to happen? I think kubelet should not create …"
   },
   "state_tokens": 221,
   "tier": "common",
   "primary": "kind",
   "questions": [
    {
     "id": "kind",
     "type": "choice",
     "instructions": "Read `repo`, `title` and `body`. Which kubernetes issue kind will maintainers assign to this issue?",
     "options": [
      [
       "bug",
       "Something is broken or behaves contrary to documentation (kind/bug)"
      ],
      [
       "feature",
       "Request for new functionality or an enhancement (kind/feature)"
      ],
      [
       "cleanup",
       "Refactor, tech-debt removal, deprecation of old code, no user-facing change (kind/cleanup)"
      ],
      [
       "documentation",
       "Docs are missing, wrong or unclear (kind/documentation)"
      ],
      [
       "support",
       "A usage question or help request rather than a defect (kind/support)"
      ],
      [
       "failing_test",
       "A test fails consistently in CI (kind/failing-test)"
      ],
      [
       "flake",
       "A test fails intermittently / flaky CI job (kind/flake)"
      ]
     ],
     "graded": true,
     "expected": "bug",
     "responses": {
      "decider-2b": {
       "answered": true,
       "pick": "bug",
       "correct": true,
       "probabilities": {
        "bug": 0.8194,
        "support": 0.0416,
        "failing_test": 0.0396,
        "feature": 0.0327,
        "documentation": 0.0297,
        "cleanup": 0.0245
       },
       "probability_of_expected": 0.8194,
       "proper_score": 0.9807
      },
      "Jev": {
       "answered": true,
       "pick": "bug",
       "correct": true,
       "probabilities": {
        "bug": 1.0,
        "documentation": 0.0,
        "feature": 0.0,
        "cleanup": 0.0,
        "support": 0.0,
        "failing_test": 0.0
       },
       "probability_of_expected": 1.0,
       "proper_score": 1.0
      },
      "JevFish": {
       "answered": true,
       "pick": "bug",
       "correct": true,
       "probabilities": {
        "bug": 0.8507,
        "cleanup": 0.0441,
        "support": 0.0398,
        "feature": 0.0298,
        "failing_test": 0.0153,
        "documentation": 0.0104
       },
       "probability_of_expected": 0.8507,
       "proper_score": 0.9864
      },
      "NanoJev": {
       "answered": true,
       "pick": "feature",
       "correct": false,
       "probabilities": {
        "feature": 0.2106,
        "failing_test": 0.197,
        "bug": 0.1621,
        "flake": 0.1307,
        "support": 0.1277,
        "cleanup": 0.1059
       },
       "probability_of_expected": 0.1621,
       "proper_score": 0.5829
      },
      "OpenJev": {
       "answered": true,
       "pick": "bug",
       "correct": true,
       "probabilities": {
        "bug": 0.9911,
        "feature": 0.0087,
        "documentation": 0.0001,
        "flake": 0.0001,
        "support": 0.0,
        "cleanup": 0.0
       },
       "probability_of_expected": 0.9911,
       "proper_score": 0.9999
      },
      "System One Scorer": {
       "answered": true,
       "pick": "bug",
       "correct": true,
       "probabilities": {
        "bug": 0.6629,
        "support": 0.1083,
        "cleanup": 0.0923,
        "flake": 0.0561,
        "documentation": 0.0321,
        "feature": 0.0263
       },
       "probability_of_expected": 0.6629,
       "proper_score": 0.9304
      }
     }
    }
   ],
   "latency_ms": {
    "decider-2b": 2266.0,
    "Jev": 197.3,
    "JevFish": 1514.9,
    "NanoJev": 655.6,
    "OpenJev": 338.8,
    "System One Scorer": 2876.2
   }
  },
  {
   "id": "real-time-and-agents:k8s-126119",
   "state": {
    "repo": "kubernetes/kubernetes",
    "title": "Do not start cadvisor when feature PodAndContainerStatsFromCRI is enabled",
    "body": "### What would you like to be added? Remove the need to start cadvisor, which has a periodic housekeeping task. That leaves on-demand invocations to cadvisor APIs ### Why is this needed? Cadvisor housekeeping task is a visible overhead that collects host OS stats, which can be entirely replaced by CRI APIs. The only use left for having housekeeping task is to [collect rootfs stats](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go#L151), which can be changed to an on-demand call. Also see prior issues that attempted to manage the overhead of cadvisor: https://github.com/kubernetes/kubernetes/pull/124520 …"
   },
   "state_tokens": 176,
   "tier": "common",
   "primary": "kind",
   "questions": [
    {
     "id": "kind",
     "type": "choice",
     "instructions": "Read `repo`, `title` and `body`. Which kubernetes issue kind will maintainers assign to this issue?",
     "options": [
      [
       "bug",
       "Something is broken or behaves contrary to documentation (kind/bug)"
      ],
      [
       "feature",
       "Request for new functionality or an enhancement (kind/feature)"
      ],
      [
       "cleanup",
       "Refactor, tech-debt removal, deprecation of old code, no user-facing change (kind/cleanup)"
      ],
      [
       "documentation",
       "Docs are missing, wrong or unclear (kind/documentation)"
      ],
      [
       "support",
       "A usage question or help request rather than a defect (kind/support)"
      ],
      [
       "failing_test",
       "A test fails consistently in CI (kind/failing-test)"
      ],
      [
       "flake",
       "A test fails intermittently / flaky CI job (kind/flake)"
      ]
     ],
     "graded": true,
     "expected": "feature",
     "responses": {
      "decider-2b": {
       "answered": true,
       "pick": "cleanup",
       "correct": false,
       "probabilities": {
        "cleanup": 0.8207,
        "feature": 0.109,
        "documentation": 0.0258,
        "bug": 0.0193,
        "failing_test": 0.0125,
        "support": 0.0077
       },
       "probability_of_expected": 0.109,
       "proper_score": 0.2656
      },
      "Jev": {
       "answered": true,
       "pick": "feature",
       "correct": true,
       "probabilities": {
        "feature": 0.87,
        "cleanup": 0.12,
        "bug": 0.01,
        "support": 0.0,
        "documentation": 0.0,
        "failing_test": 0.0
       },
       "probability_of_expected": 0.87,
       "proper_score": 0.9843
      },
      "JevFish": {
       "answered": true,
       "pick": "cleanup",
       "correct": false,
       "probabilities": {
        "cleanup": 0.4516,
        "feature": 0.4506,
        "support": 0.0379,
        "bug": 0.0364,
        "documentation": 0.0176,
        "flake": 0.0053
       },
       "probability_of_expected": 0.4506,
       "proper_score": 0.7455
      },
      "NanoJev": {
       "answered": true,
       "pick": "feature",
       "correct": true,
       "probabilities": {
        "feature": 0.3317,
        "failing_test": 0.2513,
        "flake": 0.1335,
        "bug": 0.1304,
        "support": 0.0832,
        "cleanup": 0.0515
       },
       "probability_of_expected": 0.3317,
       "proper_score": 0.7227
      },
      "OpenJev": {
       "answered": true,
       "pick": "feature",
       "correct": true,
       "probabilities": {
        "feature": 0.9984,
        "bug": 0.0006,
        "flake": 0.0005,
        "support": 0.0003,
        "documentation": 0.0001,
        "cleanup": 0.0
       },
       "probability_of_expected": 0.9984,
       "proper_score": 1.0
      },
      "System One Scorer": {
       "answered": true,
       "pick": "cleanup",
       "correct": false,
       "probabilities": {
        "cleanup": 0.5772,
        "feature": 0.1969,
        "bug": 0.0756,
        "support": 0.0587,
        "documentation": 0.035,
        "flake": 0.035
       },
       "probability_of_expected": 0.1969,
       "proper_score": 0.5049
      }
     }
    }
   ],
   "latency_ms": {
    "decider-2b": 2027.2,
    "Jev": 197.4,
    "JevFish": 1737.4,
    "NanoJev": 644.7,
    "OpenJev": 370.2,
    "System One Scorer": 2980.4
   }
  },
  {
   "id": "real-time-and-agents:k8s-129759",
   "state": {
    "repo": "kubernetes/kubernetes",
    "title": "Documentation of pod selection for node-pressure eviction is confusing regarding QoS",
    "body": "In [the Node-pressure doc explaining the ranking of pods for eviction](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#pod-selection-for-kubelet-eviction), there is a note saying that QoS are not used for this ranking (which appears to be correct, AFAICT from the code). However, this section still mentions QoS extensively, and it reads a bit ambiguous. In particular, this section: >As a result, kubelet ranks and evicts pods in the following order: > > 1. BestEffort or Burstable pods where the usage exceeds requests. These pods are evicted based on their Priority and then by how much their usage level exceeds the request. > 2. Guaranteed pods and Burstable …"
   },
   "state_tokens": 183,
   "tier": "common",
   "primary": "kind",
   "questions": [
    {
     "id": "kind",
     "type": "choice",
     "instructions": "Read `repo`, `title` and `body`. Which kubernetes issue kind will maintainers assign to this issue?",
     "options": [
      [
       "bug",
       "Something is broken or behaves contrary to documentation (kind/bug)"
      ],
      [
       "feature",
       "Request for new functionality or an enhancement (kind/feature)"
      ],
      [
       "cleanup",
       "Refactor, tech-debt removal, deprecation of old code, no user-facing change (kind/cleanup)"
      ],
      [
       "documentation",
       "Docs are missing, wrong or unclear (kind/documentation)"
      ],
      [
       "support",
       "A usage question or help request rather than a defect (kind/support)"
      ],
      [
       "failing_test",
       "A test fails consistently in CI (kind/failing-test)"
      ],
      [
       "flake",
       "A test fails intermittently / flaky CI job (kind/flake)"
      ]
     ],
     "graded": true,
     "expected": "documentation",
     "responses": {
      "decider-2b": {
       "answered": true,
       "pick": "documentation",
       "correct": true,
       "probabilities": {
        "documentation": 0.8754,
        "bug": 0.0959,
        "cleanup": 0.0154,
        "feature": 0.0051,
        "support": 0.0033,
        "failing_test": 0.003
       },
       "probability_of_expected": 0.8754,
       "proper_score": 0.9875
      },
      "Jev": {
       "answered": true,
       "pick": "documentation",
       "correct": true,
       "probabilities": {
        "documentation": 1.0,
        "flake": 0.0,
        "cleanup": 0.0,
        "support": 0.0,
        "failing_test": 0.0,
        "feature": 0.0
       },
       "probability_of_expected": 1.0,
       "proper_score": 1.0
      },
      "JevFish": {
       "answered": true,
       "pick": "documentation",
       "correct": true,
       "probabilities": {
        "documentation": 0.7858,
        "bug": 0.1808,
        "cleanup": 0.0159,
        "support": 0.011,
        "failing_test": 0.0026,
        "feature": 0.0026
       },
       "probability_of_expected": 0.7858,
       "proper_score": 0.9605
      },
      "NanoJev": {
       "answered": true,
       "pick": "documentation",
       "correct": true,
       "probabilities": {
        "documentation": 0.4926,
        "failing_test": 0.1231,
        "support": 0.1203,
        "feature": 0.0866,
        "bug": 0.0795,
        "flake": 0.068
       },
       "probability_of_expected": 0.4926,
       "proper_score": 0.8468
      },
      "OpenJev": {
       "answered": true,
       "pick": "documentation",
       "correct": true,
       "probabilities": {
        "documentation": 0.9962,
        "bug": 0.0011,
        "support": 0.001,
        "feature": 0.0009,
        "flake": 0.0007,
        "cleanup": 0.0001
       },
       "probability_of_expected": 0.9962,
       "proper_score": 1.0
      },
      "System One Scorer": {
       "answered": true,
       "pick": "documentation",
       "correct": true,
       "probabilities": {
        "documentation": 0.5931,
        "bug": 0.1277,
        "support": 0.1219,
        "cleanup": 0.1114,
        "feature": 0.0197,
        "flake": 0.0168
       },
       "probability_of_expected": 0.5931,
       "proper_score": 0.895
      }
     }
    }
   ],
   "latency_ms": {
    "decider-2b": 2027.4,
    "Jev": 164.7,
    "JevFish": 1617.2,
    "NanoJev": 628.1,
    "OpenJev": 374.0,
    "System One Scorer": 3003.1
   }
  },
  {
   "id": "real-time-and-agents:k8s-129802",
   "state": {
    "repo": "kubernetes/kubernetes",
    "title": "[Flaking Test] [sig-api-machinery] k8s.io/kubernetes/test/integration/apiserver/coordinatedleaderelection.coordinatedleaderelection",
    "body": "### Which jobs are flaking? master-blocking - integration-master ### Which tests are flaking? k8s.io/kubernetes/test/integration/apiserver/coordinatedleaderelection.coordinatedleaderelection [Prow](https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-integration-master/1881787128904421376) [Triage](https://storage.googleapis.com/k8s-triage/index.html?test=k8s.io%2Fkubernetes%2Ftest%2Fintegration%2Fapiserver%2Fcoordinatedleaderelection.coordinatedleaderelection&xjob=e2e-kops) ### Since when has it been flaking? [1/14/2025, 12:41:26 AM](https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-integration-master/1878882224963588096) [1/18/2025, 1:29:23 …"
   },
   "state_tokens": 288,
   "tier": "common",
   "primary": "kind",
   "questions": [
    {
     "id": "kind",
     "type": "choice",
     "instructions": "Read `repo`, `title` and `body`. Which kubernetes issue kind will maintainers assign to this issue?",
     "options": [
      [
       "bug",
       "Something is broken or behaves contrary to documentation (kind/bug)"
      ],
      [
       "feature",
       "Request for new functionality or an enhancement (kind/feature)"
      ],
      [
       "cleanup",
       "Refactor, tech-debt removal, deprecation of old code, no user-facing change (kind/cleanup)"
      ],
      [
       "documentation",
       "Docs are missing, wrong or unclear (kind/documentation)"
      ],
      [
       "support",
       "A usage question or help request rather than a defect (kind/support)"
      ],
      [
       "failing_test",
       "A test fails consistently in CI (kind/failing-test)"
      ],
      [
       "flake",
       "A test fails intermittently / flaky CI job (kind/flake)"
      ]
     ],
     "graded": true,
     "expected": "flake",
     "responses": {
      "decider-2b": {
       "answered": true,
       "pick": "flake",
       "correct": true,
       "probabilities": {
        "flake": 0.8786,
        "failing_test": 0.0962,
        "bug": 0.0179,
        "cleanup": 0.003,
        "feature": 0.0016,
        "documentation": 0.0015
       },
       "probability_of_expected": 0.8786,
       "proper_score": 0.9878
      },
      "Jev": {
       "answered": true,
       "pick": "flake",
       "correct": true,
       "probabilities": {
        "flake": 1.0,
        "feature": 0.0,
        "cleanup": 0.0,
        "support": 0.0,
        "documentation": 0.0,
        "failing_test": 0.0
       },
       "probability_of_expected": 1.0,
       "proper_score": 1.0
      },
      "JevFish": {
       "answered": true,
       "pick": "flake",
       "correct": true,
       "probabilities": {
        "flake": 0.6783,
        "failing_test": 0.2296,
        "cleanup": 0.0482,
        "bug": 0.0213,
        "feature": 0.0121,
        "support": 0.0086
       },
       "probability_of_expected": 0.6783,
       "proper_score": 0.9204
      },
      "NanoJev": {
       "answered": true,
       "pick": "flake",
       "correct": true,
       "probabilities": {
        "flake": 0.2586,
        "failing_test": 0.2094,
        "feature": 0.165,
        "bug": 0.1526,
        "support": 0.1029,
        "cleanup": 0.0767
       },
       "probability_of_expected": 0.2586,
       "proper_score": 0.6691
      },
      "OpenJev": {
       "answered": true,
       "pick": "flake",
       "correct": true,
       "probabilities": {
        "flake": 0.9988,
        "failing_test": 0.0009,
        "bug": 0.0002,
        "feature": 0.0001,
        "documentation": 0.0,
        "cleanup": 0.0
       },
       "probability_of_expected": 0.9988,
       "proper_score": 1.0
      },
      "System One Scorer": {
       "answered": true,
       "pick": "flake",
       "correct": true,
       "probabilities": {
        "flake": 0.7763,
        "failing_test": 0.0762,
        "bug": 0.054,
        "cleanup": 0.0442,
        "support": 0.0224,
        "documentation": 0.0137
       },
       "probability_of_expected": 0.7763,
       "proper_score": 0.9692
      }
     }
    }
   ],
   "latency_ms": {
    "decider-2b": 2442.7,
    "Jev": 154.6,
    "JevFish": 1617.1,
    "NanoJev": 650.4,
    "OpenJev": 412.3,
    "System One Scorer": 3162.3
   }
  }
 ]
}