trialgpt-criterion-support
citation verification · healthcare · 159 items · primary question type choice. Source ncbi/TrialGPT-Criterion-Annotations (Public domain (NCBI)); labels are expert labels, never an LLM judge. Download these samples (JSON).
| System | DecisionScore | Accuracy (top pick) | Answered |
|---|---|---|---|
| Jev | 73.8 | 64.1% | 159 / 159 |
| decider-2b | 73.3 | 57.9% | 159 / 159 |
| System One Scorer | 70.6 | 53.5% | 159 / 159 |
| JevFish | 70.5 | 53.5% | 159 / 159 |
| OpenJev | 60.6 | 53.5% | 159 / 159 |
| NanoJev | 58.7 | 12.6% | 159 / 159 |
retrieval-and-knowledge:trialgpt-205 · 134 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"patient_note": "0. A 25-year-old woman presents to the clinic complaining of prolonged fatigue.\n1. She denies difficulty sleeping and sleeps an average of 8 hours a night.\n2. She also notes hair loss, a change in her voice and weight gain during the previous 6 months.\n3. She complains of cold intolerance.\n4. On examination she has a prominent, soft, uniform anterior cervical mass at the midline.\n5. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
"criterion_type": "exclusion",
"criterion": "having evidence of end-stage thyroiditis"
}Question criterion_relation · choice · primary (ranked)
The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?
4 options
criterion_satisfied— The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).criterion_not_satisfied— The note gives evidence the patient does not meet this criterion (not included / not excluded).not_enough_information— The criterion is relevant to this patient but the note lacks the information needed to decide.not_applicable— The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: not_enough_information
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | criterion_not_satisfied | wrong | 35.0% | 0.667 |
|
| decider-2b | criterion_not_satisfied | wrong | 10.4% | 0.259 |
|
| System One Scorer | criterion_satisfied | wrong | 29.9% | 0.645 |
|
| JevFish | criterion_not_satisfied | wrong | 13.5% | 0.376 |
|
| OpenJev | not_enough_information | correct | 73.8% | 0.932 |
|
| NanoJev | criterion_satisfied | wrong | 22.8% | 0.584 |
|
retrieval-and-knowledge:trialgpt-609 · 132 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"patient_note": "0. A 62 yo male presents with four days of non-productive cough and one day of fever.\n1. He is on immunosuppressive medications, including prednisone.\n2. He is admitted to the hospital, and his work-up includes bronchoscopy with bronchoalveolar lavage (BAL).\n3. BAL fluid examination reveals owl's eye inclusion bodies in the nuclei of infection cells.\n4. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
"criterion_type": "inclusion",
"criterion": "Age 18-65"
}Question criterion_relation · choice · primary (ranked)
The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?
4 options
criterion_satisfied— The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).criterion_not_satisfied— The note gives evidence the patient does not meet this criterion (not included / not excluded).not_enough_information— The criterion is relevant to this patient but the note lacks the information needed to decide.not_applicable— The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: criterion_satisfied
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | criterion_satisfied | correct | 100.0% | 1.000 |
|
| decider-2b | criterion_satisfied | correct | 65.4% | 0.919 |
|
| System One Scorer | criterion_satisfied | correct | 66.6% | 0.916 |
|
| JevFish | criterion_satisfied | correct | 44.6% | 0.787 |
|
| OpenJev | criterion_satisfied | correct | 100.0% | 1.000 |
|
| NanoJev | criterion_satisfied | correct | 37.2% | 0.735 |
|
retrieval-and-knowledge:trialgpt-627 · 150 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"patient_note": "0. A 65 yo male with no significant history of cardiovascular disease presents to the emergency room with acute onset of shortness of breath, tachypnea, and left-sided chest pain that worsens with inspiration.\n1. Of note, he underwent a right total hip replacement two weeks prior to presentation and was unable to begin physical therapy and rehabilitation for several days following the surgery due to poor pain management.\n2. Relevant physical exam findings include a respiratory rate of 35 and right calf pain.\n3. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
"criterion_type": "inclusion",
"criterion": "Able to provide written informed consent."
}Question criterion_relation · choice · primary (ranked)
The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?
4 options
criterion_satisfied— The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).criterion_not_satisfied— The note gives evidence the patient does not meet this criterion (not included / not excluded).not_enough_information— The criterion is relevant to this patient but the note lacks the information needed to decide.not_applicable— The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: criterion_satisfied
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | criterion_satisfied | correct | 98.0% | 1.000 |
|
| decider-2b | criterion_satisfied | correct | 80.2% | 0.973 |
|
| System One Scorer | criterion_satisfied | correct | 61.2% | 0.868 |
|
| JevFish | criterion_satisfied | correct | 34.2% | 0.703 |
|
| OpenJev | criterion_satisfied | correct | 100.0% | 1.000 |
|
| NanoJev | criterion_satisfied | correct | 34.9% | 0.705 |
|
retrieval-and-knowledge:trialgpt-789 · 277 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"patient_note": "0. A 27-year-old woman at 11 weeks gestation in her second pregnancy is found to have a hemoglobin (Hb) of 9.0 g/dL, white blood cell count 6.3 x 109/L, platelet count 119 x 109/L, mean corpuscular volume 109 fL.\n1. Further investigations reveal mild iron deficiency.\n2. She already receives iron supplementation.\n3. The obstetrician repeats the complete blood cell count 2 weeks later.\n4. The Hb is 8.9 g/dL, WBC count 7.1 x 109/L, and platelets 108 x 109/L.\n5. She describes difficulty swallowing.\n6. A reticulocyte count is performed and found elevated at 180 x 109/L.\n7. The obstetrician requests a hematology consult.\n8. The following additional results were found: Negative DAT, normal clotting screen, elevated LDH (2000 IU/L), normal urea and electrolytes, normal alanine aminotransferase …",
"criterion_type": "exclusion",
"criterion": "Obvious mental retardation or metabolic disorder"
}Question criterion_relation · choice · primary (ranked)
The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?
4 options
criterion_satisfied— The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).criterion_not_satisfied— The note gives evidence the patient does not meet this criterion (not included / not excluded).not_enough_information— The criterion is relevant to this patient but the note lacks the information needed to decide.not_applicable— The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: criterion_not_satisfied
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | criterion_not_satisfied | correct | 49.0% | 0.759 |
|
| decider-2b | criterion_not_satisfied | correct | 88.0% | 0.990 |
|
| System One Scorer | criterion_not_satisfied | correct | 36.7% | 0.718 |
|
| JevFish | criterion_not_satisfied | correct | 70.1% | 0.938 |
|
| OpenJev | not_applicable | wrong | 0.1% | 0.050 |
|
| NanoJev | criterion_satisfied | wrong | 21.0% | 0.578 |
|
retrieval-and-knowledge:trialgpt-918 · 145 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"patient_note": "0. A 32-year-old male presents to your office complaining of diarrhea, abdominal cramping and flatulence.\n1. Stools are greasy and foul-smelling.\n2. He also has loss of appetite and malaise.\n3. He recently returned home from a hiking trip in the mountains where he drank water from natural sources.\n4. An iodine-stained stool smear revealed ellipsoidal cysts with smooth, well-defined walls and 2+ nuclei.\n5. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
"criterion_type": "inclusion",
"criterion": "Able to provide written, informed consent"
}Question criterion_relation · choice · primary (ranked)
The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?
4 options
criterion_satisfied— The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).criterion_not_satisfied— The note gives evidence the patient does not meet this criterion (not included / not excluded).not_enough_information— The criterion is relevant to this patient but the note lacks the information needed to decide.not_applicable— The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: criterion_satisfied
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | criterion_satisfied | correct | 99.0% | 1.000 |
|
| decider-2b | criterion_satisfied | correct | 79.9% | 0.973 |
|
| System One Scorer | criterion_satisfied | correct | 67.5% | 0.909 |
|
| JevFish | criterion_satisfied | correct | 38.2% | 0.734 |
|
| OpenJev | criterion_satisfied | correct | 100.0% | 1.000 |
|
| NanoJev | criterion_satisfied | correct | 45.3% | 0.797 |
|
retrieval-and-knowledge:trialgpt-950 · 141 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"patient_note": "0. A 65-year-old male with a history of tuberculosis has started to complain of productive cough with tinges of blood.\n1. Chest X-ray reveals a round opaque mass within a cavity in his left upper lobe.\n2. The spherical mass moved in the cavity during supine and prone CT imaging.\n3. Culture of the sputum revealed an organism with septated, low-angle branching hyphae that had straight, parallel walls.\n4. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
"criterion_type": "exclusion",
"criterion": "The patient has hepatic cirrhosis."
}Question criterion_relation · choice · primary (ranked)
The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?
4 options
criterion_satisfied— The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).criterion_not_satisfied— The note gives evidence the patient does not meet this criterion (not included / not excluded).not_enough_information— The criterion is relevant to this patient but the note lacks the information needed to decide.not_applicable— The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: not_applicable
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | not_enough_information | wrong | 0.0% | 0.082 |
|
| decider-2b | criterion_not_satisfied | wrong | 11.5% | 0.269 |
|
| System One Scorer | not_enough_information | wrong | 5.1% | 0.377 |
|
| JevFish | criterion_not_satisfied | wrong | 32.0% | 0.584 |
|
| OpenJev | not_applicable | correct | 73.1% | 0.938 |
|
| NanoJev | criterion_satisfied | wrong | 15.9% | 0.505 |
|