Skip to samples
Typed Decision Bench · tasks · Retrieval and knowledge

trialgpt-criterion-support

citation verification · healthcare · 159 items · primary question type choice. Source ncbi/TrialGPT-Criterion-Annotations (Public domain (NCBI)); labels are expert labels, never an LLM judge. Download these samples (JSON).

SystemDecisionScoreAccuracy (top pick)Answered
Jev73.864.1%159 / 159
decider-2b73.357.9%159 / 159
System One Scorer70.653.5%159 / 159
JevFish70.553.5%159 / 159
OpenJev60.653.5%159 / 159
NanoJev58.712.6%159 / 159
Sample 1 of 6 · retrieval-and-knowledge:trialgpt-205 · 134 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "patient_note": "0. A 25-year-old woman presents to the clinic complaining of prolonged fatigue.\n1. She denies difficulty sleeping and sleeps an average of 8 hours a night.\n2. She also notes hair loss, a change in her voice and weight gain during the previous 6 months.\n3. She complains of cold intolerance.\n4. On examination she has a prominent, soft, uniform anterior cervical mass at the midline.\n5. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
  "criterion_type": "exclusion",
  "criterion": "having evidence of end-stage thyroiditis"
}

Question criterion_relation · choice · primary (ranked)

The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?

4 options
  • criterion_satisfied — The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).
  • criterion_not_satisfied — The note gives evidence the patient does not meet this criterion (not included / not excluded).
  • not_enough_information — The criterion is relevant to this patient but the note lacks the information needed to decide.
  • not_applicable — The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: not_enough_information

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevcriterion_not_satisfiedwrong35.0%0.667
  • criterion_not_satisfied 45.0%
  • not_enough_information 35.0%
  • criterion_satisfied 20.0%
  • not_applicable 0.0%
decider-2bcriterion_not_satisfiedwrong10.4%0.259
  • criterion_not_satisfied 82.2%
  • not_enough_information 10.4%
  • not_applicable 6.4%
  • criterion_satisfied 1.0%
System One Scorercriterion_satisfiedwrong29.9%0.645
  • criterion_satisfied 34.5%
  • criterion_not_satisfied 31.2%
  • not_enough_information 29.9%
  • not_applicable 4.3%
JevFishcriterion_not_satisfiedwrong13.5%0.376
  • criterion_not_satisfied 69.1%
  • not_applicable 14.2%
  • not_enough_information 13.5%
  • criterion_satisfied 3.1%
OpenJevnot_enough_informationcorrect73.8%0.932
  • not_enough_information 73.8%
  • criterion_not_satisfied 25.9%
  • criterion_satisfied 0.3%
  • not_applicable 0.1%
NanoJevcriterion_satisfiedwrong22.8%0.584
  • criterion_satisfied 41.6%
  • not_enough_information 22.8%
  • criterion_not_satisfied 18.7%
  • not_applicable 16.9%
Sample 2 of 6 · retrieval-and-knowledge:trialgpt-609 · 132 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "patient_note": "0. A 62 yo male presents with four days of non-productive cough and one day of fever.\n1. He is on immunosuppressive medications, including prednisone.\n2. He is admitted to the hospital, and his work-up includes bronchoscopy with bronchoalveolar lavage (BAL).\n3. BAL fluid examination reveals owl's eye inclusion bodies in the nuclei of infection cells.\n4. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
  "criterion_type": "inclusion",
  "criterion": "Age 18-65"
}

Question criterion_relation · choice · primary (ranked)

The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?

4 options
  • criterion_satisfied — The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).
  • criterion_not_satisfied — The note gives evidence the patient does not meet this criterion (not included / not excluded).
  • not_enough_information — The criterion is relevant to this patient but the note lacks the information needed to decide.
  • not_applicable — The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: criterion_satisfied

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevcriterion_satisfiedcorrect100.0%1.000
  • criterion_satisfied 100.0%
  • not_applicable 0.0%
  • criterion_not_satisfied 0.0%
  • not_enough_information 0.0%
decider-2bcriterion_satisfiedcorrect65.4%0.919
  • criterion_satisfied 65.4%
  • criterion_not_satisfied 15.5%
  • not_enough_information 9.6%
  • not_applicable 9.6%
System One Scorercriterion_satisfiedcorrect66.6%0.916
  • criterion_satisfied 66.6%
  • not_enough_information 22.2%
  • criterion_not_satisfied 7.0%
  • not_applicable 4.2%
JevFishcriterion_satisfiedcorrect44.6%0.787
  • criterion_satisfied 44.6%
  • criterion_not_satisfied 26.3%
  • not_applicable 21.0%
  • not_enough_information 8.0%
OpenJevcriterion_satisfiedcorrect100.0%1.000
  • criterion_satisfied 100.0%
  • criterion_not_satisfied 0.0%
  • not_applicable 0.0%
  • not_enough_information 0.0%
NanoJevcriterion_satisfiedcorrect37.2%0.735
  • criterion_satisfied 37.2%
  • not_enough_information 25.6%
  • not_applicable 19.9%
  • criterion_not_satisfied 17.4%
Sample 3 of 6 · retrieval-and-knowledge:trialgpt-627 · 150 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "patient_note": "0. A 65 yo male with no significant history of cardiovascular disease presents to the emergency room with acute onset of shortness of breath, tachypnea, and left-sided chest pain that worsens with inspiration.\n1. Of note, he underwent a right total hip replacement two weeks prior to presentation and  was unable to begin physical therapy and rehabilitation for several days following the surgery due to poor pain management.\n2. Relevant physical exam findings include a respiratory rate of 35 and right calf pain.\n3. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
  "criterion_type": "inclusion",
  "criterion": "Able to provide written informed consent."
}

Question criterion_relation · choice · primary (ranked)

The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?

4 options
  • criterion_satisfied — The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).
  • criterion_not_satisfied — The note gives evidence the patient does not meet this criterion (not included / not excluded).
  • not_enough_information — The criterion is relevant to this patient but the note lacks the information needed to decide.
  • not_applicable — The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: criterion_satisfied

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevcriterion_satisfiedcorrect98.0%1.000
  • criterion_satisfied 98.0%
  • not_enough_information 2.0%
  • criterion_not_satisfied 0.0%
  • not_applicable 0.0%
decider-2bcriterion_satisfiedcorrect80.2%0.973
  • criterion_satisfied 80.2%
  • not_enough_information 9.2%
  • not_applicable 5.4%
  • criterion_not_satisfied 5.2%
System One Scorercriterion_satisfiedcorrect61.2%0.868
  • criterion_satisfied 61.2%
  • not_enough_information 33.4%
  • criterion_not_satisfied 3.5%
  • not_applicable 2.0%
JevFishcriterion_satisfiedcorrect34.2%0.703
  • criterion_satisfied 34.2%
  • criterion_not_satisfied 29.9%
  • not_applicable 24.2%
  • not_enough_information 11.7%
OpenJevcriterion_satisfiedcorrect100.0%1.000
  • criterion_satisfied 100.0%
  • criterion_not_satisfied 0.0%
  • not_applicable 0.0%
  • not_enough_information 0.0%
NanoJevcriterion_satisfiedcorrect34.9%0.705
  • criterion_satisfied 34.9%
  • not_enough_information 34.4%
  • not_applicable 15.8%
  • criterion_not_satisfied 14.9%
Sample 4 of 6 · retrieval-and-knowledge:trialgpt-789 · 277 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "patient_note": "0. A 27-year-old woman at 11 weeks gestation in her second pregnancy is found to have a hemoglobin (Hb) of 9.0 g/dL, white blood cell count 6.3 x 109/L, platelet count 119 x 109/L, mean corpuscular volume 109 fL.\n1. Further investigations reveal mild iron deficiency.\n2. She already receives iron supplementation.\n3. The obstetrician repeats the complete blood cell count 2 weeks later.\n4. The Hb is 8.9 g/dL, WBC count 7.1 x 109/L, and platelets 108 x 109/L.\n5. She describes difficulty swallowing.\n6. A reticulocyte count is performed and found elevated at 180 x 109/L.\n7. The obstetrician requests a hematology consult.\n8. The following additional results were found: Negative DAT, normal clotting screen, elevated LDH (2000 IU/L), normal urea and electrolytes, normal alanine aminotransferase …",
  "criterion_type": "exclusion",
  "criterion": "Obvious mental retardation or metabolic disorder"
}

Question criterion_relation · choice · primary (ranked)

The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?

4 options
  • criterion_satisfied — The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).
  • criterion_not_satisfied — The note gives evidence the patient does not meet this criterion (not included / not excluded).
  • not_enough_information — The criterion is relevant to this patient but the note lacks the information needed to decide.
  • not_applicable — The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: criterion_not_satisfied

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevcriterion_not_satisfiedcorrect49.0%0.759
  • criterion_not_satisfied 49.0%
  • not_enough_information 47.0%
  • not_applicable 4.0%
  • criterion_satisfied 0.0%
decider-2bcriterion_not_satisfiedcorrect88.0%0.990
  • criterion_not_satisfied 88.0%
  • not_enough_information 6.0%
  • criterion_satisfied 3.0%
  • not_applicable 3.0%
System One Scorercriterion_not_satisfiedcorrect36.7%0.718
  • criterion_not_satisfied 36.7%
  • not_enough_information 30.2%
  • criterion_satisfied 25.6%
  • not_applicable 7.5%
JevFishcriterion_not_satisfiedcorrect70.1%0.938
  • criterion_not_satisfied 70.1%
  • not_applicable 12.8%
  • not_enough_information 12.6%
  • criterion_satisfied 4.5%
OpenJevnot_applicablewrong0.1%0.050
  • not_applicable 94.9%
  • not_enough_information 4.9%
  • criterion_not_satisfied 0.1%
  • criterion_satisfied 0.1%
NanoJevcriterion_satisfiedwrong21.0%0.578
  • criterion_satisfied 35.0%
  • not_enough_information 25.6%
  • criterion_not_satisfied 21.0%
  • not_applicable 18.4%
Sample 5 of 6 · retrieval-and-knowledge:trialgpt-918 · 145 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "patient_note": "0. A 32-year-old male presents to your office complaining of diarrhea, abdominal cramping and flatulence.\n1. Stools are greasy and foul-smelling.\n2. He also has loss of appetite and malaise.\n3. He recently returned home from a hiking trip in the mountains where he drank water from natural sources.\n4. An iodine-stained stool smear revealed ellipsoidal cysts with smooth, well-defined walls and 2+ nuclei.\n5. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
  "criterion_type": "inclusion",
  "criterion": "Able to provide written, informed consent"
}

Question criterion_relation · choice · primary (ranked)

The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?

4 options
  • criterion_satisfied — The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).
  • criterion_not_satisfied — The note gives evidence the patient does not meet this criterion (not included / not excluded).
  • not_enough_information — The criterion is relevant to this patient but the note lacks the information needed to decide.
  • not_applicable — The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: criterion_satisfied

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevcriterion_satisfiedcorrect99.0%1.000
  • criterion_satisfied 99.0%
  • not_enough_information 1.0%
  • not_applicable 0.0%
  • criterion_not_satisfied 0.0%
decider-2bcriterion_satisfiedcorrect79.9%0.973
  • criterion_satisfied 79.9%
  • criterion_not_satisfied 7.6%
  • not_enough_information 7.6%
  • not_applicable 4.9%
System One Scorercriterion_satisfiedcorrect67.5%0.909
  • criterion_satisfied 67.5%
  • not_enough_information 27.1%
  • criterion_not_satisfied 3.3%
  • not_applicable 2.1%
JevFishcriterion_satisfiedcorrect38.2%0.734
  • criterion_satisfied 38.2%
  • criterion_not_satisfied 31.9%
  • not_applicable 18.4%
  • not_enough_information 11.5%
OpenJevcriterion_satisfiedcorrect100.0%1.000
  • criterion_satisfied 100.0%
  • criterion_not_satisfied 0.0%
  • not_applicable 0.0%
  • not_enough_information 0.0%
NanoJevcriterion_satisfiedcorrect45.3%0.797
  • criterion_satisfied 45.3%
  • not_enough_information 25.1%
  • criterion_not_satisfied 15.0%
  • not_applicable 14.6%
Sample 6 of 6 · retrieval-and-knowledge:trialgpt-950 · 141 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "patient_note": "0. A 65-year-old male with a history of tuberculosis has started to complain of productive cough with tinges of blood.\n1. Chest X-ray reveals a round opaque mass within a cavity in his left upper lobe.\n2. The spherical mass moved in the cavity during supine and prone CT imaging.\n3. Culture of the sputum revealed an organism with septated, low-angle branching hyphae that had straight, parallel walls.\n4. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
  "criterion_type": "exclusion",
  "criterion": "The patient has hepatic cirrhosis."
}

Question criterion_relation · choice · primary (ranked)

The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?

4 options
  • criterion_satisfied — The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).
  • criterion_not_satisfied — The note gives evidence the patient does not meet this criterion (not included / not excluded).
  • not_enough_information — The criterion is relevant to this patient but the note lacks the information needed to decide.
  • not_applicable — The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: not_applicable

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnot_enough_informationwrong0.0%0.082
  • not_enough_information 91.0%
  • criterion_not_satisfied 9.0%
  • criterion_satisfied 0.0%
  • not_applicable 0.0%
decider-2bcriterion_not_satisfiedwrong11.5%0.269
  • criterion_not_satisfied 82.2%
  • not_applicable 11.5%
  • not_enough_information 4.8%
  • criterion_satisfied 1.5%
System One Scorernot_enough_informationwrong5.1%0.377
  • not_enough_information 43.1%
  • criterion_not_satisfied 37.5%
  • criterion_satisfied 14.3%
  • not_applicable 5.1%
JevFishcriterion_not_satisfiedwrong32.0%0.584
  • criterion_not_satisfied 60.6%
  • not_applicable 32.0%
  • not_enough_information 5.6%
  • criterion_satisfied 1.8%
OpenJevnot_applicablecorrect73.1%0.938
  • not_applicable 73.1%
  • not_enough_information 22.2%
  • criterion_not_satisfied 4.6%
  • criterion_satisfied 0.0%
NanoJevcriterion_satisfiedwrong15.9%0.505
  • criterion_satisfied 45.5%
  • not_enough_information 21.8%
  • criterion_not_satisfied 16.8%
  • not_applicable 15.9%