Skip to samples
Typed Decision Bench · tasks · Retrieval and knowledge

trialgpt-criterion-support

citation verification · healthcare · 159 items · primary question type choice. Source ncbi/TrialGPT-Criterion-Annotations (Public domain (NCBI)); labels are expert labels, never an LLM judge. Download these samples (JSON).

SystemDecisionScoreAccuracy (top pick)Answered
Jev73.864.1%159 / 159
JevFish73.561.6%159 / 159
decider-2b73.357.9%159 / 159
System One Scorer70.653.5%159 / 159
OpenJev60.653.5%159 / 159
NanoJev58.712.6%159 / 159
Sample 1 of 6 · retrieval-and-knowledge:trialgpt-205 · 134 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "patient_note": "0. A 25-year-old woman presents to the clinic complaining of prolonged fatigue.\n1. She denies difficulty sleeping and sleeps an average of 8 hours a night.\n2. She also notes hair loss, a change in her voice and weight gain during the previous 6 months.\n3. She complains of cold intolerance.\n4. On examination she has a prominent, soft, uniform anterior cervical mass at the midline.\n5. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
  "criterion_type": "exclusion",
  "criterion": "having evidence of end-stage thyroiditis"
}

Question criterion_relation · choice · primary (ranked)

The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?

4 options
  • criterion_satisfied — The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).
  • criterion_not_satisfied — The note gives evidence the patient does not meet this criterion (not included / not excluded).
  • not_enough_information — The criterion is relevant to this patient but the note lacks the information needed to decide.
  • not_applicable — The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: not_enough_information

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevcriterion_not_satisfiedwrong35.0%0.667
  • criterion_not_satisfied 45.0%
  • not_enough_information 35.0%
  • criterion_satisfied 20.0%
  • not_applicable 0.0%
JevFishcriterion_not_satisfiedwrong14.0%0.356
  • criterion_not_satisfied 73.0%
  • not_enough_information 14.0%
  • not_applicable 11.8%
  • criterion_satisfied 1.2%
decider-2bcriterion_not_satisfiedwrong10.4%0.259
  • criterion_not_satisfied 82.2%
  • not_enough_information 10.4%
  • not_applicable 6.4%
  • criterion_satisfied 1.0%
System One Scorercriterion_satisfiedwrong29.9%0.645
  • criterion_satisfied 34.5%
  • criterion_not_satisfied 31.2%
  • not_enough_information 29.9%
  • not_applicable 4.3%
OpenJevnot_enough_informationcorrect73.8%0.932
  • not_enough_information 73.8%
  • criterion_not_satisfied 25.9%
  • criterion_satisfied 0.3%
  • not_applicable 0.1%
NanoJevcriterion_satisfiedwrong22.8%0.584
  • criterion_satisfied 41.6%
  • not_enough_information 22.8%
  • criterion_not_satisfied 18.7%
  • not_applicable 16.9%
Sample 2 of 6 · retrieval-and-knowledge:trialgpt-413 · 167 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "patient_note": "0. A 21-year-old female is evaluated for progressive arthralgias and malaise.\n1. On examination she is found to have alopecia, a rash mainly distributed on the bridge of her nose and her cheeks, a delicate non-palpable purpura on her calves, and swelling and tenderness of her wrists and ankles.\n2. Her lab shows normocytic anemia, thrombocytopenia, a 4/4 positive ANA and anti-dsDNA.\n3. Her urine is positive for protein and RBC casts.\n4. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
  "criterion_type": "inclusion",
  "criterion": "Patients with systemic Lupus erythematosus"
}

Question criterion_relation · choice · primary (ranked)

The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?

4 options
  • criterion_satisfied — The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).
  • criterion_not_satisfied — The note gives evidence the patient does not meet this criterion (not included / not excluded).
  • not_enough_information — The criterion is relevant to this patient but the note lacks the information needed to decide.
  • not_applicable — The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: criterion_satisfied

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevcriterion_satisfiedcorrect100.0%1.000
  • criterion_satisfied 100.0%
  • criterion_not_satisfied 0.0%
  • not_enough_information 0.0%
  • not_applicable 0.0%
JevFishcriterion_satisfiedcorrect82.6%0.979
  • criterion_satisfied 82.6%
  • not_enough_information 7.6%
  • criterion_not_satisfied 6.9%
  • not_applicable 2.9%
decider-2bcriterion_satisfiedcorrect92.1%0.996
  • criterion_satisfied 92.1%
  • criterion_not_satisfied 2.8%
  • not_enough_information 2.8%
  • not_applicable 2.4%
System One Scorercriterion_satisfiedcorrect88.8%0.990
  • criterion_satisfied 88.8%
  • not_enough_information 7.9%
  • criterion_not_satisfied 2.2%
  • not_applicable 1.1%
OpenJevcriterion_satisfiedcorrect99.9%1.000
  • criterion_satisfied 99.9%
  • criterion_not_satisfied 0.1%
  • not_applicable 0.0%
  • not_enough_information 0.0%
NanoJevcriterion_satisfiedcorrect47.2%0.812
  • criterion_satisfied 47.2%
  • not_enough_information 22.4%
  • not_applicable 16.2%
  • criterion_not_satisfied 14.2%
Sample 3 of 6 · retrieval-and-knowledge:trialgpt-46 · 121 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "patient_note": "0. A 58-year-old nonsmoker white female with mild exertional dyspnea and occasional cough is found to have a left lung mass on chest x-ray.\n1. She is otherwise asymptomatic.\n2. A neurologic examination is unremarkable, but a CT scan of the head shows a solitary mass in the right frontal lobe.\n3. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
  "criterion_type": "inclusion",
  "criterion": "Pulmonary nodule on a recent CT"
}

Question criterion_relation · choice · primary (ranked)

The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?

4 options
  • criterion_satisfied — The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).
  • criterion_not_satisfied — The note gives evidence the patient does not meet this criterion (not included / not excluded).
  • not_enough_information — The criterion is relevant to this patient but the note lacks the information needed to decide.
  • not_applicable — The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: criterion_not_satisfied

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnot_enough_informationwrong39.0%0.694
  • not_enough_information 47.0%
  • criterion_not_satisfied 39.0%
  • criterion_satisfied 14.0%
  • not_applicable 0.0%
JevFishcriterion_not_satisfiedcorrect38.6%0.740
  • criterion_not_satisfied 38.6%
  • not_enough_information 27.0%
  • not_applicable 24.5%
  • criterion_satisfied 10.0%
decider-2bcriterion_satisfiedwrong28.3%0.646
  • criterion_satisfied 32.7%
  • criterion_not_satisfied 28.3%
  • not_enough_information 27.0%
  • not_applicable 11.9%
System One Scorercriterion_satisfiedwrong6.4%0.352
  • criterion_satisfied 47.7%
  • not_enough_information 44.0%
  • criterion_not_satisfied 6.4%
  • not_applicable 1.9%
OpenJevnot_enough_informationwrong17.0%0.316
  • not_enough_information 82.4%
  • criterion_not_satisfied 17.0%
  • not_applicable 0.5%
  • criterion_satisfied 0.1%
NanoJevcriterion_satisfiedwrong13.7%0.491
  • criterion_satisfied 39.0%
  • not_enough_information 30.9%
  • not_applicable 16.4%
  • criterion_not_satisfied 13.7%
Sample 4 of 6 · retrieval-and-knowledge:trialgpt-640 · 263 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "patient_note": "0. An 82-year-old woman comes to the emergency department because of chest pain and shortness of breath after being awakened in the morning by stabbing substernal chest pain radiating to the left shoulder and jaw.\n1. The patient had hypertension, renal-artery stenosis with chronic renal insufficiency, hypercholesterolemia, osteoporosis and dementia.\n2. Blood pressure was 199/108 mm Hg, respiratory rate 18 bpm, oxygen saturation 98% on ambient air.\n3. The heart sounds were rapid and with no murmurs.\n4. CK-MB was 10.9 ng/ml, CK was 89 U/l, CK index was 12.2% and Troponin T was 0.40 ng/ml.\n5. An EKG showed sinus regular tachycardia of 119 bpm, with ST-segment elevations up to 3 mm in V1, V2, and V3.\n6. A chest radiograph showed low lung volumes and right basilar subsegmental atelectasis.\n7. …",
  "criterion_type": "inclusion",
  "criterion": "typical chest pain"
}

Question criterion_relation · choice · primary (ranked)

The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?

4 options
  • criterion_satisfied — The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).
  • criterion_not_satisfied — The note gives evidence the patient does not meet this criterion (not included / not excluded).
  • not_enough_information — The criterion is relevant to this patient but the note lacks the information needed to decide.
  • not_applicable — The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: criterion_satisfied

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevcriterion_satisfiedcorrect100.0%1.000
  • criterion_satisfied 100.0%
  • criterion_not_satisfied 0.0%
  • not_enough_information 0.0%
  • not_applicable 0.0%
JevFishcriterion_satisfiedcorrect42.2%0.769
  • criterion_satisfied 42.2%
  • not_enough_information 27.0%
  • criterion_not_satisfied 21.9%
  • not_applicable 8.9%
decider-2bcriterion_satisfiedcorrect76.8%0.962
  • criterion_satisfied 76.8%
  • not_enough_information 13.0%
  • criterion_not_satisfied 6.0%
  • not_applicable 4.3%
System One Scorercriterion_satisfiedcorrect86.0%0.986
  • criterion_satisfied 86.0%
  • not_enough_information 8.8%
  • criterion_not_satisfied 3.6%
  • not_applicable 1.6%
OpenJevcriterion_satisfiedcorrect100.0%1.000
  • criterion_satisfied 100.0%
  • criterion_not_satisfied 0.1%
  • not_applicable 0.0%
  • not_enough_information 0.0%
NanoJevcriterion_satisfiedcorrect41.8%0.772
  • criterion_satisfied 41.8%
  • not_enough_information 24.4%
  • not_applicable 17.6%
  • criterion_not_satisfied 16.1%
Sample 5 of 6 · retrieval-and-knowledge:trialgpt-839 · 263 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "patient_note": "0. A 4 year old boy presents to the emergency room with wheezing.\n1. He has had a history of allergic rhinitis, but no history of wheezing.\n2. His mother reports that 5 hours ago patient was playing in the backyard sandbox when she heard him suddenly start coughing.\n3. The coughing lasted only moments, but he has been audibly wheezing since.\n4. Mother was concerned, because his breathing has not returned to normal, so she brought him to the ED.\n5. On exam, the child is playful and well appearing.\n6. Wheezing is heard in the mid-right chest area.\n7. O2 sats are 100% on room air.\n8. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
  "criterion_type": "inclusion",
  "criterion": "Allergic sensitization to at least one of the following allergen preparations: (House Dust Mite f, House dust mite p, Cockroach, Tree mix, Grass Mix, Weed Mix, Mold Mix 1, Mold Mix 2, Rat, Mouse, Guinea Pig, Rabbit, Cat or Dog) confirmed by positive AST."
}

Question criterion_relation · choice · primary (ranked)

The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?

4 options
  • criterion_satisfied — The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).
  • criterion_not_satisfied — The note gives evidence the patient does not meet this criterion (not included / not excluded).
  • not_enough_information — The criterion is relevant to this patient but the note lacks the information needed to decide.
  • not_applicable — The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: not_enough_information

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnot_enough_informationcorrect96.0%0.999
  • not_enough_information 96.0%
  • not_applicable 2.0%
  • criterion_not_satisfied 2.0%
  • criterion_satisfied 0.0%
JevFishnot_enough_informationcorrect41.4%0.737
  • not_enough_information 41.4%
  • criterion_not_satisfied 38.2%
  • not_applicable 19.0%
  • criterion_satisfied 1.5%
decider-2bnot_enough_informationcorrect43.9%0.748
  • not_enough_information 43.9%
  • criterion_not_satisfied 41.8%
  • not_applicable 12.0%
  • criterion_satisfied 2.3%
System One Scorernot_enough_informationcorrect55.9%0.861
  • not_enough_information 55.9%
  • criterion_satisfied 22.5%
  • criterion_not_satisfied 17.9%
  • not_applicable 3.8%
OpenJevnot_enough_informationcorrect100.0%1.000
  • not_enough_information 100.0%
  • criterion_not_satisfied 0.0%
  • not_applicable 0.0%
  • criterion_satisfied 0.0%
NanoJevcriterion_satisfiedwrong24.7%0.605
  • criterion_satisfied 40.2%
  • not_enough_information 24.7%
  • criterion_not_satisfied 18.9%
  • not_applicable 16.2%
Sample 6 of 6 · retrieval-and-knowledge:trialgpt-918 · 145 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "patient_note": "0. A 32-year-old male presents to your office complaining of diarrhea, abdominal cramping and flatulence.\n1. Stools are greasy and foul-smelling.\n2. He also has loss of appetite and malaise.\n3. He recently returned home from a hiking trip in the mountains where he drank water from natural sources.\n4. An iodine-stained stool smear revealed ellipsoidal cysts with smooth, well-defined walls and 2+ nuclei.\n5. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
  "criterion_type": "inclusion",
  "criterion": "Able to provide written, informed consent"
}

Question criterion_relation · choice · primary (ranked)

The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?

4 options
  • criterion_satisfied — The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).
  • criterion_not_satisfied — The note gives evidence the patient does not meet this criterion (not included / not excluded).
  • not_enough_information — The criterion is relevant to this patient but the note lacks the information needed to decide.
  • not_applicable — The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: criterion_satisfied

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevcriterion_satisfiedcorrect99.0%1.000
  • criterion_satisfied 99.0%
  • not_enough_information 1.0%
  • not_applicable 0.0%
  • criterion_not_satisfied 0.0%
JevFishcriterion_satisfiedcorrect76.9%0.964
  • criterion_satisfied 76.9%
  • not_enough_information 8.8%
  • criterion_not_satisfied 8.8%
  • not_applicable 5.6%
decider-2bcriterion_satisfiedcorrect79.9%0.973
  • criterion_satisfied 79.9%
  • criterion_not_satisfied 7.6%
  • not_enough_information 7.6%
  • not_applicable 4.9%
System One Scorercriterion_satisfiedcorrect67.5%0.909
  • criterion_satisfied 67.5%
  • not_enough_information 27.1%
  • criterion_not_satisfied 3.3%
  • not_applicable 2.1%
OpenJevcriterion_satisfiedcorrect100.0%1.000
  • criterion_satisfied 100.0%
  • criterion_not_satisfied 0.0%
  • not_applicable 0.0%
  • not_enough_information 0.0%
NanoJevcriterion_satisfiedcorrect45.3%0.797
  • criterion_satisfied 45.3%
  • not_enough_information 25.1%
  • criterion_not_satisfied 15.0%
  • not_applicable 14.6%