trialgpt-criterion-support
citation verification · healthcare · 159 items · primary question type choice. Source ncbi/TrialGPT-Criterion-Annotations (Public domain (NCBI)); labels are expert labels, never an LLM judge. Download these samples (JSON).
| System | DecisionScore | Accuracy (top pick) | Answered |
|---|---|---|---|
| Jev | 73.8 | 64.1% | 159 / 159 |
| JevFish | 73.5 | 61.6% | 159 / 159 |
| decider-2b | 73.3 | 57.9% | 159 / 159 |
| System One Scorer | 70.6 | 53.5% | 159 / 159 |
| OpenJev | 60.6 | 53.5% | 159 / 159 |
| NanoJev | 58.7 | 12.6% | 159 / 159 |
retrieval-and-knowledge:trialgpt-205 · 134 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"patient_note": "0. A 25-year-old woman presents to the clinic complaining of prolonged fatigue.\n1. She denies difficulty sleeping and sleeps an average of 8 hours a night.\n2. She also notes hair loss, a change in her voice and weight gain during the previous 6 months.\n3. She complains of cold intolerance.\n4. On examination she has a prominent, soft, uniform anterior cervical mass at the midline.\n5. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
"criterion_type": "exclusion",
"criterion": "having evidence of end-stage thyroiditis"
}Question criterion_relation · choice · primary (ranked)
The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?
4 options
criterion_satisfied— The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).criterion_not_satisfied— The note gives evidence the patient does not meet this criterion (not included / not excluded).not_enough_information— The criterion is relevant to this patient but the note lacks the information needed to decide.not_applicable— The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: not_enough_information
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | criterion_not_satisfied | wrong | 35.0% | 0.667 |
|
| JevFish | criterion_not_satisfied | wrong | 14.0% | 0.356 |
|
| decider-2b | criterion_not_satisfied | wrong | 10.4% | 0.259 |
|
| System One Scorer | criterion_satisfied | wrong | 29.9% | 0.645 |
|
| OpenJev | not_enough_information | correct | 73.8% | 0.932 |
|
| NanoJev | criterion_satisfied | wrong | 22.8% | 0.584 |
|
retrieval-and-knowledge:trialgpt-413 · 167 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"patient_note": "0. A 21-year-old female is evaluated for progressive arthralgias and malaise.\n1. On examination she is found to have alopecia, a rash mainly distributed on the bridge of her nose and her cheeks, a delicate non-palpable purpura on her calves, and swelling and tenderness of her wrists and ankles.\n2. Her lab shows normocytic anemia, thrombocytopenia, a 4/4 positive ANA and anti-dsDNA.\n3. Her urine is positive for protein and RBC casts.\n4. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
"criterion_type": "inclusion",
"criterion": "Patients with systemic Lupus erythematosus"
}Question criterion_relation · choice · primary (ranked)
The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?
4 options
criterion_satisfied— The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).criterion_not_satisfied— The note gives evidence the patient does not meet this criterion (not included / not excluded).not_enough_information— The criterion is relevant to this patient but the note lacks the information needed to decide.not_applicable— The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: criterion_satisfied
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | criterion_satisfied | correct | 100.0% | 1.000 |
|
| JevFish | criterion_satisfied | correct | 82.6% | 0.979 |
|
| decider-2b | criterion_satisfied | correct | 92.1% | 0.996 |
|
| System One Scorer | criterion_satisfied | correct | 88.8% | 0.990 |
|
| OpenJev | criterion_satisfied | correct | 99.9% | 1.000 |
|
| NanoJev | criterion_satisfied | correct | 47.2% | 0.812 |
|
retrieval-and-knowledge:trialgpt-46 · 121 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"patient_note": "0. A 58-year-old nonsmoker white female with mild exertional dyspnea and occasional cough is found to have a left lung mass on chest x-ray.\n1. She is otherwise asymptomatic.\n2. A neurologic examination is unremarkable, but a CT scan of the head shows a solitary mass in the right frontal lobe.\n3. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
"criterion_type": "inclusion",
"criterion": "Pulmonary nodule on a recent CT"
}Question criterion_relation · choice · primary (ranked)
The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?
4 options
criterion_satisfied— The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).criterion_not_satisfied— The note gives evidence the patient does not meet this criterion (not included / not excluded).not_enough_information— The criterion is relevant to this patient but the note lacks the information needed to decide.not_applicable— The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: criterion_not_satisfied
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | not_enough_information | wrong | 39.0% | 0.694 |
|
| JevFish | criterion_not_satisfied | correct | 38.6% | 0.740 |
|
| decider-2b | criterion_satisfied | wrong | 28.3% | 0.646 |
|
| System One Scorer | criterion_satisfied | wrong | 6.4% | 0.352 |
|
| OpenJev | not_enough_information | wrong | 17.0% | 0.316 |
|
| NanoJev | criterion_satisfied | wrong | 13.7% | 0.491 |
|
retrieval-and-knowledge:trialgpt-640 · 263 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"patient_note": "0. An 82-year-old woman comes to the emergency department because of chest pain and shortness of breath after being awakened in the morning by stabbing substernal chest pain radiating to the left shoulder and jaw.\n1. The patient had hypertension, renal-artery stenosis with chronic renal insufficiency, hypercholesterolemia, osteoporosis and dementia.\n2. Blood pressure was 199/108 mm Hg, respiratory rate 18 bpm, oxygen saturation 98% on ambient air.\n3. The heart sounds were rapid and with no murmurs.\n4. CK-MB was 10.9 ng/ml, CK was 89 U/l, CK index was 12.2% and Troponin T was 0.40 ng/ml.\n5. An EKG showed sinus regular tachycardia of 119 bpm, with ST-segment elevations up to 3 mm in V1, V2, and V3.\n6. A chest radiograph showed low lung volumes and right basilar subsegmental atelectasis.\n7. …",
"criterion_type": "inclusion",
"criterion": "typical chest pain"
}Question criterion_relation · choice · primary (ranked)
The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?
4 options
criterion_satisfied— The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).criterion_not_satisfied— The note gives evidence the patient does not meet this criterion (not included / not excluded).not_enough_information— The criterion is relevant to this patient but the note lacks the information needed to decide.not_applicable— The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: criterion_satisfied
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | criterion_satisfied | correct | 100.0% | 1.000 |
|
| JevFish | criterion_satisfied | correct | 42.2% | 0.769 |
|
| decider-2b | criterion_satisfied | correct | 76.8% | 0.962 |
|
| System One Scorer | criterion_satisfied | correct | 86.0% | 0.986 |
|
| OpenJev | criterion_satisfied | correct | 100.0% | 1.000 |
|
| NanoJev | criterion_satisfied | correct | 41.8% | 0.772 |
|
retrieval-and-knowledge:trialgpt-839 · 263 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"patient_note": "0. A 4 year old boy presents to the emergency room with wheezing.\n1. He has had a history of allergic rhinitis, but no history of wheezing.\n2. His mother reports that 5 hours ago patient was playing in the backyard sandbox when she heard him suddenly start coughing.\n3. The coughing lasted only moments, but he has been audibly wheezing since.\n4. Mother was concerned, because his breathing has not returned to normal, so she brought him to the ED.\n5. On exam, the child is playful and well appearing.\n6. Wheezing is heard in the mid-right chest area.\n7. O2 sats are 100% on room air.\n8. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
"criterion_type": "inclusion",
"criterion": "Allergic sensitization to at least one of the following allergen preparations: (House Dust Mite f, House dust mite p, Cockroach, Tree mix, Grass Mix, Weed Mix, Mold Mix 1, Mold Mix 2, Rat, Mouse, Guinea Pig, Rabbit, Cat or Dog) confirmed by positive AST."
}Question criterion_relation · choice · primary (ranked)
The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?
4 options
criterion_satisfied— The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).criterion_not_satisfied— The note gives evidence the patient does not meet this criterion (not included / not excluded).not_enough_information— The criterion is relevant to this patient but the note lacks the information needed to decide.not_applicable— The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: not_enough_information
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | not_enough_information | correct | 96.0% | 0.999 |
|
| JevFish | not_enough_information | correct | 41.4% | 0.737 |
|
| decider-2b | not_enough_information | correct | 43.9% | 0.748 |
|
| System One Scorer | not_enough_information | correct | 55.9% | 0.861 |
|
| OpenJev | not_enough_information | correct | 100.0% | 1.000 |
|
| NanoJev | criterion_satisfied | wrong | 24.7% | 0.605 |
|
retrieval-and-knowledge:trialgpt-918 · 145 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"patient_note": "0. A 32-year-old male presents to your office complaining of diarrhea, abdominal cramping and flatulence.\n1. Stools are greasy and foul-smelling.\n2. He also has loss of appetite and malaise.\n3. He recently returned home from a hiking trip in the mountains where he drank water from natural sources.\n4. An iodine-stained stool smear revealed ellipsoidal cysts with smooth, well-defined walls and 2+ nuclei.\n5. The patient will provide informed consent, and will comply with the trial protocol without any practical issues.",
"criterion_type": "inclusion",
"criterion": "Able to provide written, informed consent"
}Question criterion_relation · choice · primary (ranked)
The state holds a patient_note, a clinical-trial criterion and its criterion_type (inclusion or exclusion). How does the note relate to the criterion?
4 options
criterion_satisfied— The note gives evidence the patient meets this criterion (inclusion: included; exclusion: the exclusion applies).criterion_not_satisfied— The note gives evidence the patient does not meet this criterion (not included / not excluded).not_enough_information— The criterion is relevant to this patient but the note lacks the information needed to decide.not_applicable— The criterion cannot apply to this patient at all (for example a sex- or age-specific criterion that does not concern them).
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: criterion_satisfied
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | criterion_satisfied | correct | 99.0% | 1.000 |
|
| JevFish | criterion_satisfied | correct | 76.9% | 0.964 |
|
| decider-2b | criterion_satisfied | correct | 79.9% | 0.973 |
|
| System One Scorer | criterion_satisfied | correct | 67.5% | 0.909 |
|
| OpenJev | criterion_satisfied | correct | 100.0% | 1.000 |
|
| NanoJev | criterion_satisfied | correct | 45.3% | 0.797 |
|