Skip to samples
Typed Decision Bench · tasks · Data and operations

arxiv-archive-subcategory

hierarchical classification · science publishing · 200 items · primary question type choice. Source gfissore/arxiv-abstracts-2021 (CC0-1.0); labels are outcome labels, never an LLM judge. Download these samples (JSON).

SystemDecisionScoreAccuracy (top pick)Answered
Jev76.888.5%200 / 200
System One Scorer70.681.5%200 / 200
decider-2b70.178.0%200 / 200
JevFish69.378.5%200 / 200
OpenJev65.486.5%200 / 200
NanoJev53.729.5%200 / 200
Sample 1 of 6 · data-and-operations:arxiv-34d4b5d025fe · 166 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "title": "New examples of Calabi-Yau threefolds and genus zero surfaces",
  "abstract": "We classify the subgroups of the automorphism group of the product of 4 projective lines admitting an invariant anticanonical smooth divisor on which the action is free. As a first application, we describe new examples of Calabi-Yau 3-folds with small Hodge numbers. In particular, the Picard number is 1 and the number of moduli is 5. Furthermore, the fundamental group is non-trivial. We also construct a new family of minimal surfaces of general type with geometric genus zero, K^2=3 and fundamental group of order 16. We show that this family dominates an irreducible component of dimension 4 of the moduli space of the surfaces of general type."
}

Question archive · choice · primary (ranked)

Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in?

11 options
  • astro_ph — Astrophysics
  • cond_mat — Condensed matter physics
  • cs — Computer science
  • econ — Economics
  • eess — Electrical engineering and systems science
  • math — Mathematics
  • nlin — Nonlinear sciences
  • physics — Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...)
  • q_bio — Quantitative biology
  • q_fin — Quantitative finance
  • stat — Statistics

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevmathcorrect100.0%1.000
  • math 100.0%
  • q_fin 0.0%
  • stat 0.0%
  • cs 0.0%
  • astro_ph 0.0%
  • cond_mat 0.0%
System One Scorermathcorrect71.3%0.954
  • math 71.3%
  • econ 6.0%
  • q_fin 4.0%
  • nlin 3.1%
  • astro_ph 2.9%
  • cond_mat 2.7%
decider-2bmathcorrect99.0%1.000
  • math 99.0%
  • nlin 0.3%
  • astro_ph 0.2%
  • physics 0.2%
  • cs 0.1%
  • eess 0.1%
JevFishmathcorrect96.7%0.999
  • math 96.7%
  • astro_ph 0.8%
  • stat 0.7%
  • nlin 0.4%
  • eess 0.3%
  • physics 0.3%
OpenJevmathcorrect99.9%1.000
  • math 99.9%
  • nlin 0.0%
  • stat 0.0%
  • astro_ph 0.0%
  • eess 0.0%
  • econ 0.0%
NanoJevstatwrong33.5%0.683
  • stat 42.8%
  • math 33.5%
  • physics 6.7%
  • nlin 4.6%
  • cs 2.7%
  • astro_ph 2.2%

Question subcategory · choice

Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories.

30 options
  • math_ac — math.ac
  • math_ag — math.ag
  • math_ap — math.ap
  • math_at — math.at
  • math_ca — math.ca
  • math_co — math.co
  • math_ct — math.ct
  • math_cv — math.cv
  • math_dg — math.dg
  • math_ds — math.ds
  • math_fa — math.fa
  • math_gm — math.gm
  • math_gn — math.gn
  • math_gr — math.gr
  • math_gt — math.gt
  • math_ho — math.ho
  • math_kt — math.kt
  • math_lo — math.lo
  • math_mg — math.mg
  • math_na — math.na
  • math_nt — math.nt
  • math_oa — math.oa
  • math_oc — math.oc
  • math_pr — math.pr
  • math_qa — math.qa
  • math_ra — math.ra
  • math_rt — math.rt
  • math_sg — math.sg
  • math_sp — math.sp
  • math_st — math.st

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math_ag

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevmath_agcorrect96.0%0.999
  • math_ag 96.0%
  • math_cv 2.0%
  • math_ca 1.0%
  • math_dg 1.0%
  • math_lo 0.0%
  • math_sp 0.0%
System One Scorermath_agcorrect13.8%0.605
  • math_ag 13.8%
  • math_mg 13.8%
  • math_ac 6.8%
  • math_ct 6.1%
  • math_oa 5.5%
  • math_at 5.5%
decider-2bmath_acwrong12.3%0.546
  • math_ac 29.3%
  • math_dg 20.0%
  • math_ag 12.3%
  • math_ca 8.4%
  • math_ap 3.9%
  • math_co 3.2%
JevFishmath_acwrong5.1%0.505
  • math_ac 19.8%
  • math_st 17.6%
  • math_sg 6.6%
  • math_dg 6.2%
  • math_ag 5.1%
  • math_sp 4.2%
OpenJevmath_cawrong1.1%0.424
  • math_ca 34.3%
  • math_ct 16.4%
  • math_gm 9.3%
  • math_st 8.6%
  • math_sp 7.3%
  • math_ap 4.4%
  • math_ag 1.1%
NanoJevmath_qawrong3.5%0.517
  • math_qa 4.9%
  • math_dg 4.8%
  • math_mg 4.7%
  • math_gn 4.7%
  • math_gm 4.5%
  • math_na 4.3%
  • math_ag 3.5%
Sample 2 of 6 · data-and-operations:arxiv-623ca84587da · 84 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "title": "On the existence of certain weak Fano threefolds of Picard number two",
  "abstract": "This article settles the question of existence of smooth weak Fano threefolds of Picard number two with small anti-canonical map and previously classified numerical invariants obtained by blowing up certain curves on smooth Fano threefolds of Picard number 1 with the exception of 12 numerical cases."
}

Question archive · choice · primary (ranked)

Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in?

11 options
  • astro_ph — Astrophysics
  • cond_mat — Condensed matter physics
  • cs — Computer science
  • econ — Economics
  • eess — Electrical engineering and systems science
  • math — Mathematics
  • nlin — Nonlinear sciences
  • physics — Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...)
  • q_bio — Quantitative biology
  • q_fin — Quantitative finance
  • stat — Statistics

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevmathcorrect100.0%1.000
  • math 100.0%
  • physics 0.0%
  • nlin 0.0%
  • astro_ph 0.0%
  • cs 0.0%
  • stat 0.0%
System One Scorermathcorrect64.8%0.929
  • math 64.8%
  • econ 9.4%
  • q_fin 5.5%
  • stat 3.6%
  • astro_ph 3.2%
  • physics 2.9%
decider-2bmathcorrect99.3%1.000
  • math 99.3%
  • nlin 0.3%
  • physics 0.1%
  • astro_ph 0.1%
  • cs 0.1%
  • cond_mat 0.0%
JevFishmathcorrect96.2%0.999
  • math 96.2%
  • stat 0.9%
  • astro_ph 0.9%
  • nlin 0.4%
  • eess 0.4%
  • physics 0.3%
OpenJevmathcorrect99.9%1.000
  • math 99.9%
  • nlin 0.0%
  • astro_ph 0.0%
  • cond_mat 0.0%
  • eess 0.0%
  • stat 0.0%
NanoJevmathcorrect47.5%0.804
  • math 47.5%
  • stat 32.1%
  • physics 10.7%
  • cs 1.8%
  • nlin 1.8%
  • econ 1.5%

Question subcategory · choice

Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories.

30 options
  • math_ac — math.ac
  • math_ag — math.ag
  • math_ap — math.ap
  • math_at — math.at
  • math_ca — math.ca
  • math_co — math.co
  • math_ct — math.ct
  • math_cv — math.cv
  • math_dg — math.dg
  • math_ds — math.ds
  • math_fa — math.fa
  • math_gm — math.gm
  • math_gn — math.gn
  • math_gr — math.gr
  • math_gt — math.gt
  • math_ho — math.ho
  • math_kt — math.kt
  • math_lo — math.lo
  • math_mg — math.mg
  • math_na — math.na
  • math_nt — math.nt
  • math_oa — math.oa
  • math_oc — math.oc
  • math_pr — math.pr
  • math_qa — math.qa
  • math_ra — math.ra
  • math_rt — math.rt
  • math_sg — math.sg
  • math_sp — math.sp
  • math_st — math.st

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math_ag

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevmath_agcorrect98.0%1.000
  • math_ag 98.0%
  • math_cv 1.0%
  • math_ac 1.0%
  • math_gm 0.0%
  • math_na 0.0%
  • math_co 0.0%
System One Scorermath_mgwrong10.5%0.573
  • math_mg 14.3%
  • math_ag 10.5%
  • math_fa 7.4%
  • math_ac 6.7%
  • math_ct 6.1%
  • math_cv 6.0%
decider-2bmath_acwrong14.3%0.552
  • math_ac 34.1%
  • math_dg 20.1%
  • math_ag 14.3%
  • math_ca 4.1%
  • math_ap 3.7%
  • math_at 2.8%
JevFishmath_acwrong6.4%0.517
  • math_ac 20.6%
  • math_st 17.7%
  • math_ag 6.4%
  • math_fa 5.7%
  • math_sg 4.7%
  • math_sp 4.3%
OpenJevmath_stwrong1.2%0.430
  • math_st 29.1%
  • math_ca 24.7%
  • math_at 6.8%
  • math_ct 5.7%
  • math_ap 4.9%
  • math_sp 4.9%
  • math_ag 1.2%
NanoJevmath_qawrong3.4%0.515
  • math_qa 6.1%
  • math_gn 5.8%
  • math_gm 5.4%
  • math_mg 4.9%
  • math_dg 4.9%
  • math_oa 4.5%
  • math_ag 3.4%
Sample 3 of 6 · data-and-operations:arxiv-896e849c9b4e · 191 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "title": "Stochastic theory of protein synthesis and polysome: ribosome profile on a single mRNA transcript",
  "abstract": "The process of polymerizing a protein by a ribosome, using a messenger RNA (mRNA) as the corresponding template, is called {\\it translation}. Ribosome may be regarded as a molecular motor for which the mRNA template serves also as the track. Often several ribosomes may translate the same (mRNA) simultaneously. The ribosomes bound simultaneously to a single mRNA transcript are the members of a polyribosome (or, simply, {\\it polysome}). Experimentally measured {\\it polysome profile} gives the distribution of polysome {\\it sizes}. Recently a breakthrough in determining the instantaneous {\\it positions} of the ribosomes on a given mRNA track has been achieved and the technique is called {\\it ribosome profiling} \\cite{ingolia10,guo10}."
}

Question archive · choice · primary (ranked)

Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in?

11 options
  • astro_ph — Astrophysics
  • cond_mat — Condensed matter physics
  • cs — Computer science
  • econ — Economics
  • eess — Electrical engineering and systems science
  • math — Mathematics
  • nlin — Nonlinear sciences
  • physics — Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...)
  • q_bio — Quantitative biology
  • q_fin — Quantitative finance
  • stat — Statistics

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: physics

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevq_biowrong2.0%0.049
  • q_bio 97.0%
  • physics 2.0%
  • math 1.0%
  • astro_ph 0.0%
  • cs 0.0%
  • stat 0.0%
System One Scorerq_biowrong1.7%0.217
  • q_bio 76.9%
  • math 5.9%
  • stat 4.8%
  • econ 2.6%
  • nlin 2.2%
  • cond_mat 1.7%
  • physics 1.7%
decider-2bq_biowrong1.4%0.123
  • q_bio 88.3%
  • math 3.0%
  • stat 2.3%
  • cond_mat 1.9%
  • nlin 1.5%
  • physics 1.4%
JevFishq_biowrong1.9%0.182
  • q_bio 81.8%
  • nlin 4.7%
  • cond_mat 2.9%
  • math 2.1%
  • physics 1.9%
  • stat 1.6%
OpenJevq_biowrong27.7%0.548
  • q_bio 61.2%
  • physics 27.7%
  • astro_ph 5.9%
  • cond_mat 4.5%
  • eess 0.3%
  • cs 0.1%
NanoJevstatwrong7.5%0.298
  • stat 72.2%
  • q_bio 16.5%
  • physics 7.5%
  • math 1.8%
  • nlin 0.7%
  • astro_ph 0.4%

Question subcategory · choice

Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories.

21 options
  • physics_acc_ph — physics.acc.ph
  • physics_ao_ph — physics.ao.ph
  • physics_atm_clus — physics.atm.clus
  • physics_atom_ph — physics.atom.ph
  • physics_bio_ph — physics.bio.ph
  • physics_chem_ph — physics.chem.ph
  • physics_class_ph — physics.class.ph
  • physics_comp_ph — physics.comp.ph
  • physics_data_an — physics.data.an
  • physics_ed_ph — physics.ed.ph
  • physics_flu_dyn — physics.flu.dyn
  • physics_gen_ph — physics.gen.ph
  • physics_geo_ph — physics.geo.ph
  • physics_hist_ph — physics.hist.ph
  • physics_ins_det — physics.ins.det
  • physics_med_ph — physics.med.ph
  • physics_optics — physics.optics
  • physics_plasm_ph — physics.plasm.ph
  • physics_pop_ph — physics.pop.ph
  • physics_soc_ph — physics.soc.ph
  • physics_space_ph — physics.space.ph

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: physics_bio_ph

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevphysics_bio_phcorrect99.0%1.000
  • physics_bio_ph 99.0%
  • physics_gen_ph 1.0%
  • physics_hist_ph 0.0%
  • physics_pop_ph 0.0%
  • physics_ed_ph 0.0%
  • physics_class_ph 0.0%
System One Scorerphysics_chem_phwrong6.2%0.522
  • physics_chem_ph 14.2%
  • physics_soc_ph 12.1%
  • physics_ao_ph 11.1%
  • physics_comp_ph 10.1%
  • physics_bio_ph 6.2%
  • physics_med_ph 5.8%
decider-2bphysics_bio_phcorrect72.6%0.952
  • physics_bio_ph 72.6%
  • physics_chem_ph 14.2%
  • physics_comp_ph 2.5%
  • physics_ao_ph 2.3%
  • physics_acc_ph 1.1%
  • physics_gen_ph 0.9%
JevFishphysics_bio_phcorrect71.7%0.955
  • physics_bio_ph 71.7%
  • physics_acc_ph 7.6%
  • physics_chem_ph 3.9%
  • physics_gen_ph 2.8%
  • physics_comp_ph 1.9%
  • physics_med_ph 1.5%
OpenJevphysics_bio_phcorrect41.9%0.790
  • physics_bio_ph 41.9%
  • physics_gen_ph 25.8%
  • physics_class_ph 7.7%
  • physics_acc_ph 7.2%
  • physics_atom_ph 4.9%
  • physics_ed_ph 4.0%
NanoJevphysics_data_anwrong5.1%0.525
  • physics_data_an 7.3%
  • physics_hist_ph 7.0%
  • physics_flu_dyn 7.0%
  • physics_ao_ph 6.9%
  • physics_gen_ph 6.0%
  • physics_acc_ph 5.6%
  • physics_bio_ph 5.1%
Sample 4 of 6 · data-and-operations:arxiv-8d6972262680 · 112 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "title": "On the lack of dispersion for a class of magnetic Dirac flows",
  "abstract": "We show that global Strichartz estimates for magnetic Dirac operators generally fails, if the potentials do not decay fast enough at infinity. In order to prove this, we construct some explicit examples of homogeneous magnetic potentials with less than Coulomb decay, i. e. with homogeneity-degree more than -1, such that the magnetic field points to a fixed direction, which does not depend on $x\\in \\mathbb{R}^3$."
}

Question archive · choice · primary (ranked)

Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in?

11 options
  • astro_ph — Astrophysics
  • cond_mat — Condensed matter physics
  • cs — Computer science
  • econ — Economics
  • eess — Electrical engineering and systems science
  • math — Mathematics
  • nlin — Nonlinear sciences
  • physics — Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...)
  • q_bio — Quantitative biology
  • q_fin — Quantitative finance
  • stat — Statistics

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevmathcorrect95.0%0.998
  • math 95.0%
  • physics 5.0%
  • astro_ph 0.0%
  • nlin 0.0%
  • econ 0.0%
  • q_bio 0.0%
System One Scorermathcorrect51.1%0.865
  • math 51.1%
  • cond_mat 10.6%
  • econ 7.2%
  • nlin 6.9%
  • physics 5.6%
  • astro_ph 5.3%
decider-2bmathcorrect72.2%0.939
  • math 72.2%
  • nlin 20.7%
  • physics 3.9%
  • cond_mat 1.7%
  • astro_ph 0.4%
  • eess 0.3%
JevFishmathcorrect56.8%0.843
  • math 56.8%
  • nlin 35.4%
  • physics 2.3%
  • cond_mat 1.7%
  • stat 1.2%
  • astro_ph 1.1%
OpenJevmathcorrect77.2%0.952
  • math 77.2%
  • physics 21.1%
  • cond_mat 0.6%
  • astro_ph 0.5%
  • nlin 0.2%
  • eess 0.2%
NanoJevstatwrong28.4%0.542
  • stat 63.3%
  • math 28.4%
  • physics 5.4%
  • nlin 1.4%
  • cond_mat 0.4%
  • astro_ph 0.4%

Question subcategory · choice

Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories.

30 options
  • math_ac — math.ac
  • math_ag — math.ag
  • math_ap — math.ap
  • math_at — math.at
  • math_ca — math.ca
  • math_co — math.co
  • math_ct — math.ct
  • math_cv — math.cv
  • math_dg — math.dg
  • math_ds — math.ds
  • math_fa — math.fa
  • math_gm — math.gm
  • math_gn — math.gn
  • math_gr — math.gr
  • math_gt — math.gt
  • math_ho — math.ho
  • math_kt — math.kt
  • math_lo — math.lo
  • math_mg — math.mg
  • math_na — math.na
  • math_nt — math.nt
  • math_oa — math.oa
  • math_oc — math.oc
  • math_pr — math.pr
  • math_qa — math.qa
  • math_ra — math.ra
  • math_rt — math.rt
  • math_sg — math.sg
  • math_sp — math.sp
  • math_st — math.st

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math_ap

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevmath_fawrong10.0%0.515
  • math_fa 29.0%
  • math_ac 22.0%
  • math_sp 11.0%
  • math_ap 10.0%
  • math_ds 9.0%
  • math_oc 7.0%
System One Scorermath_apcorrect9.5%0.570
  • math_ap 9.5%
  • math_pr 8.1%
  • math_ac 6.8%
  • math_ag 6.3%
  • math_sp 5.5%
  • math_ct 5.1%
decider-2bmath_acwrong5.8%0.494
  • math_ac 29.6%
  • math_ag 13.7%
  • math_dg 7.0%
  • math_ca 6.1%
  • math_ap 5.8%
  • math_at 5.0%
JevFishmath_acwrong4.6%0.500
  • math_ac 19.5%
  • math_st 18.4%
  • math_ag 6.1%
  • math_sg 5.4%
  • math_sp 5.2%
  • math_ap 4.6%
OpenJevmath_stwrong3.0%0.456
  • math_st 34.1%
  • math_gm 11.0%
  • math_ca 7.9%
  • math_ct 6.4%
  • math_co 4.7%
  • math_ds 4.4%
  • math_ap 3.0%
NanoJevmath_mgwrong2.5%0.506
  • math_mg 6.4%
  • math_gm 5.9%
  • math_gn 5.5%
  • math_qa 4.9%
  • math_dg 4.7%
  • math_na 4.7%
  • math_ap 2.5%
Sample 5 of 6 · data-and-operations:arxiv-b87c1c2f175e · 189 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "title": "Tanaka's equation on the circle and stochastic flows",
  "abstract": "We define a Tanaka's equation on an oriented graph with two edges and two vertices. This graph will be embedded in the unit circle. Extending this equation to flows of kernels, we show that the laws of the flows of kernels $K$ solution of Tanaka's equation can be classified by pairs of probability measures $(m^+,m^-)$ on $[0,1]$, with mean 1/2. What happens at the first vertex is governed by $m^+$, and at the second by $m^-$. For each vertex $P$, we construct a sequence of stopping times along which the image of the whole circle by $K$ is reduced to $P$. We also prove that the supports of these flows contains a finite number of points, and that except for some particular cases this number of points can be arbitrarily large."
}

Question archive · choice · primary (ranked)

Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in?

11 options
  • astro_ph — Astrophysics
  • cond_mat — Condensed matter physics
  • cs — Computer science
  • econ — Economics
  • eess — Electrical engineering and systems science
  • math — Mathematics
  • nlin — Nonlinear sciences
  • physics — Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...)
  • q_bio — Quantitative biology
  • q_fin — Quantitative finance
  • stat — Statistics

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevmathcorrect99.0%1.000
  • math 99.0%
  • nlin 1.0%
  • cs 0.0%
  • eess 0.0%
  • astro_ph 0.0%
  • cond_mat 0.0%
System One Scorermathcorrect36.6%0.766
  • math 36.6%
  • nlin 16.5%
  • stat 14.4%
  • q_fin 8.7%
  • econ 6.1%
  • cond_mat 4.8%
decider-2bmathcorrect61.7%0.881
  • math 61.7%
  • nlin 30.0%
  • stat 4.2%
  • physics 1.5%
  • cs 0.6%
  • cond_mat 0.5%
JevFishmathcorrect59.5%0.867
  • math 59.5%
  • nlin 31.8%
  • stat 4.4%
  • astro_ph 0.9%
  • cond_mat 0.9%
  • physics 0.8%
OpenJevmathcorrect99.6%1.000
  • math 99.6%
  • stat 0.2%
  • nlin 0.1%
  • astro_ph 0.1%
  • q_bio 0.0%
  • eess 0.0%
NanoJevmathcorrect50.8%0.842
  • math 50.8%
  • stat 25.8%
  • physics 5.3%
  • nlin 3.6%
  • econ 3.4%
  • q_fin 3.3%

Question subcategory · choice

Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories.

30 options
  • math_ac — math.ac
  • math_ag — math.ag
  • math_ap — math.ap
  • math_at — math.at
  • math_ca — math.ca
  • math_co — math.co
  • math_ct — math.ct
  • math_cv — math.cv
  • math_dg — math.dg
  • math_ds — math.ds
  • math_fa — math.fa
  • math_gm — math.gm
  • math_gn — math.gn
  • math_gr — math.gr
  • math_gt — math.gt
  • math_ho — math.ho
  • math_kt — math.kt
  • math_lo — math.lo
  • math_mg — math.mg
  • math_na — math.na
  • math_nt — math.nt
  • math_oa — math.oa
  • math_oc — math.oc
  • math_pr — math.pr
  • math_qa — math.qa
  • math_ra — math.ra
  • math_rt — math.rt
  • math_sg — math.sg
  • math_sp — math.sp
  • math_st — math.st

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math_pr

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevmath_stwrong2.0%0.376
  • math_st 36.0%
  • math_ds 33.0%
  • math_sp 22.0%
  • math_sg 3.0%
  • math_pr 2.0%
  • math_ra 2.0%
System One Scorermath_stwrong7.8%0.554
  • math_st 10.7%
  • math_pr 7.8%
  • math_ct 6.3%
  • math_oc 6.0%
  • math_ag 5.9%
  • math_oa 5.1%
decider-2bmath_acwrong2.5%0.459
  • math_ac 30.4%
  • math_ag 14.1%
  • math_ap 6.2%
  • math_ca 6.2%
  • math_dg 5.6%
  • math_at 3.8%
  • math_pr 2.5%
JevFishmath_stwrong2.8%0.482
  • math_st 22.6%
  • math_ac 14.4%
  • math_ag 5.8%
  • math_sg 5.8%
  • math_sp 4.7%
  • math_ap 4.7%
  • math_pr 2.8%
OpenJevmath_stwrong0.2%0.351
  • math_st 44.2%
  • math_gm 31.3%
  • math_at 6.9%
  • math_ds 4.3%
  • math_ca 1.5%
  • math_ct 1.2%
  • math_pr 0.2%
NanoJevmath_qawrong3.1%0.513
  • math_qa 4.8%
  • math_gn 4.5%
  • math_dg 4.3%
  • math_gm 4.2%
  • math_mg 4.2%
  • math_nt 4.2%
  • math_pr 3.1%
Sample 6 of 6 · data-and-operations:arxiv-baf88e3ff5a7 · 115 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "title": "Asymptotics of the $s$-perimeter as $s\\searrow 0$",
  "abstract": "We deal with the asymptotic behavior of the $s$-perimeter of a set $E$ inside a domain $\\Omega$ as $s\\searrow0$. We prove necessary and sufficient conditions for the existence of such limit, by also providing an explicit formulation in terms of the Lebesgue measure of $E$ and $\\Omega$. Moreover, we construct examples of sets for which the limit does not exist."
}

Question archive · choice · primary (ranked)

Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in?

11 options
  • astro_ph — Astrophysics
  • cond_mat — Condensed matter physics
  • cs — Computer science
  • econ — Economics
  • eess — Electrical engineering and systems science
  • math — Mathematics
  • nlin — Nonlinear sciences
  • physics — Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...)
  • q_bio — Quantitative biology
  • q_fin — Quantitative finance
  • stat — Statistics

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevmathcorrect100.0%1.000
  • math 100.0%
  • physics 0.0%
  • econ 0.0%
  • cs 0.0%
  • q_fin 0.0%
  • astro_ph 0.0%
System One Scorermathcorrect55.2%0.888
  • math 55.2%
  • cond_mat 7.4%
  • econ 7.3%
  • nlin 5.4%
  • physics 4.7%
  • stat 4.7%
decider-2bmathcorrect88.1%0.989
  • math 88.1%
  • nlin 8.8%
  • stat 1.3%
  • physics 0.6%
  • cs 0.3%
  • eess 0.2%
JevFishmathcorrect83.3%0.979
  • math 83.3%
  • nlin 11.3%
  • stat 1.9%
  • astro_ph 1.0%
  • physics 0.6%
  • eess 0.5%
OpenJevmathcorrect99.9%1.000
  • math 99.9%
  • nlin 0.0%
  • stat 0.0%
  • cond_mat 0.0%
  • eess 0.0%
  • astro_ph 0.0%
NanoJevmathcorrect52.9%0.856
  • math 52.9%
  • stat 23.4%
  • physics 7.8%
  • cs 4.3%
  • econ 3.8%
  • nlin 2.4%

Question subcategory · choice

Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories.

30 options
  • math_ac — math.ac
  • math_ag — math.ag
  • math_ap — math.ap
  • math_at — math.at
  • math_ca — math.ca
  • math_co — math.co
  • math_ct — math.ct
  • math_cv — math.cv
  • math_dg — math.dg
  • math_ds — math.ds
  • math_fa — math.fa
  • math_gm — math.gm
  • math_gn — math.gn
  • math_gr — math.gr
  • math_gt — math.gt
  • math_ho — math.ho
  • math_kt — math.kt
  • math_lo — math.lo
  • math_mg — math.mg
  • math_na — math.na
  • math_nt — math.nt
  • math_oa — math.oa
  • math_oc — math.oc
  • math_pr — math.pr
  • math_qa — math.qa
  • math_ra — math.ra
  • math_rt — math.rt
  • math_sg — math.sg
  • math_sp — math.sp
  • math_st — math.st

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math_ap

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevmath_mgwrong2.0%0.374
  • math_mg 46.0%
  • math_ca 26.0%
  • math_cv 7.0%
  • math_oc 5.0%
  • math_ac 4.0%
  • math_dg 3.0%
  • math_ap 2.0%
System One Scorermath_agwrong3.0%0.499
  • math_ag 17.3%
  • math_mg 6.5%
  • math_st 6.4%
  • math_gr 6.3%
  • math_ac 5.3%
  • math_co 4.5%
  • math_ap 3.0%
decider-2bmath_acwrong6.4%0.517
  • math_ac 23.4%
  • math_ag 11.4%
  • math_dg 8.1%
  • math_ca 7.0%
  • math_ap 6.4%
  • math_at 4.6%
JevFishmath_acwrong5.0%0.507
  • math_ac 18.5%
  • math_st 17.5%
  • math_sg 6.5%
  • math_ag 5.9%
  • math_sp 5.4%
  • math_ap 5.0%
OpenJevmath_ctwrong2.8%0.432
  • math_ct 38.0%
  • math_gm 13.5%
  • math_st 12.6%
  • math_fa 7.6%
  • math_sp 4.6%
  • math_ra 3.7%
  • math_ap 2.8%
NanoJevmath_qawrong3.1%0.513
  • math_qa 5.9%
  • math_gn 5.5%
  • math_gm 4.6%
  • math_dg 4.5%
  • math_oa 4.5%
  • math_mg 4.4%
  • math_ap 3.1%