Typed Decision Bench · tasks · Data and operations

`arxiv-archive-subcategory`

hierarchical classification · science publishing · 200 items · primary question type choice. Source gfissore/arxiv-abstracts-2021 (CC0-1.0); labels are outcome labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
Jev	76.8	88.5%	200 / 200
JevFish	70.7	81.0%	200 / 200
System One Scorer	70.6	81.5%	200 / 200
decider-2b	70.1	78.0%	200 / 200
OpenJev	65.4	86.5%	200 / 200
NanoJev	53.7	29.5%	200 / 200

Sample 1 of 6 · data-and-operations:arxiv-34f5b5f7bde3 · 132 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "title": "Von Neumann's growth model: statistical mechanics and biological applications",
  "abstract": "We review recent work on the statistical mechanics of Von Neumann's growth model and discuss its application to cellular metabolic networks. In this context, we present a detailed analysis of the physiological scenario underlying optimality a la Von Neumann in the metabolism of the bacterium Escherichia coli, showing that optimal solutions are characterized by a considerable microscopic flexibility accompanied by a robust emergent picture for the key physiological functions. This suggests that the ideas behind optimal economic growth in Von Neumann's model can be helpful in uncovering functional organization principles of cell energetics."
}

Question archive · choice · primary (ranked)

Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in?

11 options

astro_ph — Astrophysics
cond_mat — Condensed matter physics
cs — Computer science
econ — Economics
eess — Electrical engineering and systems science
math — Mathematics
nlin — Nonlinear sciences
physics — Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...)
q_bio — Quantitative biology
q_fin — Quantitative finance
stat — Statistics

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: cond_mat

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`q_bio`	wrong	6.0%	0.170	`q_bio` 88.0% `cond_mat` 6.0% `physics` 3.0% `econ` 1.0% `nlin` 1.0% `stat` 1.0%
JevFish	`q_bio`	wrong	6.7%	0.461	`q_bio` 36.4% `math` 16.7% `nlin` 15.9% `stat` 10.5% `physics` 10.0% `cond_mat` 6.7%
System One Scorer	`q_bio`	wrong	4.5%	0.316	`q_bio` 66.5% `econ` 7.5% `math` 6.7% `cond_mat` 4.5% `q_fin` 3.8% `nlin` 3.1%
decider-2b	`q_bio`	wrong	5.9%	0.427	`q_bio` 44.5% `nlin` 16.2% `math` 15.5% `stat` 8.3% `physics` 6.5% `cond_mat` 5.9%
OpenJev	`nlin`	wrong	11.3%	0.251	`nlin` 84.3% `cond_mat` 11.3% `physics` 3.5% `astro_ph` 0.3% `cs` 0.2% `math` 0.2%
NanoJev	`stat`	wrong	0.1%	0.080	`stat` 91.6% `q_bio` 6.0% `physics` 1.1% `econ` 0.6% `math` 0.3% `cond_mat` 0.1%

Question subcategory · choice

Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories.

9 options

cond_mat_dis_nn — cond.mat.dis.nn
cond_mat_mes_hall — cond.mat.mes.hall
cond_mat_mtrl_sci — cond.mat.mtrl.sci
cond_mat_other — cond.mat.other
cond_mat_quant_gas — cond.mat.quant.gas
cond_mat_soft — cond.mat.soft
cond_mat_stat_mech — cond.mat.stat.mech
cond_mat_str_el — cond.mat.str.el
cond_mat_supr_con — cond.mat.supr.con

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: cond_mat_dis_nn

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`cond_mat_stat_mech`	wrong	0.0%	0.048	`cond_mat_stat_mech` 95.0% `cond_mat_soft` 3.0% `cond_mat_other` 2.0% `cond_mat_mtrl_sci` 0.0% `cond_mat_mes_hall` 0.0% `cond_mat_quant_gas` 0.0% `cond_mat_dis_nn` 0.0%
JevFish	`cond_mat_stat_mech`	wrong	3.4%	0.230	`cond_mat_stat_mech` 77.4% `cond_mat_supr_con` 5.7% `cond_mat_soft` 5.6% `cond_mat_other` 3.6% `cond_mat_dis_nn` 3.4% `cond_mat_mtrl_sci` 2.1%
System One Scorer	`cond_mat_soft`	wrong	11.5%	0.517	`cond_mat_soft` 29.4% `cond_mat_stat_mech` 24.2% `cond_mat_other` 17.4% `cond_mat_dis_nn` 11.5% `cond_mat_str_el` 6.3% `cond_mat_mes_hall` 4.3%
decider-2b	`cond_mat_stat_mech`	wrong	2.2%	0.153	`cond_mat_stat_mech` 85.6% `cond_mat_soft` 6.1% `cond_mat_other` 2.3% `cond_mat_dis_nn` 2.2% `cond_mat_mes_hall` 1.0% `cond_mat_mtrl_sci` 0.9%
OpenJev	`cond_mat_stat_mech`	wrong	0.4%	0.089	`cond_mat_stat_mech` 91.0% `cond_mat_other` 4.8% `cond_mat_mtrl_sci` 1.3% `cond_mat_soft` 1.2% `cond_mat_mes_hall` 0.6% `cond_mat_str_el` 0.4% `cond_mat_dis_nn` 0.4%
NanoJev	`cond_mat_stat_mech`	wrong	13.7%	0.575	`cond_mat_stat_mech` 19.0% `cond_mat_dis_nn` 13.7% `cond_mat_quant_gas` 12.4% `cond_mat_mes_hall` 12.3% `cond_mat_mtrl_sci` 11.7% `cond_mat_str_el` 10.2%

Sample 2 of 6 · data-and-operations:arxiv-5193e00f8dc9 · 143 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "title": "Tannaka-Krein duality for compact quantum homogeneous spaces. II. Classification of quantum homogeneous spaces for quantum SU(2)",
  "abstract": "We apply the Tannaka-Krein duality theory for quantum homogeneous spaces, developed in the first part of this series of papers, to the case of the quantum SU(2) groups. We obtain a classification of their quantum homogeneous spaces in terms of weighted oriented graphs. The equivariant maps between these quantum homogeneous spaces can be characterized by certain quadratic equations associated with the braiding on the representations of SUq(2). We show that, for |q| close to 1, all quantum homogeneous spaces are realized by coideals."
}

Question archive · choice · primary (ranked)

Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in?

11 options

astro_ph — Astrophysics
cond_mat — Condensed matter physics
cs — Computer science
econ — Economics
eess — Electrical engineering and systems science
math — Mathematics
nlin — Nonlinear sciences
physics — Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...)
q_bio — Quantitative biology
q_fin — Quantitative finance
stat — Statistics

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`math`	correct	100.0%	1.000	`math` 100.0% `q_fin` 0.0% `astro_ph` 0.0% `cond_mat` 0.0% `eess` 0.0% `stat` 0.0%
JevFish	`math`	correct	92.1%	0.996	`math` 92.1% `physics` 4.7% `nlin` 1.5% `cs` 0.5% `cond_mat` 0.3% `astro_ph` 0.3%
System One Scorer	`math`	correct	68.2%	0.943	`math` 68.2% `nlin` 6.5% `econ` 5.1% `cond_mat` 5.0% `physics` 3.3% `q_fin` 2.8%
decider-2b	`math`	correct	97.3%	1.000	`math` 97.3% `physics` 1.4% `nlin` 0.4% `cs` 0.3% `cond_mat` 0.2% `stat` 0.1%
OpenJev	`math`	correct	99.7%	1.000	`math` 99.7% `nlin` 0.1% `stat` 0.0% `cond_mat` 0.0% `eess` 0.0% `astro_ph` 0.0%
NanoJev	`math`	correct	40.9%	0.771	`math` 40.9% `stat` 29.5% `physics` 13.9% `q_fin` 2.8% `cs` 2.6% `nlin` 2.4%

Question subcategory · choice

Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories.

30 options

math_ac — math.ac
math_ag — math.ag
math_ap — math.ap
math_at — math.at
math_ca — math.ca
math_co — math.co
math_ct — math.ct
math_cv — math.cv
math_dg — math.dg
math_ds — math.ds
math_fa — math.fa
math_gm — math.gm
math_gn — math.gn
math_gr — math.gr
math_gt — math.gt
math_ho — math.ho
math_kt — math.kt
math_lo — math.lo
math_mg — math.mg
math_na — math.na
math_nt — math.nt
math_oa — math.oa
math_oc — math.oc
math_pr — math.pr
math_qa — math.qa
math_ra — math.ra
math_rt — math.rt
math_sg — math.sg
math_sp — math.sp
math_st — math.st

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math_oa

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`math_qa`	wrong	12.0%	0.292	`math_qa` 80.0% `math_oa` 12.0% `math_kt` 3.0% `math_ct` 2.0% `math_ra` 1.0% `math_mg` 1.0%
JevFish	`math_qa`	wrong	1.0%	0.466	`math_qa` 20.9% `math_ac` 13.2% `math_st` 10.0% `math_ag` 6.0% `math_sp` 5.7% `math_ca` 4.5% `math_oa` 1.0%
System One Scorer	`math_oa`	correct	9.3%	0.569	`math_oa` 9.3% `math_ct` 7.9% `math_at` 6.6% `math_oc` 5.9% `math_mg` 5.4% `math_ag` 4.8%
decider-2b	`math_ac`	wrong	0.4%	0.427	`math_ac` 34.4% `math_ag` 12.0% `math_ca` 8.1% `math_qa` 7.8% `math_ap` 4.4% `math_dg` 3.1% `math_oa` 0.4%
OpenJev	`math_gm`	wrong	0.0%	0.436	`math_gm` 20.6% `math_st` 18.2% `math_ca` 16.6% `math_ct` 9.6% `math_kt` 9.2% `math_qa` 5.6% `math_oa` 0.0%
NanoJev	`math_qa`	wrong	3.5%	0.517	`math_qa` 5.5% `math_gn` 4.9% `math_gm` 4.3% `math_na` 4.3% `math_gt` 4.3% `math_dg` 4.2% `math_oa` 3.5%

Sample 3 of 6 · data-and-operations:arxiv-89d8203a971c · 158 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "title": "Model Based Framework for Estimating Mutation Rate of Hepatitis C Virus in Egypt",
  "abstract": "Hepatitis C virus (HCV) is a widely spread disease all over the world. HCV has very high mutation rate that makes it resistant to antibodies. Modeling HCV to identify the virus mutation process is essential to its detection and predicting its evolution. This paper presents a model based framework for estimating mutation rate of HCV in two steps. Firstly profile hidden Markov model (PHMM) architecture was builder to select the sequences which represents sequence per year. Secondly mutation rate was calculated by using pair-wise distance method between sequences. A pilot study is conducted on NS5B zone of HCV dataset of genotype 4 subtype a (HCV4a) in Egypt."
}

Question archive · choice · primary (ranked)

Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in?

11 options

astro_ph — Astrophysics
cond_mat — Condensed matter physics
cs — Computer science
econ — Economics
eess — Electrical engineering and systems science
math — Mathematics
nlin — Nonlinear sciences
physics — Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...)
q_bio — Quantitative biology
q_fin — Quantitative finance
stat — Statistics

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: cs

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`q_bio`	wrong	1.0%	0.020	`q_bio` 99.0% `cs` 1.0% `cond_mat` 0.0% `stat` 0.0% `physics` 0.0% `astro_ph` 0.0%
JevFish	`q_bio`	wrong	13.8%	0.472	`q_bio` 51.5% `stat` 18.9% `cs` 13.8% `math` 10.6% `nlin` 2.9% `eess` 0.9%
System One Scorer	`q_bio`	wrong	1.9%	0.207	`q_bio` 78.5% `econ` 4.2% `stat` 3.9% `math` 3.4% `nlin` 3.0% `q_fin` 2.0% `cs` 1.9%
decider-2b	`q_bio`	wrong	7.2%	0.379	`q_bio` 56.7% `stat` 21.7% `math` 10.5% `cs` 7.2% `nlin` 1.3% `eess` 0.9%
OpenJev	`cs`	correct	98.2%	1.000	`cs` 98.2% `eess` 0.5% `stat` 0.3% `cond_mat` 0.2% `math` 0.2% `astro_ph` 0.2%
NanoJev	`stat`	wrong	0.1%	0.257	`stat` 59.2% `q_bio` 36.9% `physics` 2.6% `math` 0.7% `nlin` 0.3% `astro_ph` 0.1% `cs` 0.1%

Question subcategory · choice

Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories.

40 options

cs_ai — cs.ai
cs_ar — cs.ar
cs_cc — cs.cc
cs_ce — cs.ce
cs_cg — cs.cg
cs_cl — cs.cl
cs_cr — cs.cr
cs_cv — cs.cv
cs_cy — cs.cy
cs_db — cs.db
cs_dc — cs.dc
cs_dl — cs.dl
cs_dm — cs.dm
cs_ds — cs.ds
cs_et — cs.et
cs_fl — cs.fl
cs_gl — cs.gl
cs_gr — cs.gr
cs_gt — cs.gt
cs_hc — cs.hc
cs_ir — cs.ir
cs_it — cs.it
cs_lg — cs.lg
cs_lo — cs.lo
cs_ma — cs.ma
cs_mm — cs.mm
cs_ms — cs.ms
cs_na — cs.na
cs_ne — cs.ne
cs_ni — cs.ni
cs_oh — cs.oh
cs_os — cs.os
cs_pf — cs.pf
cs_pl — cs.pl
cs_ro — cs.ro
cs_sc — cs.sc
cs_sd — cs.sd
cs_se — cs.se
cs_si — cs.si
cs_sy — cs.sy

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: cs_ai

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`cs_hc`	wrong	13.0%	0.574	`cs_hc` 21.0% `cs_lg` 14.0% `cs_ai` 13.0% `cs_ds` 13.0% `cs_sc` 8.0% `cs_cg` 6.0%
JevFish	`cs_hc`	wrong	7.4%	0.556	`cs_hc` 7.8% `cs_ai` 7.4% `cs_sc` 7.1% `cs_db` 5.1% `cs_sy` 4.4% `cs_si` 4.2%
System One Scorer	`cs_dm`	wrong	0.7%	0.493	`cs_dm` 5.1% `cs_dl` 3.8% `cs_sy` 3.7% `cs_hc` 3.6% `cs_ms` 3.5% `cs_ne` 3.3% `cs_ai` 0.7%
decider-2b	`cs_ai`	correct	10.0%	0.581	`cs_ai` 10.0% `cs_db` 5.6% `cs_cv` 5.4% `cs_ar` 5.1% `cs_hc` 5.1% `cs_cc` 4.9%
OpenJev	`cs_dl`	wrong	3.0%	0.422	`cs_dl` 39.1% `cs_se` 21.4% `cs_os` 9.3% `cs_ms` 5.4% `cs_sd` 3.6% `cs_sy` 3.4% `cs_ai` 3.0%
NanoJev	`cs_hc`	wrong	1.8%	0.504	`cs_hc` 6.8% `cs_lg` 4.4% `cs_ni` 4.3% `cs_na` 3.5% `cs_mm` 3.4% `cs_gl` 3.3% `cs_ai` 1.8%

Sample 4 of 6 · data-and-operations:arxiv-8d72305567dc · 157 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "title": "Comments to a series of works by V.K. Mukhomorov on the theory of a continuum polaron and two-center bipolaron (axially symmetrical quasimolecular dimer)",
  "abstract": "Some critical remarks are made regarding a series of works by V.K. Mukhomorov dealing with polarons and oscillatory and rotational spectrum of a large-radius bipolaron near a subsidiary minimum corresponding to its two-center configuration. It is shown that V.K. Mukhomorov's conclusion that by varying the bipolaron functional one should look for a constrained rather than absolute minimum is erroneous. Consideration of interelectronic correlations corresponding to a direct dependence of the wave function of the studied system on the distance between the electrons does not break the virial theorem."
}

Question archive · choice · primary (ranked)

Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in?

11 options

astro_ph — Astrophysics
cond_mat — Condensed matter physics
cs — Computer science
econ — Economics
eess — Electrical engineering and systems science
math — Mathematics
nlin — Nonlinear sciences
physics — Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...)
q_bio — Quantitative biology
q_fin — Quantitative finance
stat — Statistics

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: cond_mat

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`cond_mat`	correct	96.0%	0.998	`cond_mat` 96.0% `physics` 4.0% `cs` 0.0% `q_fin` 0.0% `stat` 0.0% `math` 0.0%
JevFish	`cond_mat`	correct	47.8%	0.819	`cond_mat` 47.8% `physics` 19.7% `nlin` 19.4% `math` 10.8% `stat` 1.0% `astro_ph` 0.4%
System One Scorer	`cond_mat`	correct	58.0%	0.899	`cond_mat` 58.0% `math` 8.4% `physics` 7.8% `econ` 7.0% `nlin` 5.8% `q_bio` 2.8%
decider-2b	`cond_mat`	correct	51.6%	0.837	`cond_mat` 51.6% `physics` 25.1% `nlin` 16.3% `math` 5.4% `stat` 0.4% `astro_ph` 0.3%
OpenJev	`cond_mat`	correct	97.8%	1.000	`cond_mat` 97.8% `physics` 1.4% `eess` 0.2% `nlin` 0.2% `astro_ph` 0.2% `econ` 0.1%
NanoJev	`physics`	wrong	6.7%	0.449	`physics` 33.8% `stat` 27.8% `math` 19.3% `cond_mat` 6.7% `nlin` 4.0% `q_bio` 2.2%

Question subcategory · choice

Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories.

9 options

cond_mat_dis_nn — cond.mat.dis.nn
cond_mat_mes_hall — cond.mat.mes.hall
cond_mat_mtrl_sci — cond.mat.mtrl.sci
cond_mat_other — cond.mat.other
cond_mat_quant_gas — cond.mat.quant.gas
cond_mat_soft — cond.mat.soft
cond_mat_stat_mech — cond.mat.stat.mech
cond_mat_str_el — cond.mat.str.el
cond_mat_supr_con — cond.mat.supr.con

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: cond_mat_other

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`cond_mat_str_el`	wrong	33.0%	0.634	`cond_mat_str_el` 52.0% `cond_mat_other` 33.0% `cond_mat_dis_nn` 11.0% `cond_mat_stat_mech` 3.0% `cond_mat_supr_con` 1.0% `cond_mat_mes_hall` 0.0%
JevFish	`cond_mat_stat_mech`	wrong	8.0%	0.381	`cond_mat_stat_mech` 61.0% `cond_mat_supr_con` 9.4% `cond_mat_other` 8.0% `cond_mat_str_el` 6.2% `cond_mat_quant_gas` 5.7% `cond_mat_mtrl_sci` 5.7%
System One Scorer	`cond_mat_soft`	wrong	18.0%	0.603	`cond_mat_soft` 22.6% `cond_mat_other` 18.0% `cond_mat_stat_mech` 17.4% `cond_mat_str_el` 12.1% `cond_mat_supr_con` 11.6% `cond_mat_mtrl_sci` 8.2%
decider-2b	`cond_mat_stat_mech`	wrong	5.3%	0.272	`cond_mat_stat_mech` 74.2% `cond_mat_quant_gas` 6.4% `cond_mat_other` 5.3% `cond_mat_mtrl_sci` 5.0% `cond_mat_str_el` 4.6% `cond_mat_soft` 2.8%
OpenJev	`cond_mat_stat_mech`	wrong	24.5%	0.476	`cond_mat_stat_mech` 69.1% `cond_mat_other` 24.5% `cond_mat_str_el` 1.8% `cond_mat_mtrl_sci` 1.6% `cond_mat_quant_gas` 1.2% `cond_mat_soft` 0.8%
NanoJev	`cond_mat_dis_nn`	wrong	8.7%	0.528	`cond_mat_dis_nn` 14.5% `cond_mat_mes_hall` 13.5% `cond_mat_mtrl_sci` 13.1% `cond_mat_str_el` 12.3% `cond_mat_stat_mech` 11.4% `cond_mat_quant_gas` 11.0% `cond_mat_other` 8.7%

Sample 5 of 6 · data-and-operations:arxiv-b0a52a8630a5 · 96 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "title": "Laboratorio di matematica: una sintesi di contenuti e metodologie",
  "abstract": "In this paper, our aim is to discuss the outcome of a ten-year experimental workshop activity carried out by the Center `matematita' (http://www.matematita.it/) with classes of pre-university students of any age. We analyse the potential benefits of this way of teaching and learning mathematics, both from the point of view of students and teachers."
}

Question archive · choice · primary (ranked)

Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in?

11 options

astro_ph — Astrophysics
cond_mat — Condensed matter physics
cs — Computer science
econ — Economics
eess — Electrical engineering and systems science
math — Mathematics
nlin — Nonlinear sciences
physics — Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...)
q_bio — Quantitative biology
q_fin — Quantitative finance
stat — Statistics

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`math`	correct	100.0%	1.000	`math` 100.0% `astro_ph` 0.0% `econ` 0.0% `nlin` 0.0% `q_bio` 0.0% `q_fin` 0.0%
JevFish	`math`	correct	89.9%	0.993	`math` 89.9% `nlin` 4.0% `stat` 3.9% `cs` 0.7% `econ` 0.3% `eess` 0.3%
System One Scorer	`math`	correct	51.1%	0.861	`math` 51.1% `econ` 16.1% `q_bio` 5.8% `q_fin` 5.5% `nlin` 4.4% `stat` 3.8%
decider-2b	`math`	correct	90.6%	0.994	`math` 90.6% `stat` 3.8% `nlin` 2.7% `cs` 0.9% `econ` 0.5% `eess` 0.3%
OpenJev	`math`	correct	99.9%	1.000	`math` 99.9% `nlin` 0.0% `eess` 0.0% `astro_ph` 0.0% `stat` 0.0% `cond_mat` 0.0%
NanoJev	`math`	correct	99.2%	1.000	`math` 99.2% `stat` 0.4% `physics` 0.4% `nlin` 0.0% `q_fin` 0.0% `astro_ph` 0.0%

Question subcategory · choice

Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories.

30 options

math_ac — math.ac
math_ag — math.ag
math_ap — math.ap
math_at — math.at
math_ca — math.ca
math_co — math.co
math_ct — math.ct
math_cv — math.cv
math_dg — math.dg
math_ds — math.ds
math_fa — math.fa
math_gm — math.gm
math_gn — math.gn
math_gr — math.gr
math_gt — math.gt
math_ho — math.ho
math_kt — math.kt
math_lo — math.lo
math_mg — math.mg
math_na — math.na
math_nt — math.nt
math_oa — math.oa
math_oc — math.oc
math_pr — math.pr
math_qa — math.qa
math_ra — math.ra
math_rt — math.rt
math_sg — math.sg
math_sp — math.sp
math_st — math.st

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math_ho

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`math_ho`	correct	25.0%	0.686	`math_ho` 25.0% `math_ac` 20.0% `math_fa` 8.0% `math_ap` 7.0% `math_ct` 6.0% `math_at` 6.0%
JevFish	`math_ac`	wrong	2.4%	0.497	`math_ac` 11.7% `math_st` 11.1% `math_pr` 7.4% `math_ag` 5.7% `math_at` 5.4% `math_ca` 4.6% `math_ho` 2.4%
System One Scorer	`math_ct`	wrong	4.3%	0.522	`math_ct` 7.2% `math_ac` 6.3% `math_st` 6.1% `math_oa` 5.5% `math_lo` 4.9% `math_nt` 4.8% `math_ho` 4.3%
decider-2b	`math_ac`	wrong	2.1%	0.493	`math_ac` 14.1% `math_ag` 7.2% `math_at` 6.5% `math_pr` 6.5% `math_ap` 4.9% `math_ca` 4.9% `math_ho` 2.1%
OpenJev	`math_sp`	wrong	0.1%	0.387	`math_sp` 40.0% `math_sg` 18.0% `math_st` 15.1% `math_rt` 8.5% `math_ca` 5.9% `math_ap` 3.0% `math_ho` 0.1%
NanoJev	`math_qa`	wrong	2.3%	0.504	`math_qa` 5.9% `math_dg` 5.3% `math_gn` 5.1% `math_at` 5.0% `math_ag` 4.9% `math_mg` 4.9% `math_ho` 2.3%

Sample 6 of 6 · data-and-operations:arxiv-b87c1c2f175e · 189 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "title": "Tanaka's equation on the circle and stochastic flows",
  "abstract": "We define a Tanaka's equation on an oriented graph with two edges and two vertices. This graph will be embedded in the unit circle. Extending this equation to flows of kernels, we show that the laws of the flows of kernels $K$ solution of Tanaka's equation can be classified by pairs of probability measures $(m^+,m^-)$ on $[0,1]$, with mean 1/2. What happens at the first vertex is governed by $m^+$, and at the second by $m^-$. For each vertex $P$, we construct a sequence of stopping times along which the image of the whole circle by $K$ is reduced to $P$. We also prove that the supports of these flows contains a finite number of points, and that except for some particular cases this number of points can be arbitrarily large."
}

Question archive · choice · primary (ranked)

Read state.title and state.abstract. Which arXiv archive (top-level subject group) is this paper's primary category in?

11 options

astro_ph — Astrophysics
cond_mat — Condensed matter physics
cs — Computer science
econ — Economics
eess — Electrical engineering and systems science
math — Mathematics
nlin — Nonlinear sciences
physics — Physics (general physics archive: optics, fluids, plasma, atomic, chemical, applied, ...)
q_bio — Quantitative biology
q_fin — Quantitative finance
stat — Statistics

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`math`	correct	99.0%	1.000	`math` 99.0% `nlin` 1.0% `cs` 0.0% `eess` 0.0% `astro_ph` 0.0% `cond_mat` 0.0%
JevFish	`math`	correct	66.1%	0.912	`math` 66.1% `nlin` 23.9% `stat` 6.8% `physics` 0.9% `q_fin` 0.5% `cs` 0.5%
System One Scorer	`math`	correct	36.6%	0.766	`math` 36.6% `nlin` 16.5% `stat` 14.4% `q_fin` 8.7% `econ` 6.1% `cond_mat` 4.8%
decider-2b	`math`	correct	61.7%	0.881	`math` 61.7% `nlin` 30.0% `stat` 4.2% `physics` 1.5% `cs` 0.6% `cond_mat` 0.5%
OpenJev	`math`	correct	99.6%	1.000	`math` 99.6% `stat` 0.2% `nlin` 0.1% `astro_ph` 0.1% `q_bio` 0.0% `eess` 0.0%
NanoJev	`math`	correct	50.8%	0.842	`math` 50.8% `stat` 25.8% `physics` 5.3% `nlin` 3.6% `econ` 3.4% `q_fin` 3.3%

Question subcategory · choice

Read state.title and state.abstract. Within the paper's true archive, which subcategory is its primary category? The options are that archive's subcategories.

30 options

math_ac — math.ac
math_ag — math.ag
math_ap — math.ap
math_at — math.at
math_ca — math.ca
math_co — math.co
math_ct — math.ct
math_cv — math.cv
math_dg — math.dg
math_ds — math.ds
math_fa — math.fa
math_gm — math.gm
math_gn — math.gn
math_gr — math.gr
math_gt — math.gt
math_ho — math.ho
math_kt — math.kt
math_lo — math.lo
math_mg — math.mg
math_na — math.na
math_nt — math.nt
math_oa — math.oa
math_oc — math.oc
math_pr — math.pr
math_qa — math.qa
math_ra — math.ra
math_rt — math.rt
math_sg — math.sg
math_sp — math.sp
math_st — math.st

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: math_pr

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`math_st`	wrong	2.0%	0.376	`math_st` 36.0% `math_ds` 33.0% `math_sp` 22.0% `math_sg` 3.0% `math_pr` 2.0% `math_ra` 2.0%
JevFish	`math_st`	wrong	5.1%	0.515	`math_st` 17.0% `math_ac` 13.3% `math_sp` 7.1% `math_ag` 6.5% `math_pr` 5.1% `math_ca` 4.9%
System One Scorer	`math_st`	wrong	7.8%	0.554	`math_st` 10.7% `math_pr` 7.8% `math_ct` 6.3% `math_oc` 6.0% `math_ag` 5.9% `math_oa` 5.1%
decider-2b	`math_ac`	wrong	2.5%	0.459	`math_ac` 30.4% `math_ag` 14.1% `math_ap` 6.2% `math_ca` 6.2% `math_dg` 5.6% `math_at` 3.8% `math_pr` 2.5%
OpenJev	`math_st`	wrong	0.2%	0.351	`math_st` 44.2% `math_gm` 31.3% `math_at` 6.9% `math_ds` 4.3% `math_ca` 1.5% `math_ct` 1.2% `math_pr` 0.2%
NanoJev	`math_qa`	wrong	3.1%	0.513	`math_qa` 4.8% `math_gn` 4.5% `math_dg` 4.3% `math_gm` 4.2% `math_mg` 4.2% `math_nt` 4.2% `math_pr` 3.1%