Typed Decision Bench · tasks · Retrieval and knowledge

`esci-product-rerank`

semantic reranking · e commerce · 490 items · primary question type noul. Source tasksource/esci (Apache-2.0); labels are crowd labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
System One Scorer	75.5	61.6%	490 / 490
Jev	75.0	73.1%	490 / 490
JevFish	74.4	63.1%	490 / 490
decider-2b	74.2	62.0%	490 / 490
OpenJev	71.8	72.2%	490 / 490
NanoJev	71.1	59.3%	489 / 490

Sample 1 of 6 · retrieval-and-knowledge:esci-113870-B01L7PSJFO · 105 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "query": "workout headphones",
  "product": {
    "title": "Bose SoundSport - Auriculares inalámbricos (Bluetooth, NFC, micrófono), color negro",
    "brand": "Bose",
    "color": "Negro",
    "bullet_points": "Micrófono en el cable para llamadas y música\nControl mediante la aplicación Bose Connect\nEmparejamiento Bluetooth y NFC\nCubiertas StayHear+ para una mayor estabilidad y comodidad\nSonido equilibrado a cualquier volumen"
  }
}

Question is_exact_match · noul · primary (ranked)

The state holds a shopper's search query and one candidate product. Would this product fully satisfy the query as an exact match of what the shopper asked for, rather than a substitute, an accessory, or an unrelated item? Yes only if every specification stated in the query is met.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`yes`	correct	52.8%	0.777	`yes` 52.8% `no` 47.2%
Jev	`yes`	correct	89.0%	0.988	`yes` 89.0% `no` 11.0%
JevFish	`yes`	correct	74.3%	0.934	`yes` 74.3% `no` 25.7%
decider-2b	`yes`	correct	76.9%	0.947	`yes` 76.9% `no` 23.1%
OpenJev	`yes`	correct	86.3%	0.981	`yes` 86.3% `no` 13.7%
NanoJev	`yes`	correct	68.9%	0.903	`yes` 68.9% `no` 31.1%

Question relation · choice

How does the product relate to what the shopper searched for?

4 options

exact — It is what the shopper asked for; every stated specification is met.
substitute — Not an exact match, but it could stand in for the requested item (a different size, brand or spec).
complement — It does not do what was asked but is used together with it (an accessory or add-on).
irrelevant — It is unrelated to the query or misses a central requirement.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: exact

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`exact`	correct	64.5%	0.915	`exact` 64.5% `substitute` 16.0% `complement` 11.5% `irrelevant` 8.0%
Jev	`exact`	correct	99.0%	1.000	`exact` 99.0% `substitute` 1.0% `irrelevant` 0.0% `complement` 0.0%
JevFish	`exact`	correct	75.5%	0.954	`exact` 75.5% `substitute` 16.4% `complement` 7.0% `irrelevant` 1.1%
decider-2b	`exact`	correct	79.9%	0.969	`exact` 79.9% `substitute` 13.5% `complement` 4.9% `irrelevant` 1.7%
OpenJev	`exact`	correct	99.3%	1.000	`exact` 99.3% `substitute` 0.7% `irrelevant` 0.0% `complement` 0.0%
NanoJev	`exact`	correct	84.1%	0.983	`exact` 84.1% `complement` 6.5% `substitute` 5.4% `irrelevant` 3.9%

Sample 2 of 6 · retrieval-and-knowledge:esci-15515-B083FG5PNZ · 116 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "query": "bicycling shorts padded men 3xl",
  "product": {
    "title": "sponeed Cycling Padded Shorts Riding Bottoms Mountain Bike Pants Padding US Large Grey Black",
    "brand": "sponeed",
    "color": "Grey Black",
    "bullet_points": "Size suggestion and washing tip,road bike shorts is runs a bit smaller than US standard, please check the size chart in “product description” picture shown,please reference weight height and waist.Wash it under 100 degrees F,hand washing or machine (Put in wash pocket when washing in a machine).Do …"
  }
}

Question is_exact_match · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`no`	wrong	31.4%	0.529	`yes` 31.4% `no` 68.7%
Jev	`no`	wrong	5.0%	0.098	`yes` 5.0% `no` 95.0%
JevFish	`no`	wrong	36.7%	0.600	`yes` 36.7% `no` 63.3%
decider-2b	`no`	wrong	36.0%	0.590	`yes` 36.0% `no` 64.0%
OpenJev	`no`	wrong	1.8%	0.036	`yes` 1.8% `no` 98.2%
NanoJev	`yes`	correct	71.0%	0.916	`yes` 71.0% `no` 29.0%

Question relation · choice

How does the product relate to what the shopper searched for?

4 options

exact — It is what the shopper asked for; every stated specification is met.
substitute — Not an exact match, but it could stand in for the requested item (a different size, brand or spec).
complement — It does not do what was asked but is used together with it (an accessory or add-on).
irrelevant — It is unrelated to the query or misses a central requirement.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: exact

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`substitute`	wrong	16.8%	0.406	`substitute` 69.5% `exact` 16.8% `irrelevant` 11.0% `complement` 2.7%
Jev	`substitute`	wrong	1.0%	0.020	`substitute` 99.0% `exact` 1.0% `complement` 0.0% `irrelevant` 0.0%
JevFish	`substitute`	wrong	31.9%	0.589	`substitute` 59.5% `exact` 31.9% `complement` 4.8% `irrelevant` 3.9%
decider-2b	`substitute`	wrong	42.7%	0.701	`substitute` 51.7% `exact` 42.7% `irrelevant` 3.7% `complement` 1.9%
OpenJev	`substitute`	wrong	1.6%	0.032	`substitute` 98.3% `exact` 1.6% `irrelevant` 0.1% `complement` 0.0%
NanoJev	`exact`	correct	83.5%	0.981	`exact` 83.5% `substitute` 6.9% `complement` 6.4% `irrelevant` 3.3%

Sample 3 of 6 · retrieval-and-knowledge:esci-2915-B07PLN1N24 · 207 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "query": "21st birthday sash and tiara",
  "product": {
    "title": "21st Birthday, Finally 21 Birthday, 21st Birthday Gifts for Her, 21 Tiara Pink, 21st Birthday Crown, 21 Birthday Tiara, 21 Sash and Tiara, 21st Crown, 21st Birthday Decorations, 21 Birthday Sash",
    "brand": "MEANT2TOBE",
    "color": "Pink",
    "bullet_points": "Beautiful elegant pink satin sash: “FINALLY 21”, Dimension: 27\"- 32\" and 3.75\" wide\nDouble pink glittery 21st birthday tiara/crown: 100% eco-friendly alloy metal，crystals and rhinestones, nickel free and anti-allergic, Dimension: 5.12” on the front side, the size of number 21 is: 1.07” x …"
  }
}

Question is_exact_match · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`yes`	correct	62.5%	0.859	`yes` 62.5% `no` 37.5%
Jev	`yes`	correct	90.0%	0.990	`yes` 90.0% `no` 10.0%
JevFish	`yes`	correct	77.7%	0.950	`yes` 77.7% `no` 22.3%
decider-2b	`yes`	correct	72.3%	0.923	`yes` 72.3% `no` 27.7%
OpenJev	`yes`	correct	98.0%	1.000	`yes` 98.0% `no` 2.0%
NanoJev	`yes`	correct	58.9%	0.831	`yes` 58.9% `no` 41.1%

Question relation · choice

How does the product relate to what the shopper searched for?

4 options

exact — It is what the shopper asked for; every stated specification is met.
substitute — Not an exact match, but it could stand in for the requested item (a different size, brand or spec).
complement — It does not do what was asked but is used together with it (an accessory or add-on).
irrelevant — It is unrelated to the query or misses a central requirement.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: exact

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`exact`	correct	78.3%	0.968	`exact` 78.3% `substitute` 7.8% `complement` 7.8% `irrelevant` 6.1%
Jev	`exact`	correct	100.0%	1.000	`exact` 100.0% `substitute` 0.0% `complement` 0.0% `irrelevant` 0.0%
JevFish	`exact`	correct	83.5%	0.980	`exact` 83.5% `substitute` 10.3% `complement` 5.0% `irrelevant` 1.3%
decider-2b	`exact`	correct	82.7%	0.978	`exact` 82.7% `substitute` 11.5% `complement` 3.6% `irrelevant` 2.1%
OpenJev	`exact`	correct	99.7%	1.000	`exact` 99.7% `substitute` 0.3% `irrelevant` 0.0% `complement` 0.0%
NanoJev	`exact`	correct	62.8%	0.908	`exact` 62.8% `complement` 13.4% `substitute` 12.7% `irrelevant` 11.1%

Sample 4 of 6 · retrieval-and-knowledge:esci-71799-B01LEFGMAQ · 56 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "query": "my amazon music playlist",
  "product": {
    "title": "Relaxing Records",
    "brand": "Polypack-uk Ltd",
    "color": "",
    "bullet_points": "music to help with studying or working\nmusic to help with sleeping problems or stress\nmusic to help with meditation and yoga"
  }
}

Question is_exact_match · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`no`	wrong	32.5%	0.544	`yes` 32.5% `no` 67.5%
Jev	`no`	wrong	4.0%	0.078	`yes` 4.0% `no` 96.0%
JevFish	`no`	wrong	7.9%	0.151	`yes` 7.9% `no` 92.1%
decider-2b	`no`	wrong	10.3%	0.196	`yes` 10.3% `no` 89.7%
OpenJev	`no`	wrong	0.0%	0.000	`yes` 0.0% `no` 100.0%
NanoJev	`yes`	correct	73.4%	0.929	`yes` 73.4% `no` 26.6%

Question relation · choice

How does the product relate to what the shopper searched for?

4 options

exact — It is what the shopper asked for; every stated specification is met.
substitute — Not an exact match, but it could stand in for the requested item (a different size, brand or spec).
complement — It does not do what was asked but is used together with it (an accessory or add-on).
irrelevant — It is unrelated to the query or misses a central requirement.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: exact

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`irrelevant`	wrong	20.9%	0.552	`irrelevant` 36.9% `substitute` 36.2% `exact` 20.9% `complement` 6.1%
Jev	`irrelevant`	wrong	1.0%	0.174	`irrelevant` 81.0% `substitute` 10.0% `complement` 8.0% `exact` 1.0%
JevFish	`irrelevant`	wrong	1.4%	0.259	`irrelevant` 67.6% `substitute` 20.4% `complement` 10.6% `exact` 1.4%
decider-2b	`irrelevant`	wrong	1.4%	0.213	`irrelevant` 75.6% `substitute` 15.5% `complement` 7.5% `exact` 1.4%
OpenJev	`irrelevant`	wrong	0.0%	0.000	`irrelevant` 100.0% `substitute` 0.0% `exact` 0.0% `complement` 0.0%
NanoJev	`exact`	correct	84.9%	0.985	`exact` 84.9% `complement` 6.3% `substitute` 5.4% `irrelevant` 3.4%

Sample 5 of 6 · retrieval-and-knowledge:esci-8411-B000DZGHHC · 97 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "query": "aluminum sheet",
  "product": {
    "title": "AMERIMAX HOME PRODUCTS 66320 20-Inch x 10-Feet Aluminum Flashing",
    "brand": "Amerimax Home Products",
    "color": "",
    "bullet_points": "The product is 20\"x10' alum flashing\nEasy and simple use kit\nThe product is manufactured in china\nComposed of durable, refurbished aluminum\nQuick, easy repair for roofs and gutters"
  }
}

Question is_exact_match · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`no`	correct	61.0%	0.848	`yes` 39.0% `no` 61.0%
Jev	`no`	correct	77.0%	0.947	`yes` 23.0% `no` 77.0%
JevFish	`yes`	wrong	49.3%	0.743	`yes` 50.7% `no` 49.3%
decider-2b	`no`	correct	61.8%	0.854	`yes` 38.2% `no` 61.8%
OpenJev	`no`	correct	82.0%	0.968	`yes` 18.0% `no` 82.0%
NanoJev	`yes`	wrong	23.4%	0.413	`yes` 76.6% `no` 23.4%

Question relation · choice

How does the product relate to what the shopper searched for?

4 options

exact — It is what the shopper asked for; every stated specification is met.
substitute — Not an exact match, but it could stand in for the requested item (a different size, brand or spec).
complement — It does not do what was asked but is used together with it (an accessory or add-on).
irrelevant — It is unrelated to the query or misses a central requirement.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: substitute

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`substitute`	correct	39.2%	0.737	`substitute` 39.2% `exact` 34.8% `irrelevant` 16.2% `complement` 9.8%
Jev	`substitute`	correct	88.0%	0.990	`substitute` 88.0% `exact` 5.0% `complement` 4.0% `irrelevant` 3.0%
JevFish	`substitute`	correct	45.7%	0.772	`substitute` 45.7% `exact` 38.1% `complement` 11.4% `irrelevant` 4.8%
decider-2b	`substitute`	correct	56.9%	0.857	`substitute` 56.9% `exact` 30.5% `complement` 7.2% `irrelevant` 5.4%
OpenJev	`substitute`	correct	86.9%	0.983	`substitute` 86.9% `exact` 12.8% `irrelevant` 0.3% `complement` 0.0%
NanoJev	`exact`	wrong	6.1%	0.209	`exact` 83.3% `complement` 6.6% `substitute` 6.1% `irrelevant` 4.0%

Sample 6 of 6 · retrieval-and-knowledge:esci-8430-B07YJ7F2S3 · 80 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "query": "always hungry shirt men",
  "product": {
    "title": "Always Hungry Funny Men's T-Shirt (Black Berry, Small)",
    "brand": "CLOTHING WORLD",
    "color": "Black Berry",
    "bullet_points": "Seamless rib at neck\nTaped shoulder-to-shoulder; Double-needle stitching throughout\nQuarter-turned to eliminate center crease\nClassic fit; 7/8\" collar"
  }
}

Question is_exact_match · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`yes`	correct	55.2%	0.799	`yes` 55.2% `no` 44.8%
Jev	`yes`	correct	81.0%	0.964	`yes` 81.0% `no` 19.0%
JevFish	`yes`	correct	53.4%	0.783	`yes` 53.4% `no` 46.6%
decider-2b	`yes`	correct	58.3%	0.826	`yes` 58.3% `no` 41.7%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%
NanoJev	`yes`	correct	70.7%	0.914	`yes` 70.7% `no` 29.3%

Question relation · choice

How does the product relate to what the shopper searched for?

4 options

exact — It is what the shopper asked for; every stated specification is met.
substitute — Not an exact match, but it could stand in for the requested item (a different size, brand or spec).
complement — It does not do what was asked but is used together with it (an accessory or add-on).
irrelevant — It is unrelated to the query or misses a central requirement.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: exact

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`exact`	correct	68.0%	0.925	`exact` 68.0% `substitute` 19.9% `irrelevant` 9.1% `complement` 2.9%
Jev	`exact`	correct	89.0%	0.988	`exact` 89.0% `substitute` 11.0% `irrelevant` 0.0% `complement` 0.0%
JevFish	`exact`	correct	57.6%	0.866	`exact` 57.6% `substitute` 27.3% `irrelevant` 11.5% `complement` 3.6%
decider-2b	`exact`	correct	66.1%	0.914	`exact` 66.1% `substitute` 20.8% `irrelevant` 11.2% `complement` 1.9%
OpenJev	`exact`	correct	100.0%	1.000	`exact` 100.0% `substitute` 0.0% `irrelevant` 0.0% `complement` 0.0%
NanoJev	`exact`	correct	85.0%	0.985	`exact` 85.0% `complement` 6.6% `substitute` 4.3% `irrelevant` 4.1%