Typed Decision Bench · tasks · Retrieval and knowledge

`esci-product-rerank`

semantic reranking · e commerce · 490 items · primary question type noul. Source tasksource/esci (Apache-2.0); labels are crowd labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
System One Scorer	75.5	61.6%	490 / 490
Jev	75.0	73.1%	490 / 490
decider-2b	74.2	62.0%	490 / 490
OpenJev	71.8	72.2%	490 / 490
NanoJev	71.1	59.3%	489 / 490
JevFish	68.2	57.1%	490 / 490

Sample 1 of 6 · retrieval-and-knowledge:esci-111035-B00S1TC442 · 132 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "query": "white comforter",
  "product": {
    "title": "Utopia Bedding Comforter Duvet Insert - Quilted Comforter with Corner Tabs - Box Stitched Down Alternative Comforter (Queen, White)",
    "brand": "Utopia Bedding",
    "color": "White",
    "bullet_points": "COMFORTER DUVET INSERT - Queen comforter duvet insert measures 88 inches x 88 inches\nBOX STYLE STITCHING - Crisp looking comforter with siliconized fiber filling features piped edges with an elegant style box stitching that prevent the fill from shifting\nSILICONIZED FIBERFILL - Extremely soft …"
  }
}

Question is_exact_match · noul · primary (ranked)

The state holds a shopper's search query and one candidate product. Would this product fully satisfy the query as an exact match of what the shopper asked for, rather than a substitute, an accessory, or an unrelated item? Yes only if every specification stated in the query is met.

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`yes`	correct	62.7%	0.861	`yes` 62.7% `no` 37.3%
Jev	`yes`	correct	91.0%	0.992	`yes` 91.0% `no` 9.0%
decider-2b	`yes`	correct	79.3%	0.957	`yes` 79.3% `no` 20.6%
OpenJev	`yes`	correct	79.4%	0.958	`yes` 79.4% `no` 20.6%
NanoJev	`yes`	correct	72.3%	0.923	`yes` 72.3% `no` 27.7%
JevFish	`yes`	correct	70.6%	0.914	`yes` 70.6% `no` 29.4%

Question relation · choice

How does the product relate to what the shopper searched for?

4 options

exact — It is what the shopper asked for; every stated specification is met.
substitute — Not an exact match, but it could stand in for the requested item (a different size, brand or spec).
complement — It does not do what was asked but is used together with it (an accessory or add-on).
irrelevant — It is unrelated to the query or misses a central requirement.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: exact

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`exact`	correct	67.5%	0.919	`exact` 67.5% `substitute` 22.7% `irrelevant` 6.5% `complement` 3.3%
Jev	`exact`	correct	99.0%	1.000	`exact` 99.0% `substitute` 1.0% `irrelevant` 0.0% `complement` 0.0%
decider-2b	`exact`	correct	87.5%	0.987	`exact` 87.5% `substitute` 10.1% `complement` 1.2% `irrelevant` 1.2%
OpenJev	`exact`	correct	94.5%	0.997	`exact` 94.5% `substitute` 5.4% `irrelevant` 0.1% `complement` 0.0%
NanoJev	`exact`	correct	80.1%	0.973	`exact` 80.1% `complement` 8.1% `substitute` 6.3% `irrelevant` 5.4%
JevFish	`exact`	correct	51.1%	0.831	`exact` 51.1% `substitute` 27.2% `irrelevant` 13.8% `complement` 7.9%

Sample 2 of 6 · retrieval-and-knowledge:esci-15246-B07S8YVD2N · 118 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "query": "bettervent",
  "product": {
    "title": "Supplying Demand 211 Indoor Electric Clothes Dryer Vent Kit For Small Living Spaces With Poor Venting",
    "brand": "Supplying Demand",
    "color": "",
    "bullet_points": "A quick easy way to save your dryer and electric bill\nSnap together and add a half cup of water at the bottom to catch the lint\nNOTE: Do not use with gas dryers\nMultiple mounting options\nIncludes circular lint trap (must empty every few loads), 4-inch by 5-foot flex duct, 2 clamps, and a mounting …"
  }
}

Question is_exact_match · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`no`	correct	60.3%	0.842	`yes` 39.8% `no` 60.3%
Jev	`no`	correct	87.0%	0.983	`yes` 13.0% `no` 87.0%
decider-2b	`no`	correct	66.2%	0.886	`yes` 33.8% `no` 66.2%
OpenJev	`no`	correct	90.6%	0.991	`yes` 9.3% `no` 90.6%
NanoJev	`yes`	wrong	21.6%	0.385	`yes` 78.4% `no` 21.6%
JevFish	`no`	correct	55.4%	0.801	`yes` 44.6% `no` 55.4%

Question relation · choice

How does the product relate to what the shopper searched for?

4 options

exact — It is what the shopper asked for; every stated specification is met.
substitute — Not an exact match, but it could stand in for the requested item (a different size, brand or spec).
complement — It does not do what was asked but is used together with it (an accessory or add-on).
irrelevant — It is unrelated to the query or misses a central requirement.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: complement

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`substitute`	wrong	17.2%	0.542	`substitute` 29.2% `exact` 27.3% `irrelevant` 26.3% `complement` 17.2%
Jev	`substitute`	wrong	2.0%	0.299	`substitute` 59.0% `irrelevant` 29.0% `exact` 10.0% `complement` 2.0%
decider-2b	`substitute`	wrong	7.5%	0.396	`substitute` 44.3% `exact` 38.3% `irrelevant` 10.0% `complement` 7.5%
OpenJev	`substitute`	wrong	23.1%	0.560	`substitute` 42.0% `exact` 33.5% `complement` 23.1% `irrelevant` 1.4%
NanoJev	`exact`	wrong	6.7%	0.202	`exact` 84.9% `complement` 6.7% `substitute` 5.1% `irrelevant` 3.4%
JevFish	`substitute`	wrong	25.8%	0.619	`substitute` 38.6% `complement` 25.8% `irrelevant` 18.6% `exact` 17.0%

Sample 3 of 6 · retrieval-and-knowledge:esci-16237-B00E58JQA8 · 111 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "query": "black & white striped legging",
  "product": {
    "title": "Simplicity Women's Soft Black White Horizontal Striped Leggings White/Black Pockets, Black/White",
    "brand": "Simplicity",
    "color": "1-black/White",
    "bullet_points": "Soft cotton/poly blend fabric - Womens Leggings are high quality, soft, shaping and breathable. The leggings are opaque enough that you never have to worry about embarrassing see through\nHigh waist - Elastic band for comfortable fit, more comfortable than wearing jeans; with a reinforced, super …"
  }
}

Question is_exact_match · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`yes`	correct	65.8%	0.883	`yes` 65.8% `no` 34.2%
Jev	`yes`	correct	80.0%	0.960	`yes` 80.0% `no` 20.0%
decider-2b	`yes`	correct	71.4%	0.918	`yes` 71.4% `no` 28.6%
OpenJev	`yes`	correct	93.8%	0.996	`yes` 93.8% `no` 6.3%
NanoJev	`yes`	correct	69.0%	0.904	`yes` 69.0% `no` 31.0%
JevFish	`yes`	correct	71.4%	0.918	`yes` 71.4% `no` 28.6%

Question relation · choice

How does the product relate to what the shopper searched for?

4 options

exact — It is what the shopper asked for; every stated specification is met.
substitute — Not an exact match, but it could stand in for the requested item (a different size, brand or spec).
complement — It does not do what was asked but is used together with it (an accessory or add-on).
irrelevant — It is unrelated to the query or misses a central requirement.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: exact

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`exact`	correct	75.2%	0.953	`exact` 75.2% `substitute` 17.0% `irrelevant` 6.3% `complement` 1.6%
Jev	`exact`	correct	84.0%	0.974	`exact` 84.0% `substitute` 16.0% `irrelevant` 0.0% `complement` 0.0%
decider-2b	`exact`	correct	87.8%	0.988	`exact` 87.8% `substitute` 9.2% `irrelevant` 2.3% `complement` 0.8%
OpenJev	`exact`	correct	95.8%	0.998	`exact` 95.8% `substitute` 4.1% `irrelevant` 0.1% `complement` 0.0%
NanoJev	`exact`	correct	80.0%	0.973	`exact` 80.0% `complement` 8.3% `substitute` 7.2% `irrelevant` 4.6%
JevFish	`exact`	correct	75.9%	0.960	`exact` 75.9% `substitute` 11.5% `irrelevant` 7.7% `complement` 4.9%

Sample 4 of 6 · retrieval-and-knowledge:esci-6619-B07RWF8NZ3 · 128 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "query": "adhesive patch",
  "product": {
    "title": "Lexcam Adhesive Waterproof Patches – (25-Pack) – Pre-Cut for Omnipod Hypoallergenic CGM Tape - Color Clear",
    "brand": "Lexcam",
    "color": "Clear",
    "bullet_points": "✔ If You are Active Person: This product is for you. It last twice as long, maintain appearance and cause less irritation than any other product\n✔ Patches Don't Pell Off: like the bandage ones. Clear film is thin and comfortable.\n✔ Covers & Protects: Hypoallergenic adhesive & Latex-Free overtape …"
  }
}

Question is_exact_match · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`yes`	correct	54.5%	0.793	`yes` 54.5% `no` 45.5%
Jev	`yes`	correct	62.0%	0.856	`yes` 62.0% `no` 38.0%
decider-2b	`yes`	correct	54.8%	0.796	`yes` 54.8% `no` 45.2%
OpenJev	`yes`	correct	69.6%	0.908	`yes` 69.6% `no` 30.4%
NanoJev	`yes`	correct	78.1%	0.952	`yes` 78.1% `no` 21.9%
JevFish	`yes`	correct	53.4%	0.783	`yes` 53.4% `no` 46.6%

Question relation · choice

How does the product relate to what the shopper searched for?

4 options

exact — It is what the shopper asked for; every stated specification is met.
substitute — Not an exact match, but it could stand in for the requested item (a different size, brand or spec).
complement — It does not do what was asked but is used together with it (an accessory or add-on).
irrelevant — It is unrelated to the query or misses a central requirement.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: exact

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`exact`	correct	58.8%	0.883	`exact` 58.8% `substitute` 20.4% `complement` 12.2% `irrelevant` 8.7%
Jev	`exact`	correct	67.0%	0.900	`exact` 67.0% `substitute` 30.0% `complement` 3.0% `irrelevant` 0.0%
decider-2b	`exact`	correct	63.6%	0.885	`exact` 63.6% `substitute` 30.9% `irrelevant` 4.1% `complement` 1.4%
OpenJev	`exact`	correct	54.7%	0.796	`exact` 54.7% `substitute` 45.1% `irrelevant` 0.1% `complement` 0.1%
NanoJev	`exact`	correct	90.9%	0.994	`exact` 90.9% `complement` 4.0% `substitute` 3.2% `irrelevant` 1.9%
JevFish	`substitute`	wrong	37.2%	0.715	`substitute` 37.8% `exact` 37.2% `irrelevant` 15.1% `complement` 9.9%

Sample 5 of 6 · retrieval-and-knowledge:esci-6696-B07KT3VV67 · 107 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "query": "adidas men's essentials 3-stripes pants",
  "product": {
    "title": "adidas Men's Tall Size Essentials 3-Stripes Tricot Pants, Black/Scarlet, XX-Large",
    "brand": "adidas",
    "color": "Black/Scarlet",
    "bullet_points": "Open-hem workout pants with an elastic waist for easy movement\nRegular fit strikes a comfortable balance between loose and snug\nDrawcord on elastic waist lets you customize your fit\nThese pants are made with recycled polyester to save resources and decrease emissions"
  }
}

Question is_exact_match · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`yes`	correct	53.0%	0.779	`yes` 53.0% `no` 47.0%
Jev	`yes`	correct	59.0%	0.832	`yes` 59.0% `no` 41.0%
decider-2b	`yes`	correct	72.3%	0.923	`yes` 72.3% `no` 27.7%
OpenJev	`yes`	correct	88.4%	0.987	`yes` 88.4% `no` 11.6%
NanoJev	`yes`	correct	70.7%	0.914	`yes` 70.7% `no` 29.3%
JevFish	`no`	wrong	44.8%	0.695	`yes` 44.8% `no` 55.2%

Question relation · choice

How does the product relate to what the shopper searched for?

4 options

exact — It is what the shopper asked for; every stated specification is met.
substitute — Not an exact match, but it could stand in for the requested item (a different size, brand or spec).
complement — It does not do what was asked but is used together with it (an accessory or add-on).
irrelevant — It is unrelated to the query or misses a central requirement.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: exact

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`exact`	correct	62.9%	0.892	`exact` 62.9% `substitute` 26.5% `irrelevant` 8.3% `complement` 2.3%
Jev	`exact`	correct	50.0%	0.750	`substitute` 50.0% `exact` 50.0% `irrelevant` 0.0% `complement` 0.0%
decider-2b	`exact`	correct	88.5%	0.990	`exact` 88.5% `substitute` 8.4% `irrelevant` 2.3% `complement` 0.8%
OpenJev	`exact`	correct	94.0%	0.997	`exact` 94.0% `substitute` 5.9% `complement` 0.0% `irrelevant` 0.0%
NanoJev	`exact`	correct	81.7%	0.977	`exact` 81.7% `complement` 7.5% `substitute` 6.8% `irrelevant` 4.0%
JevFish	`substitute`	wrong	31.0%	0.645	`substitute` 42.8% `exact` 31.0% `irrelevant` 22.2% `complement` 3.9%

Sample 6 of 6 · retrieval-and-knowledge:esci-8050-B07RWRYMDS · 111 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "query": "all about me posters",
  "product": {
    "title": "All About Me Poster - 30 Color-Me Activities - Kids Educational Posters, Preschool, Student of the Week and Elementary Classroom",
    "brand": "Fun Express",
    "color": "White",
    "bullet_points": "From favorite foods to future career aspirations students can share all about themselves with this fun activity\nA great addition to kids’ crafts these posters are designed with dandy doodles of cats clouds and so much more - making for an engaging and exciting coloring project\nThis is a great way …"
  }
}

Question is_exact_match · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`yes`	correct	51.5%	0.765	`yes` 51.5% `no` 48.5%
Jev	`yes`	correct	84.0%	0.974	`yes` 84.0% `no` 16.0%
decider-2b	`no`	wrong	26.7%	0.463	`yes` 26.7% `no` 73.3%
OpenJev	`yes`	correct	98.8%	1.000	`yes` 98.8% `no` 1.3%
NanoJev	`yes`	correct	76.6%	0.945	`yes` 76.6% `no` 23.4%
JevFish	`no`	wrong	31.5%	0.531	`yes` 31.5% `no` 68.5%

Question relation · choice

How does the product relate to what the shopper searched for?

4 options

exact — It is what the shopper asked for; every stated specification is met.
substitute — Not an exact match, but it could stand in for the requested item (a different size, brand or spec).
complement — It does not do what was asked but is used together with it (an accessory or add-on).
irrelevant — It is unrelated to the query or misses a central requirement.

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: exact

System	Top pick	Grade	P(expected)	Proper score	Distribution
System One Scorer	`exact`	correct	65.3%	0.913	`exact` 65.3% `substitute` 20.5% `irrelevant` 10.2% `complement` 4.1%
Jev	`exact`	correct	99.0%	1.000	`exact` 99.0% `substitute` 1.0% `complement` 0.0% `irrelevant` 0.0%
decider-2b	`exact`	correct	64.0%	0.900	`exact` 64.0% `substitute` 24.5% `irrelevant` 9.8% `complement` 1.7%
OpenJev	`exact`	correct	99.8%	1.000	`exact` 99.8% `substitute` 0.2% `irrelevant` 0.0% `complement` 0.0%
NanoJev	`exact`	correct	86.1%	0.987	`exact` 86.1% `complement` 6.0% `substitute` 4.9% `irrelevant` 3.1%
JevFish	`irrelevant`	wrong	5.7%	0.384	`irrelevant` 48.1% `substitute` 28.7% `complement` 17.5% `exact` 5.7%