Typed Decision Bench · tasks · Retrieval and knowledge

`abt-buy-product-match`

entity alignment · e commerce · 200 items · primary question type score. Source matchbench/Abt-Buy (CC (variant unconfirmed on the source page) — FLAGGED); labels are expert labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
Jev	94.9	93.0%	200 / 200
OpenJev	92.9	92.0%	200 / 200
System One Scorer	85.4	86.5%	200 / 200
decider-2b	82.7	90.5%	200 / 200
JevFish	82.7	88.5%	200 / 200
NanoJev	74.5	66.5%	200 / 200

Sample 1 of 6 · retrieval-and-knowledge:abtbuy-1024 · 193 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "record_a": {
    "source": "abt",
    "name": "samsung 67 ' black widescreen series 7 led dlp hdtv hl67a750",
    "description": "samsung 67 ' black widescreen series 7 led dlp hdtv hl67a750 1920 x 1080p resolution 10,000:1 contrast ratio digital noise reduction cinemapure color engine get the full effect of 3d led backlight ntsc/atsc/clear qam built-in tuners 60,000 hours lamp life black finish",
    "price": null
  },
  "record_b": {
    "source": "buy",
    "name": "samsung hl72a650 72 ' widescreen 1080p dlp hdtv 2,500:1 contrast ratio",
    "description": "",
    "price": "2188.7"
  }
}

Question link_state · score · primary (ranked)

The state holds two product records, record_a from retailer abt and record_b from retailer buy (name, description, price). Rate how the two records relate as products.

3 options

0 — Two different products (different model or item, or only the same brand/category).
1 — Closely related products that may or may not be the same one: a colour or bundle variant, or a name that could refer to either.
2 — One and the same product (same manufacturer model, possibly worded or abbreviated differently).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: level 0: Two different products (different model or item, or only the same brand/category).

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`0`	correct	98.0%	1.000	`0` 98.0% `1` 2.0% `2` 0.0%
OpenJev	`0`	correct	99.1%	1.000	`0` 99.1% `1` 0.6% `2` 0.3%
System One Scorer	`0`	correct	61.0%	0.884	`0` 61.0% `1` 24.1% `2` 14.9%
decider-2b	`0`	correct	52.7%	0.830	`0` 52.7% `1` 28.8% `2` 18.5%
JevFish	`0`	correct	59.2%	0.872	`0` 59.2% `1` 25.7% `2` 15.1%
NanoJev	`0`	correct	45.8%	0.769	`0` 45.8% `1` 17.0% `2` 37.3%

Sample 2 of 6 · retrieval-and-knowledge:abtbuy-1337 · 125 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "record_a": {
    "source": "abt",
    "name": "garmin streetpilot c series vehicle suction mount black finish 0101060900",
    "description": "garmin streetpilot c series vehicle suction mount 0101060900 dashboard or friction mounting options are recommended except california 12/24 volt adapter",
    "price": null
  },
  "record_b": {
    "source": "buy",
    "name": "garmin suction cup mount/12-volt adapter kit 010-10979-00",
    "description": "navigator starter kit",
    "price": null
  }
}

Question link_state · score · primary (ranked)

The state holds two product records, record_a from retailer abt and record_b from retailer buy (name, description, price). Rate how the two records relate as products.

3 options

0 — Two different products (different model or item, or only the same brand/category).
1 — Closely related products that may or may not be the same one: a colour or bundle variant, or a name that could refer to either.
2 — One and the same product (same manufacturer model, possibly worded or abbreviated differently).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: level 0: Two different products (different model or item, or only the same brand/category).

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`0`	correct	51.0%	0.782	`0` 51.0% `1` 44.0% `2` 5.0%
OpenJev	`1`	wrong	17.7%	0.448	`0` 17.7% `1` 62.1% `2` 20.2%
System One Scorer	`2`	wrong	10.0%	0.358	`0` 10.0% `1` 26.4% `2` 63.6%
decider-2b	`2`	wrong	32.0%	0.640	`0` 32.0% `1` 22.5% `2` 45.5%
JevFish	`0`	correct	65.4%	0.906	`0` 65.4% `1` 23.8% `2` 10.8%
NanoJev	`0`	correct	51.0%	0.796	`0` 51.0% `1` 8.8% `2` 40.2%

Sample 3 of 6 · retrieval-and-knowledge:abtbuy-1791 · 179 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "record_a": {
    "source": "abt",
    "name": "sony black cyber-shot digital camera dscw150b",
    "description": "sony dscw150 black cyber-shot digital camera dscw150b 8.1 megapixels 2.7 ' lcd display 15 mb internal memory carl zeiss 5x optical zoom lens 16:9 high-resolution mode 9-point auto focus automatic macro shooting high sensitivity mode burst mode face detection intelligent scene recognition mode …",
    "price": null
  },
  "record_b": {
    "source": "buy",
    "name": "sony cyber-shot dsc-t300 digital camera black dsct300/b",
    "description": "10.1 megapixel 16:9 2x digital zoom 3.5 ' active matrix tft color lcd",
    "price": "273.95"
  }
}

Question link_state · score · primary (ranked)

The state holds two product records, record_a from retailer abt and record_b from retailer buy (name, description, price). Rate how the two records relate as products.

3 options

0 — Two different products (different model or item, or only the same brand/category).
1 — Closely related products that may or may not be the same one: a colour or bundle variant, or a name that could refer to either.
2 — One and the same product (same manufacturer model, possibly worded or abbreviated differently).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: level 0: Two different products (different model or item, or only the same brand/category).

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`0`	correct	100.0%	1.000	`0` 100.0% `1` 0.0% `2` 0.0%
OpenJev	`0`	correct	99.1%	1.000	`0` 99.1% `1` 0.4% `2` 0.4%
System One Scorer	`0`	correct	52.9%	0.832	`0` 52.9% `1` 20.0% `2` 27.2%
decider-2b	`0`	correct	45.6%	0.778	`0` 45.6% `1` 24.9% `2` 29.4%
JevFish	`0`	correct	58.7%	0.868	`0` 58.7% `1` 27.3% `2` 14.0%
NanoJev	`0`	correct	44.6%	0.760	`0` 44.6% `1` 18.1% `2` 37.4%

Sample 4 of 6 · retrieval-and-knowledge:abtbuy-639 · 127 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "record_a": {
    "source": "abt",
    "name": "lg 30 ' stainless steel freestanding gas range lrg30357ss",
    "description": "lg 30 ' stainless steel freestanding gas range lrg30357ss 5.0 cu . ft. capacity oven superboil burner 5 burner system smoothtouch controls warming drawer electronic ignition system one piece recessed cooktop self-cleaning oven stainless steel finish",
    "price": null
  },
  "record_b": {
    "source": "buy",
    "name": "lg 5.0 cu.ft . freestanding gas range",
    "description": "",
    "price": null
  }
}

Question link_state · score · primary (ranked)

The state holds two product records, record_a from retailer abt and record_b from retailer buy (name, description, price). Rate how the two records relate as products.

3 options

0 — Two different products (different model or item, or only the same brand/category).
1 — Closely related products that may or may not be the same one: a colour or bundle variant, or a name that could refer to either.
2 — One and the same product (same manufacturer model, possibly worded or abbreviated differently).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: level 0: Two different products (different model or item, or only the same brand/category).

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`2`	wrong	1.0%	0.189	`0` 1.0% `1` 22.0% `2` 77.0%
OpenJev	`2`	wrong	0.1%	0.025	`0` 0.1% `1` 2.4% `2` 97.5%
System One Scorer	`2`	wrong	7.9%	0.272	`0` 7.9% `1` 15.7% `2` 76.3%
decider-2b	`2`	wrong	22.3%	0.508	`0` 22.3% `1` 19.4% `2` 58.4%
JevFish	`2`	wrong	36.7%	0.695	`0` 36.7% `1` 25.0% `2` 38.3%
NanoJev	`0`	correct	50.9%	0.797	`0` 50.9% `1` 9.8% `2` 39.3%

Sample 5 of 6 · retrieval-and-knowledge:abtbuy-680 · 122 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "record_a": {
    "source": "abt",
    "name": "sirius sportster 4 with car kit sp4tk1",
    "description": "sirius sportster 4 with car kit sp4tk1 replays 44 minutes of content large easy-to-read 6-line display 30 channel presets game alerts and game zone features one touch jump",
    "price": null
  },
  "record_b": {
    "source": "buy",
    "name": "directed electronics sportster 4 sirius satellite radio receiver sp4tk1",
    "description": "xs fm transmitter 6 lines lcd",
    "price": "169.99"
  }
}

Question link_state · score · primary (ranked)

The state holds two product records, record_a from retailer abt and record_b from retailer buy (name, description, price). Rate how the two records relate as products.

3 options

0 — Two different products (different model or item, or only the same brand/category).
1 — Closely related products that may or may not be the same one: a colour or bundle variant, or a name that could refer to either.
2 — One and the same product (same manufacturer model, possibly worded or abbreviated differently).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: level 2: One and the same product (same manufacturer model, possibly worded or abbreviated differently).

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`2`	correct	91.0%	0.993	`0` 1.0% `1` 8.0% `2` 91.0%
OpenJev	`2`	correct	99.2%	1.000	`0` 0.0% `1` 0.8% `2` 99.2%
System One Scorer	`2`	correct	85.7%	0.983	`0` 3.0% `1` 11.3% `2` 85.7%
decider-2b	`2`	correct	60.3%	0.881	`0` 17.8% `1` 21.9% `2` 60.3%
JevFish	`2`	correct	36.6%	0.698	`0` 34.0% `1` 29.3% `2` 36.6%
NanoJev	`0`	wrong	37.3%	0.653	`0` 54.2% `1` 8.5% `2` 37.3%

Sample 6 of 6 · retrieval-and-knowledge:abtbuy-768 · 215 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "record_a": {
    "source": "abt",
    "name": "samsung 32 ' series 6 lcd black flat panel hdtv ln32a650",
    "description": "samsung 32 ' series 6 lcd black flat panel hdtv ln32a650 full 1080p high definition resolution 16:9 widescreen aspect ratio 50,000:1 dynamic contrast ratio srs trusurround xt audio built-in digital tuner ( atsc/clear qam ) picture-in-picture auto volume leveler 4 hdmi slots swivel stand piano black …",
    "price": null
  },
  "record_b": {
    "source": "buy",
    "name": "samsung 4 series ln22a450 22 ' lcd tv",
    "description": "samsung ln22a450 22 ' widescreen 720p lcd hdtv 3000:1 dynamic contrast ratio 8ms response time piano black",
    "price": "397.77"
  }
}

Question link_state · score · primary (ranked)

The state holds two product records, record_a from retailer abt and record_b from retailer buy (name, description, price). Rate how the two records relate as products.

3 options

0 — Two different products (different model or item, or only the same brand/category).
1 — Closely related products that may or may not be the same one: a colour or bundle variant, or a name that could refer to either.
2 — One and the same product (same manufacturer model, possibly worded or abbreviated differently).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: level 0: Two different products (different model or item, or only the same brand/category).

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`0`	correct	100.0%	1.000	`0` 100.0% `1` 0.0% `2` 0.0%
OpenJev	`0`	correct	99.4%	1.000	`0` 99.4% `1` 0.3% `2` 0.4%
System One Scorer	`0`	correct	75.2%	0.953	`0` 75.2% `1` 16.0% `2` 8.8%
decider-2b	`0`	correct	61.4%	0.885	`0` 61.4% `1` 24.9% `2` 13.7%
JevFish	`0`	correct	71.0%	0.933	`0` 71.0% `1` 20.5% `2` 8.5%
NanoJev	`0`	correct	44.3%	0.755	`0` 44.3% `1` 16.9% `2` 38.8%