Typed Decision Bench · tasks · Retrieval and knowledge

`abt-buy-product-match`

entity alignment · e commerce · 200 items · primary question type score. Source matchbench/Abt-Buy (CC (variant unconfirmed on the source page) — FLAGGED); labels are expert labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
Jev	94.9	93.0%	200 / 200
OpenJev	92.9	92.0%	200 / 200
JevFish	87.1	89.5%	200 / 200
System One Scorer	85.4	86.5%	200 / 200
decider-2b	82.7	90.5%	200 / 200
NanoJev	74.5	66.5%	200 / 200

Sample 1 of 6 · retrieval-and-knowledge:abtbuy-1024 · 193 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "record_a": {
    "source": "abt",
    "name": "samsung 67 ' black widescreen series 7 led dlp hdtv hl67a750",
    "description": "samsung 67 ' black widescreen series 7 led dlp hdtv hl67a750 1920 x 1080p resolution 10,000:1 contrast ratio digital noise reduction cinemapure color engine get the full effect of 3d led backlight ntsc/atsc/clear qam built-in tuners 60,000 hours lamp life black finish",
    "price": null
  },
  "record_b": {
    "source": "buy",
    "name": "samsung hl72a650 72 ' widescreen 1080p dlp hdtv 2,500:1 contrast ratio",
    "description": "",
    "price": "2188.7"
  }
}

Question link_state · score · primary (ranked)

The state holds two product records, record_a from retailer abt and record_b from retailer buy (name, description, price). Rate how the two records relate as products.

3 options

0 — Two different products (different model or item, or only the same brand/category).
1 — Closely related products that may or may not be the same one: a colour or bundle variant, or a name that could refer to either.
2 — One and the same product (same manufacturer model, possibly worded or abbreviated differently).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: level 0: Two different products (different model or item, or only the same brand/category).

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`0`	correct	98.0%	1.000	`0` 98.0% `1` 2.0% `2` 0.0%
OpenJev	`0`	correct	99.1%	1.000	`0` 99.1% `1` 0.6% `2` 0.3%
JevFish	`0`	correct	67.3%	0.908	`0` 67.3% `1` 27.0% `2` 5.7%
System One Scorer	`0`	correct	61.0%	0.884	`0` 61.0% `1` 24.1% `2` 14.9%
decider-2b	`0`	correct	52.7%	0.830	`0` 52.7% `1` 28.8% `2` 18.5%
NanoJev	`0`	correct	45.8%	0.769	`0` 45.8% `1` 17.0% `2` 37.3%

Sample 2 of 6 · retrieval-and-knowledge:abtbuy-1337 · 125 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "record_a": {
    "source": "abt",
    "name": "garmin streetpilot c series vehicle suction mount black finish 0101060900",
    "description": "garmin streetpilot c series vehicle suction mount 0101060900 dashboard or friction mounting options are recommended except california 12/24 volt adapter",
    "price": null
  },
  "record_b": {
    "source": "buy",
    "name": "garmin suction cup mount/12-volt adapter kit 010-10979-00",
    "description": "navigator starter kit",
    "price": null
  }
}

Question link_state · score · primary (ranked)

The state holds two product records, record_a from retailer abt and record_b from retailer buy (name, description, price). Rate how the two records relate as products.

3 options

0 — Two different products (different model or item, or only the same brand/category).
1 — Closely related products that may or may not be the same one: a colour or bundle variant, or a name that could refer to either.
2 — One and the same product (same manufacturer model, possibly worded or abbreviated differently).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: level 0: Two different products (different model or item, or only the same brand/category).

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`0`	correct	51.0%	0.782	`0` 51.0% `1` 44.0% `2` 5.0%
OpenJev	`1`	wrong	17.7%	0.448	`0` 17.7% `1` 62.1% `2` 20.2%
JevFish	`2`	wrong	29.5%	0.628	`0` 29.5% `1` 34.2% `2` 36.3%
System One Scorer	`2`	wrong	10.0%	0.358	`0` 10.0% `1` 26.4% `2` 63.6%
decider-2b	`2`	wrong	32.0%	0.640	`0` 32.0% `1` 22.5% `2` 45.5%
NanoJev	`0`	correct	51.0%	0.796	`0` 51.0% `1` 8.8% `2` 40.2%

Sample 3 of 6 · retrieval-and-knowledge:abtbuy-1791 · 179 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "record_a": {
    "source": "abt",
    "name": "sony black cyber-shot digital camera dscw150b",
    "description": "sony dscw150 black cyber-shot digital camera dscw150b 8.1 megapixels 2.7 ' lcd display 15 mb internal memory carl zeiss 5x optical zoom lens 16:9 high-resolution mode 9-point auto focus automatic macro shooting high sensitivity mode burst mode face detection intelligent scene recognition mode …",
    "price": null
  },
  "record_b": {
    "source": "buy",
    "name": "sony cyber-shot dsc-t300 digital camera black dsct300/b",
    "description": "10.1 megapixel 16:9 2x digital zoom 3.5 ' active matrix tft color lcd",
    "price": "273.95"
  }
}

Question link_state · score · primary (ranked)

The state holds two product records, record_a from retailer abt and record_b from retailer buy (name, description, price). Rate how the two records relate as products.

3 options

0 — Two different products (different model or item, or only the same brand/category).
1 — Closely related products that may or may not be the same one: a colour or bundle variant, or a name that could refer to either.
2 — One and the same product (same manufacturer model, possibly worded or abbreviated differently).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: level 0: Two different products (different model or item, or only the same brand/category).

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`0`	correct	100.0%	1.000	`0` 100.0% `1` 0.0% `2` 0.0%
OpenJev	`0`	correct	99.1%	1.000	`0` 99.1% `1` 0.4% `2` 0.4%
JevFish	`0`	correct	64.7%	0.901	`0` 64.7% `1` 25.4% `2` 9.9%
System One Scorer	`0`	correct	52.9%	0.832	`0` 52.9% `1` 20.0% `2` 27.2%
decider-2b	`0`	correct	45.6%	0.778	`0` 45.6% `1` 24.9% `2` 29.4%
NanoJev	`0`	correct	44.6%	0.760	`0` 44.6% `1` 18.1% `2` 37.4%

Sample 4 of 6 · retrieval-and-knowledge:abtbuy-630 · 135 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "record_a": {
    "source": "abt",
    "name": "denon x-space surround bar home theatre system in black dhtfs3",
    "description": "denon x-space surround bar home theatre system in black dhtfs3 22 watts x 5 ( 6 ohms ) supports all audio formats ( dolby digital , dts , dolby pro logic ii ) slim design includes dolby headphones included subwoofer features one-touch connection remote control black finish",
    "price": "1199.0"
  },
  "record_b": {
    "source": "buy",
    "name": "denon dht-fs5 surround sound speaker system",
    "description": "5.1-channel",
    "price": null
  }
}

Question link_state · score · primary (ranked)

The state holds two product records, record_a from retailer abt and record_b from retailer buy (name, description, price). Rate how the two records relate as products.

3 options

0 — Two different products (different model or item, or only the same brand/category).
1 — Closely related products that may or may not be the same one: a colour or bundle variant, or a name that could refer to either.
2 — One and the same product (same manufacturer model, possibly worded or abbreviated differently).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: level 0: Two different products (different model or item, or only the same brand/category).

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`0`	correct	85.0%	0.979	`0` 85.0% `1` 14.0% `2` 1.0%
OpenJev	`2`	wrong	0.6%	0.031	`0` 0.6% `1` 2.0% `2` 97.4%
JevFish	`0`	correct	53.3%	0.831	`0` 53.3% `1` 30.9% `2` 15.7%
System One Scorer	`2`	wrong	13.1%	0.395	`0` 13.1% `1` 23.6% `2` 63.3%
decider-2b	`0`	correct	44.7%	0.770	`0` 44.7% `1` 29.5% `2` 25.7%
NanoJev	`0`	correct	50.3%	0.783	`0` 50.3% `1` 7.0% `2` 42.7%

Sample 5 of 6 · retrieval-and-knowledge:abtbuy-673 · 170 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "record_a": {
    "source": "abt",
    "name": "panasonic black 8.5 ' portable dvd player dvdls83",
    "description": "panasonic black 8.5 ' portable dvd player dvdls83 8.5 ' free style wide screen lcd display 6 hour battery multi format playback dual headphone jacks divx playback car dc adapter included black finish",
    "price": null
  },
  "record_b": {
    "source": "buy",
    "name": "panasonic dvd-ls83 portable dvd player",
    "description": "8.5 ' lcd dvd + rw , dvd-rw , dvd-ram , dvd + r , dvd-r , cd-rw dvd video , cd-da , mp3 , jpeg , mpeg-4 , divx , wma , video cd , svcd , highmat playback",
    "price": "148.69"
  }
}

Question link_state · score · primary (ranked)

The state holds two product records, record_a from retailer abt and record_b from retailer buy (name, description, price). Rate how the two records relate as products.

3 options

0 — Two different products (different model or item, or only the same brand/category).
1 — Closely related products that may or may not be the same one: a colour or bundle variant, or a name that could refer to either.
2 — One and the same product (same manufacturer model, possibly worded or abbreviated differently).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: level 2: One and the same product (same manufacturer model, possibly worded or abbreviated differently).

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`2`	correct	99.0%	1.000	`0` 0.0% `1` 1.0% `2` 99.0%
OpenJev	`2`	correct	99.9%	1.000	`0` 0.0% `1` 0.1% `2` 99.9%
JevFish	`2`	correct	72.6%	0.939	`0` 6.8% `1` 20.5% `2` 72.6%
System One Scorer	`2`	correct	86.6%	0.985	`0` 2.2% `1` 11.1% `2` 86.6%
decider-2b	`2`	correct	64.9%	0.907	`0` 16.0% `1` 19.2% `2` 64.9%
NanoJev	`0`	wrong	38.0%	0.682	`0` 48.2% `1` 13.8% `2` 38.0%

Sample 6 of 6 · retrieval-and-knowledge:abtbuy-785 · 160 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "record_a": {
    "source": "abt",
    "name": "panasonic black 8.5 ' portable dvd player dvdls86",
    "description": "panasonic black 8.5 ' portable dvd player dvdls86 8.5 ' free style wide screen lcd display 13 hour battery multi format playback dual headphone jacks divx playback car dc adapter included black finish",
    "price": null
  },
  "record_b": {
    "source": "buy",
    "name": "sony dvp-fx820 portable dvd player dvpfx820",
    "description": "8 ' lcd dvd + rw , dvd-rw , dvd + r , dvd-r , cd-rw dvd video , cd-da , mp3 , jpeg playback 1 disc ( s ) black",
    "price": "158.72"
  }
}

Question link_state · score · primary (ranked)

The state holds two product records, record_a from retailer abt and record_b from retailer buy (name, description, price). Rate how the two records relate as products.

3 options

0 — Two different products (different model or item, or only the same brand/category).
1 — Closely related products that may or may not be the same one: a colour or bundle variant, or a name that could refer to either.
2 — One and the same product (same manufacturer model, possibly worded or abbreviated differently).

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: level 0: Two different products (different model or item, or only the same brand/category).

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`0`	correct	100.0%	1.000	`0` 100.0% `1` 0.0% `2` 0.0%
OpenJev	`0`	correct	98.3%	1.000	`0` 98.3% `1` 1.4% `2` 0.4%
JevFish	`0`	correct	53.2%	0.823	`0` 53.2% `1` 34.7% `2` 12.1%
System One Scorer	`0`	correct	70.4%	0.928	`0` 70.4% `1` 22.6% `2` 7.0%
decider-2b	`0`	correct	48.5%	0.801	`0` 48.5% `1` 26.2% `2` 25.3%
NanoJev	`0`	correct	46.4%	0.768	`0` 46.4% `1` 13.9% `2` 39.6%