Typed Decision Bench · tasks · Data and operations

`dbpedia-ontology-hierarchy`

hierarchical classification · media and reference · 200 items · primary question type choice. Source DeveloperOats/DBPedia_Classes (CC0-1.0); labels are outcome labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
Jev	92.9	94.0%	200 / 200
JevFish	89.4	88.5%	200 / 200
OpenJev	89.3	92.5%	200 / 200
decider-2b	89.0	89.0%	200 / 200
System One Scorer	83.8	87.0%	200 / 200
NanoJev	60.6	15.5%	200 / 200

Sample 1 of 6 · data-and-operations:dbpedia-3f10e15ec7dd · 120 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "article_text": "The 2010–11 football season is Bristol City's 113th season as a professional football club and fourth consecutive season in the second division. They are competing in the Football League Championship having finished in tenth position the previous season. It was Steve Coppell's first season in charge of the club, though he resigned as manager on 12 August 2010 after just one league game in charge. Assistant Keith Millen became the club's new manager on a three-year-deal and it is his first job as a manager."
}

Question l1 · choice · primary (ranked)

Read state.article_text (the opening of a Wikipedia article). Which top-level DBpedia ontology class does the article's subject belong to?

9 options

agent — A person, group, organisation, company, team, character or other acting entity
place — A geographic or built location: settlement, building, natural feature, route, station, celestial body
species — A biological organism: animal, plant, fungus or other eukaryote (incl. named breeds)
work — A created work: album, song, book, periodical, software, comic, cartoon
event — A dated happening: tournament, race, sports event, societal event, natural event, Olympics
sports_season — One season of a sports team or league
unit_of_work — A legal case or similar unit of proceedings
device — A manufactured device such as an engine
topical_concept — An abstract topic such as a genre

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: sports_season

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`sports_season`	correct	100.0%	1.000	`sports_season` 100.0% `species` 0.0% `place` 0.0% `event` 0.0% `device` 0.0% `work` 0.0%
JevFish	`sports_season`	correct	99.5%	1.000	`sports_season` 99.5% `agent` 0.3% `event` 0.0% `topical_concept` 0.0% `place` 0.0% `species` 0.0%
OpenJev	`event`	wrong	33.3%	0.560	`event` 65.8% `sports_season` 33.3% `agent` 0.8% `place` 0.1% `work` 0.0% `topical_concept` 0.0%
decider-2b	`agent`	wrong	19.7%	0.391	`agent` 75.7% `sports_season` 19.7% `event` 2.3% `topical_concept` 0.6% `species` 0.4% `work` 0.4%
System One Scorer	`agent`	wrong	15.5%	0.521	`agent` 44.6% `event` 18.3% `sports_season` 15.5% `place` 7.0% `species` 5.2% `topical_concept` 3.4%
NanoJev	`sports_season`	correct	31.1%	0.690	`sports_season` 31.1% `unit_of_work` 26.6% `agent` 25.5% `topical_concept` 9.0% `event` 3.4% `place` 2.9%

Question l2 · choice

Read state.article_text. Within the top-level class you chose, which second-level DBpedia class fits best? The options are the second-level classes of the article's true top-level class.

2 options

football_league_season — one season of a football league
sports_team_season — one season of a sports team

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: sports_team_season

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`sports_team_season`	correct	91.0%	0.992	`sports_team_season` 91.0% `football_league_season` 9.0%
JevFish	`football_league_season`	wrong	43.6%	0.681	`football_league_season` 56.4% `sports_team_season` 43.6%
OpenJev	`sports_team_season`	correct	85.7%	0.980	`sports_team_season` 85.7% `football_league_season` 14.3%
decider-2b	`sports_team_season`	correct	83.7%	0.973	`sports_team_season` 83.7% `football_league_season` 16.3%
System One Scorer	`sports_team_season`	correct	62.4%	0.858	`sports_team_season` 62.4% `football_league_season` 37.6%
NanoJev	`football_league_season`	wrong	27.4%	0.472	`football_league_season` 72.6% `sports_team_season` 27.4%

Sample 2 of 6 · data-and-operations:dbpedia-49266c80b7d8 · 206 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "article_text": "Carl Folta is Executive Vice President, Corporate Communications for Viacom. He has served at this post since November 2006. Before that, he served as Executive Vice President, Office of the Chairman, from January 1, 2006, where he served as Sumner Redstone's senior adviser and spokesman. Previously, he was Executive Vice President, Corporate Relations of the former Viacom Inc., since November 2004. Prior to that, he served as Senior Vice President of Corporate Relations of Viacom from November 1994 to November 2004, and Vice President of Corporate Relations of Viacom from April 1994 to November 1994. Folta held various communications positions at Paramount Communications from 1984 (when the company was known as Gulf+Western, retaining this name until 1989) until joining Viacom through its purchase of Paramount in April 1994."
}

Question l1 · choice · primary (ranked)

Read state.article_text (the opening of a Wikipedia article). Which top-level DBpedia ontology class does the article's subject belong to?

9 options

agent — A person, group, organisation, company, team, character or other acting entity
place — A geographic or built location: settlement, building, natural feature, route, station, celestial body
species — A biological organism: animal, plant, fungus or other eukaryote (incl. named breeds)
work — A created work: album, song, book, periodical, software, comic, cartoon
event — A dated happening: tournament, race, sports event, societal event, natural event, Olympics
sports_season — One season of a sports team or league
unit_of_work — A legal case or similar unit of proceedings
device — A manufactured device such as an engine
topical_concept — An abstract topic such as a genre

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: agent

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`agent`	correct	100.0%	1.000	`agent` 100.0% `sports_season` 0.0% `device` 0.0% `place` 0.0% `event` 0.0% `topical_concept` 0.0%
JevFish	`agent`	correct	98.9%	1.000	`agent` 98.9% `work` 0.3% `device` 0.2% `place` 0.1% `event` 0.1% `topical_concept` 0.1%
OpenJev	`agent`	correct	100.0%	1.000	`agent` 100.0% `place` 0.0% `event` 0.0% `work` 0.0% `topical_concept` 0.0% `species` 0.0%
decider-2b	`agent`	correct	99.4%	1.000	`agent` 99.4% `work` 0.1% `topical_concept` 0.1% `place` 0.1% `species` 0.1% `event` 0.1%
System One Scorer	`agent`	correct	78.4%	0.973	`agent` 78.4% `place` 4.4% `event` 4.3% `work` 3.9% `species` 2.9% `topical_concept` 2.6%
NanoJev	`agent`	correct	39.5%	0.770	`agent` 39.5% `unit_of_work` 24.9% `topical_concept` 14.1% `place` 7.8% `species` 5.4% `event` 4.0%

Question l2 · choice

Read state.article_text. Within the top-level class you chose, which second-level DBpedia class fits best? The options are the second-level classes of the article's true top-level class.

26 options

actor — film, television or stage actor
artist — visual or performing artist not covered by a more specific class
athlete — sportsperson (generic, when no sport-specific class applies)
boxer — boxer
british_royalty — member of the British royal family
broadcaster — radio or television station or network
cleric — religious office holder
coach — sports coach
company — commercial company
educational_institution — school, college or university
fictional_character — fictional character (non-comics)
gridiron_football_player — American or Canadian football player
group — band or other group of people
motorcycle_rider — motorcycle racer
organisation — organisation not covered by a more specific class
organisation_member — organisation member
person — person not covered by a more specific class
politician — politician or office holder
racing_driver — motor racing driver
scientist — scientist or academic
sports_league — sports league
sports_manager — sports manager
sports_team — sports team or club
winter_sport_player — winter-sport athlete (ice hockey, skiing, ...)
wrestler — wrestler
writer — author, poet or journalist

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: person

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`person`	correct	72.0%	0.934	`person` 72.0% `organisation_member` 23.0% `politician` 2.0% `company` 2.0% `broadcaster` 1.0% `boxer` 0.0%
JevFish	`person`	correct	49.1%	0.851	`person` 49.1% `organisation` 11.8% `politician` 10.5% `organisation_member` 10.1% `company` 4.7% `educational_institution` 1.3%
OpenJev	`person`	correct	99.8%	1.000	`person` 99.8% `organisation_member` 0.1% `politician` 0.0% `organisation` 0.0% `artist` 0.0% `actor` 0.0%
decider-2b	`person`	correct	88.5%	0.992	`person` 88.5% `organisation` 3.4% `organisation_member` 2.6% `company` 1.2% `politician` 0.7% `wrestler` 0.6%
System One Scorer	`organisation_member`	wrong	7.4%	0.476	`organisation_member` 35.0% `broadcaster` 20.4% `company` 12.7% `organisation` 8.5% `person` 7.4% `politician` 3.4%
NanoJev	`organisation_member`	wrong	12.8%	0.584	`organisation_member` 15.7% `person` 12.8% `organisation` 11.0% `artist` 10.6% `company` 9.8% `group` 6.5%

Sample 3 of 6 · data-and-operations:dbpedia-588ab13144e8 · 84 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "article_text": "The Apeldoorn–Deventer railway is an important railway line in the Netherlands, running from Apeldoorn to Deventer. The line was opened between 1887 and 1888. It is used by long distance trains from Amsterdam, Rotterdam and The Hague towards Enschede and Berlin, and by local services between Apeldoorn and Enschede."
}

Question l1 · choice · primary (ranked)

Read state.article_text (the opening of a Wikipedia article). Which top-level DBpedia ontology class does the article's subject belong to?

9 options

agent — A person, group, organisation, company, team, character or other acting entity
place — A geographic or built location: settlement, building, natural feature, route, station, celestial body
species — A biological organism: animal, plant, fungus or other eukaryote (incl. named breeds)
work — A created work: album, song, book, periodical, software, comic, cartoon
event — A dated happening: tournament, race, sports event, societal event, natural event, Olympics
sports_season — One season of a sports team or league
unit_of_work — A legal case or similar unit of proceedings
device — A manufactured device such as an engine
topical_concept — An abstract topic such as a genre

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: place

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`place`	correct	100.0%	1.000	`place` 100.0% `unit_of_work` 0.0% `device` 0.0% `event` 0.0% `work` 0.0% `topical_concept` 0.0%
JevFish	`place`	correct	98.9%	1.000	`place` 98.9% `topical_concept` 0.3% `work` 0.2% `species` 0.2% `event` 0.2% `sports_season` 0.1%
OpenJev	`place`	correct	99.9%	1.000	`place` 99.9% `agent` 0.1% `device` 0.0% `sports_season` 0.0% `topical_concept` 0.0% `event` 0.0%
decider-2b	`place`	correct	84.7%	0.986	`place` 84.7% `topical_concept` 4.7% `agent` 3.4% `device` 3.4% `event` 0.9% `sports_season` 0.9%
System One Scorer	`place`	correct	53.7%	0.877	`place` 53.7% `agent` 10.7% `event` 8.2% `work` 7.3% `topical_concept` 5.6% `species` 5.4%
NanoJev	`unit_of_work`	wrong	9.7%	0.471	`unit_of_work` 38.0% `agent` 26.4% `topical_concept` 16.1% `place` 9.7% `species` 4.4% `event` 3.1%

Question l2 · choice

Read state.article_text. Within the top-level class you chose, which second-level DBpedia class fits best? The options are the second-level classes of the article's true top-level class.

15 options

amusement_park_attraction — ride or attraction in an amusement park
body_of_water — lake, bay, sea or other still water body
building — building (house, church, museum, ...)
celestial_body — star, planet, asteroid or galaxy
clerical_administrative_region — diocese or other church administrative region
infrastructure — dam, airport, power station, bridge or similar
natural_place — mountain, valley, cave, glacier or other landform
route_of_transportation — road, railway line or public transit route
satellite — artificial satellite
settlement — city, town or village
sport_facility — stadium, arena or golf course
station — railway or metro station
stream — river or stream
tower — tower or lighthouse
venue — theatre, concert hall or other venue

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: route_of_transportation

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`route_of_transportation`	correct	98.0%	1.000	`route_of_transportation` 98.0% `infrastructure` 2.0% `station` 0.0% `satellite` 0.0% `celestial_body` 0.0% `building` 0.0%
JevFish	`route_of_transportation`	correct	75.0%	0.952	`route_of_transportation` 75.0% `station` 18.1% `infrastructure` 0.9% `amusement_park_attraction` 0.9% `venue` 0.7% `building` 0.6%
OpenJev	`route_of_transportation`	correct	99.7%	1.000	`route_of_transportation` 99.7% `infrastructure` 0.3% `station` 0.0% `body_of_water` 0.0% `natural_place` 0.0% `sport_facility` 0.0%
decider-2b	`route_of_transportation`	correct	97.7%	1.000	`route_of_transportation` 97.7% `infrastructure` 1.0% `station` 0.5% `settlement` 0.3% `venue` 0.1% `building` 0.1%
System One Scorer	`route_of_transportation`	correct	85.2%	0.988	`route_of_transportation` 85.2% `settlement` 3.6% `infrastructure` 1.8% `station` 1.6% `clerical_administrative_region` 1.2% `venue` 1.1%
NanoJev	`station`	wrong	22.5%	0.576	`station` 48.8% `route_of_transportation` 22.5% `settlement` 4.9% `body_of_water` 4.4% `natural_place` 4.2% `celestial_body` 3.6%

Sample 4 of 6 · data-and-operations:dbpedia-82cdaa7294bf · 105 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "article_text": "Electroclash, also known as retro electro, nouveau disco, the new new wave, and tech-pop is a genre of music that fuses 1980s electro, new wave and synthpop with 1990s techno, retro-style electropop and electronic dance music. It emerged in the later 1990s, pioneered by and associated with acts such as for example I-F, Miss Kittin & The Hacker or Fischerspooner."
}

Question l1 · choice · primary (ranked)

Read state.article_text (the opening of a Wikipedia article). Which top-level DBpedia ontology class does the article's subject belong to?

9 options

agent — A person, group, organisation, company, team, character or other acting entity
place — A geographic or built location: settlement, building, natural feature, route, station, celestial body
species — A biological organism: animal, plant, fungus or other eukaryote (incl. named breeds)
work — A created work: album, song, book, periodical, software, comic, cartoon
event — A dated happening: tournament, race, sports event, societal event, natural event, Olympics
sports_season — One season of a sports team or league
unit_of_work — A legal case or similar unit of proceedings
device — A manufactured device such as an engine
topical_concept — An abstract topic such as a genre

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: topical_concept

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`topical_concept`	correct	100.0%	1.000	`topical_concept` 100.0% `species` 0.0% `work` 0.0% `place` 0.0% `device` 0.0% `unit_of_work` 0.0%
JevFish	`topical_concept`	correct	99.9%	1.000	`topical_concept` 99.9% `work` 0.0% `unit_of_work` 0.0% `agent` 0.0% `event` 0.0% `place` 0.0%
OpenJev	`topical_concept`	correct	99.8%	1.000	`topical_concept` 99.8% `work` 0.2% `agent` 0.0% `event` 0.0% `place` 0.0% `species` 0.0%
decider-2b	`topical_concept`	correct	99.7%	1.000	`topical_concept` 99.7% `work` 0.1% `agent` 0.0% `event` 0.0% `place` 0.0% `species` 0.0%
System One Scorer	`topical_concept`	correct	50.9%	0.858	`topical_concept` 50.9% `agent` 12.7% `work` 10.7% `event` 9.3% `place` 6.4% `species` 3.7%
NanoJev	`topical_concept`	correct	33.3%	0.706	`topical_concept` 33.3% `unit_of_work` 31.0% `agent` 20.4% `place` 5.1% `event` 3.1% `device` 3.1%

Sample 5 of 6 · data-and-operations:dbpedia-ad98025f16c2 · 210 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "article_text": "Franco Ballerini (11 December 1964 – 7 February 2010) was an Italian road racing cyclist. Born in Florence, his greatest exploits as a rider came with his two victories in the cycling classic Paris–Roubaix, riding for the Mapei cycling team. In 1993 he was beaten on the line by Gilbert Duclos-Lassalle in an exciting Paris–Roubaix finale. Also on his palmarès or list of accomplishments there are other one-day races such as Omloop Het Volk and Paris–Brussels. Ballerini then became manager of the Italian national cycling squad, winning the 2002 World Championships with Mario Cipollini and the 2004 Summer Olympics in Athens with ex-teammate Paolo Bettini. In 2006, 2007 and 2008 he won the World Championships with Paolo Bettini and Alessandro Ballan."
}

Question l1 · choice · primary (ranked)

Read state.article_text (the opening of a Wikipedia article). Which top-level DBpedia ontology class does the article's subject belong to?

9 options

agent — A person, group, organisation, company, team, character or other acting entity
place — A geographic or built location: settlement, building, natural feature, route, station, celestial body
species — A biological organism: animal, plant, fungus or other eukaryote (incl. named breeds)
work — A created work: album, song, book, periodical, software, comic, cartoon
event — A dated happening: tournament, race, sports event, societal event, natural event, Olympics
sports_season — One season of a sports team or league
unit_of_work — A legal case or similar unit of proceedings
device — A manufactured device such as an engine
topical_concept — An abstract topic such as a genre

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: agent

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`agent`	correct	100.0%	1.000	`agent` 100.0% `unit_of_work` 0.0% `topical_concept` 0.0% `device` 0.0% `place` 0.0% `species` 0.0%
JevFish	`agent`	correct	84.1%	0.978	`agent` 84.1% `event` 13.6% `sports_season` 1.6% `topical_concept` 0.2% `work` 0.1% `device` 0.1%
OpenJev	`agent`	correct	100.0%	1.000	`agent` 100.0% `place` 0.0% `work` 0.0% `topical_concept` 0.0% `species` 0.0% `event` 0.0%
decider-2b	`agent`	correct	97.7%	1.000	`agent` 97.7% `species` 1.2% `event` 0.3% `topical_concept` 0.2% `place` 0.1% `work` 0.1%
System One Scorer	`agent`	correct	64.7%	0.923	`agent` 64.7% `event` 13.4% `species` 8.2% `place` 5.9% `work` 2.6% `topical_concept` 2.4%
NanoJev	`agent`	correct	22.5%	0.641	`agent` 22.5% `topical_concept` 19.7% `unit_of_work` 19.5% `event` 14.6% `sports_season` 11.1% `place` 7.6%

Question l2 · choice

Read state.article_text. Within the top-level class you chose, which second-level DBpedia class fits best? The options are the second-level classes of the article's true top-level class.

26 options

actor — film, television or stage actor
artist — visual or performing artist not covered by a more specific class
athlete — sportsperson (generic, when no sport-specific class applies)
boxer — boxer
british_royalty — member of the British royal family
broadcaster — radio or television station or network
cleric — religious office holder
coach — sports coach
company — commercial company
educational_institution — school, college or university
fictional_character — fictional character (non-comics)
gridiron_football_player — American or Canadian football player
group — band or other group of people
motorcycle_rider — motorcycle racer
organisation — organisation not covered by a more specific class
organisation_member — organisation member
person — person not covered by a more specific class
politician — politician or office holder
racing_driver — motor racing driver
scientist — scientist or academic
sports_league — sports league
sports_manager — sports manager
sports_team — sports team or club
winter_sport_player — winter-sport athlete (ice hockey, skiing, ...)
wrestler — wrestler
writer — author, poet or journalist

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: athlete

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`athlete`	correct	62.0%	0.896	`athlete` 62.0% `sports_manager` 23.0% `coach` 10.0% `person` 3.0% `racing_driver` 1.0% `motorcycle_rider` 1.0%
JevFish	`sports_manager`	wrong	24.1%	0.646	`sports_manager` 33.6% `athlete` 24.1% `person` 7.9% `coach` 7.8% `racing_driver` 5.5% `motorcycle_rider` 3.7%
OpenJev	`athlete`	correct	99.5%	1.000	`athlete` 99.5% `actor` 0.1% `person` 0.1% `artist` 0.1% `wrestler` 0.0% `organisation` 0.0%
decider-2b	`athlete`	correct	52.4%	0.823	`athlete` 52.4% `person` 35.7% `organisation` 2.2% `sports_manager` 1.3% `organisation_member` 1.1% `sports_team` 1.0%
System One Scorer	`coach`	wrong	10.9%	0.568	`coach` 17.9% `athlete` 10.9% `person` 10.7% `sports_manager` 9.6% `organisation` 7.3% `organisation_member` 4.9%
NanoJev	`organisation_member`	wrong	6.4%	0.517	`organisation_member` 18.0% `person` 12.3% `organisation` 11.0% `sports_manager` 9.7% `artist` 9.1% `sports_team` 7.0% `athlete` 6.4%

Sample 6 of 6 · data-and-operations:dbpedia-ccbffe43c144 · 45 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "article_text": "The World Basketball Association, often abbreviated to the WBA, is a semi-professional men's spring basketball league in the United States. The league suspended operations after the 2013 season."
}

Question l1 · choice · primary (ranked)

Read state.article_text (the opening of a Wikipedia article). Which top-level DBpedia ontology class does the article's subject belong to?

9 options

agent — A person, group, organisation, company, team, character or other acting entity
place — A geographic or built location: settlement, building, natural feature, route, station, celestial body
species — A biological organism: animal, plant, fungus or other eukaryote (incl. named breeds)
work — A created work: album, song, book, periodical, software, comic, cartoon
event — A dated happening: tournament, race, sports event, societal event, natural event, Olympics
sports_season — One season of a sports team or league
unit_of_work — A legal case or similar unit of proceedings
device — A manufactured device such as an engine
topical_concept — An abstract topic such as a genre

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: agent

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`agent`	correct	85.0%	0.979	`agent` 85.0% `event` 14.0% `sports_season` 1.0% `place` 0.0% `unit_of_work` 0.0% `topical_concept` 0.0%
JevFish	`sports_season`	wrong	0.5%	0.048	`sports_season` 95.5% `event` 3.5% `agent` 0.5% `topical_concept` 0.2% `place` 0.1% `species` 0.1%
OpenJev	`agent`	correct	59.3%	0.860	`agent` 59.3% `event` 33.1% `sports_season` 7.1% `topical_concept` 0.4% `place` 0.1% `work` 0.0%
decider-2b	`agent`	correct	64.0%	0.917	`agent` 64.0% `topical_concept` 14.4% `event` 8.5% `sports_season` 8.5% `work` 1.7% `place` 1.1%
System One Scorer	`agent`	correct	37.3%	0.754	`agent` 37.3% `event` 27.7% `sports_season` 9.0% `work` 7.3% `place` 5.5% `topical_concept` 4.7%
NanoJev	`unit_of_work`	wrong	25.7%	0.645	`unit_of_work` 29.1% `agent` 25.7% `sports_season` 22.6% `topical_concept` 13.8% `event` 4.5% `place` 2.5%

Question l2 · choice

Read state.article_text. Within the top-level class you chose, which second-level DBpedia class fits best? The options are the second-level classes of the article's true top-level class.

26 options

actor — film, television or stage actor
artist — visual or performing artist not covered by a more specific class
athlete — sportsperson (generic, when no sport-specific class applies)
boxer — boxer
british_royalty — member of the British royal family
broadcaster — radio or television station or network
cleric — religious office holder
coach — sports coach
company — commercial company
educational_institution — school, college or university
fictional_character — fictional character (non-comics)
gridiron_football_player — American or Canadian football player
group — band or other group of people
motorcycle_rider — motorcycle racer
organisation — organisation not covered by a more specific class
organisation_member — organisation member
person — person not covered by a more specific class
politician — politician or office holder
racing_driver — motor racing driver
scientist — scientist or academic
sports_league — sports league
sports_manager — sports manager
sports_team — sports team or club
winter_sport_player — winter-sport athlete (ice hockey, skiing, ...)
wrestler — wrestler
writer — author, poet or journalist

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: sports_league

System	Top pick	Grade	P(expected)	Proper score	Distribution
Jev	`sports_league`	correct	93.0%	0.996	`sports_league` 93.0% `organisation` 5.0% `sports_team` 1.0% `motorcycle_rider` 0.0% `coach` 0.0% `winter_sport_player` 0.0%
JevFish	`sports_league`	correct	95.2%	0.999	`sports_league` 95.2% `organisation` 1.7% `sports_team` 0.3% `person` 0.2% `writer` 0.2% `racing_driver` 0.2%
OpenJev	`sports_league`	correct	95.7%	0.999	`sports_league` 95.7% `sports_team` 1.4% `person` 0.8% `broadcaster` 0.3% `actor` 0.3% `scientist` 0.2%
decider-2b	`sports_league`	correct	89.1%	0.990	`sports_league` 89.1% `organisation` 8.9% `sports_team` 0.7% `athlete` 0.2% `group` 0.1% `company` 0.1%
System One Scorer	`sports_league`	correct	60.4%	0.902	`sports_league` 60.4% `organisation` 19.1% `sports_team` 3.7% `group` 2.7% `organisation_member` 2.1% `company` 1.9%
NanoJev	`person`	wrong	6.6%	0.509	`person` 19.5% `organisation` 14.8% `artist` 13.6% `athlete` 11.2% `organisation_member` 8.6% `sports_team` 8.1% `sports_league` 6.6%