Skip to samples
Typed Decision Bench · tasks · Data and operations

dbpedia-ontology-hierarchy

hierarchical classification · media and reference · 200 items · primary question type choice. Source DeveloperOats/DBPedia_Classes (CC0-1.0); labels are outcome labels, never an LLM judge. Download these samples (JSON).

SystemDecisionScoreAccuracy (top pick)Answered
Jev92.994.0%200 / 200
JevFish89.690.0%200 / 200
OpenJev89.392.5%200 / 200
decider-2b89.089.0%200 / 200
System One Scorer83.887.0%200 / 200
NanoJev60.615.5%200 / 200
Sample 1 of 6 · data-and-operations:dbpedia-3f10e15ec7dd · 120 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "article_text": "The 2010–11 football season is Bristol City's 113th season as a professional football club and fourth consecutive season in the second division. They are competing in the Football League Championship having finished in tenth position the previous season. It was Steve Coppell's first season in charge of the club, though he resigned as manager on 12 August 2010 after just one league game in charge. Assistant Keith Millen became the club's new manager on a three-year-deal and it is his first job as a manager."
}

Question l1 · choice · primary (ranked)

Read state.article_text (the opening of a Wikipedia article). Which top-level DBpedia ontology class does the article's subject belong to?

9 options
  • agent — A person, group, organisation, company, team, character or other acting entity
  • place — A geographic or built location: settlement, building, natural feature, route, station, celestial body
  • species — A biological organism: animal, plant, fungus or other eukaryote (incl. named breeds)
  • work — A created work: album, song, book, periodical, software, comic, cartoon
  • event — A dated happening: tournament, race, sports event, societal event, natural event, Olympics
  • sports_season — One season of a sports team or league
  • unit_of_work — A legal case or similar unit of proceedings
  • device — A manufactured device such as an engine
  • topical_concept — An abstract topic such as a genre

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: sports_season

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevsports_seasoncorrect100.0%1.000
  • sports_season 100.0%
  • species 0.0%
  • place 0.0%
  • event 0.0%
  • device 0.0%
  • work 0.0%
JevFishagentwrong43.6%0.695
  • agent 54.0%
  • sports_season 43.6%
  • event 1.6%
  • topical_concept 0.3%
  • place 0.2%
  • species 0.1%
OpenJeveventwrong33.3%0.560
  • event 65.8%
  • sports_season 33.3%
  • agent 0.8%
  • place 0.1%
  • work 0.0%
  • topical_concept 0.0%
decider-2bagentwrong19.7%0.391
  • agent 75.7%
  • sports_season 19.7%
  • event 2.3%
  • topical_concept 0.6%
  • species 0.4%
  • work 0.4%
System One Scoreragentwrong15.5%0.521
  • agent 44.6%
  • event 18.3%
  • sports_season 15.5%
  • place 7.0%
  • species 5.2%
  • topical_concept 3.4%
NanoJevsports_seasoncorrect31.1%0.690
  • sports_season 31.1%
  • unit_of_work 26.6%
  • agent 25.5%
  • topical_concept 9.0%
  • event 3.4%
  • place 2.9%

Question l2 · choice

Read state.article_text. Within the top-level class you chose, which second-level DBpedia class fits best? The options are the second-level classes of the article's true top-level class.

2 options
  • football_league_season — one season of a football league
  • sports_team_season — one season of a sports team

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: sports_team_season

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevsports_team_seasoncorrect91.0%0.992
  • sports_team_season 91.0%
  • football_league_season 9.0%
JevFishsports_team_seasoncorrect71.8%0.921
  • sports_team_season 71.8%
  • football_league_season 28.2%
OpenJevsports_team_seasoncorrect85.7%0.980
  • sports_team_season 85.7%
  • football_league_season 14.3%
decider-2bsports_team_seasoncorrect83.7%0.973
  • sports_team_season 83.7%
  • football_league_season 16.3%
System One Scorersports_team_seasoncorrect62.4%0.858
  • sports_team_season 62.4%
  • football_league_season 37.6%
NanoJevfootball_league_seasonwrong27.4%0.472
  • football_league_season 72.6%
  • sports_team_season 27.4%
Sample 2 of 6 · data-and-operations:dbpedia-49266c80b7d8 · 206 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "article_text": "Carl Folta is Executive Vice President, Corporate Communications for Viacom. He has served at this post since November 2006. Before that, he served as Executive Vice President, Office of the Chairman, from January 1, 2006, where he served as Sumner Redstone's senior adviser and spokesman. Previously, he was Executive Vice President, Corporate Relations of the former Viacom Inc., since November 2004. Prior to that, he served as Senior Vice President of Corporate Relations of Viacom from November 1994 to November 2004, and Vice President of Corporate Relations of Viacom from April 1994 to November 1994. Folta held various communications positions at Paramount Communications from 1984 (when the company was known as Gulf+Western, retaining this name until 1989) until joining Viacom through its purchase of Paramount in April 1994."
}

Question l1 · choice · primary (ranked)

Read state.article_text (the opening of a Wikipedia article). Which top-level DBpedia ontology class does the article's subject belong to?

9 options
  • agent — A person, group, organisation, company, team, character or other acting entity
  • place — A geographic or built location: settlement, building, natural feature, route, station, celestial body
  • species — A biological organism: animal, plant, fungus or other eukaryote (incl. named breeds)
  • work — A created work: album, song, book, periodical, software, comic, cartoon
  • event — A dated happening: tournament, race, sports event, societal event, natural event, Olympics
  • sports_season — One season of a sports team or league
  • unit_of_work — A legal case or similar unit of proceedings
  • device — A manufactured device such as an engine
  • topical_concept — An abstract topic such as a genre

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: agent

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevagentcorrect100.0%1.000
  • agent 100.0%
  • sports_season 0.0%
  • device 0.0%
  • place 0.0%
  • event 0.0%
  • topical_concept 0.0%
JevFishagentcorrect99.9%1.000
  • agent 99.9%
  • work 0.1%
  • topical_concept 0.0%
  • species 0.0%
  • place 0.0%
  • sports_season 0.0%
OpenJevagentcorrect100.0%1.000
  • agent 100.0%
  • place 0.0%
  • event 0.0%
  • work 0.0%
  • topical_concept 0.0%
  • species 0.0%
decider-2bagentcorrect99.4%1.000
  • agent 99.4%
  • work 0.1%
  • topical_concept 0.1%
  • place 0.1%
  • species 0.1%
  • event 0.1%
System One Scoreragentcorrect78.4%0.973
  • agent 78.4%
  • place 4.4%
  • event 4.3%
  • work 3.9%
  • species 2.9%
  • topical_concept 2.6%
NanoJevagentcorrect39.5%0.770
  • agent 39.5%
  • unit_of_work 24.9%
  • topical_concept 14.1%
  • place 7.8%
  • species 5.4%
  • event 4.0%

Question l2 · choice

Read state.article_text. Within the top-level class you chose, which second-level DBpedia class fits best? The options are the second-level classes of the article's true top-level class.

26 options
  • actor — film, television or stage actor
  • artist — visual or performing artist not covered by a more specific class
  • athlete — sportsperson (generic, when no sport-specific class applies)
  • boxer — boxer
  • british_royalty — member of the British royal family
  • broadcaster — radio or television station or network
  • cleric — religious office holder
  • coach — sports coach
  • company — commercial company
  • educational_institution — school, college or university
  • fictional_character — fictional character (non-comics)
  • gridiron_football_player — American or Canadian football player
  • group — band or other group of people
  • motorcycle_rider — motorcycle racer
  • organisation — organisation not covered by a more specific class
  • organisation_member — organisation member
  • person — person not covered by a more specific class
  • politician — politician or office holder
  • racing_driver — motor racing driver
  • scientist — scientist or academic
  • sports_league — sports league
  • sports_manager — sports manager
  • sports_team — sports team or club
  • winter_sport_player — winter-sport athlete (ice hockey, skiing, ...)
  • wrestler — wrestler
  • writer — author, poet or journalist

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: person

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevpersoncorrect72.0%0.934
  • person 72.0%
  • organisation_member 23.0%
  • politician 2.0%
  • company 2.0%
  • broadcaster 1.0%
  • boxer 0.0%
JevFishpersoncorrect79.3%0.975
  • person 79.3%
  • organisation_member 7.2%
  • organisation 5.0%
  • company 1.8%
  • writer 1.0%
  • politician 0.8%
OpenJevpersoncorrect99.8%1.000
  • person 99.8%
  • organisation_member 0.1%
  • politician 0.0%
  • organisation 0.0%
  • artist 0.0%
  • actor 0.0%
decider-2bpersoncorrect88.5%0.992
  • person 88.5%
  • organisation 3.4%
  • organisation_member 2.6%
  • company 1.2%
  • politician 0.7%
  • wrestler 0.6%
System One Scorerorganisation_memberwrong7.4%0.476
  • organisation_member 35.0%
  • broadcaster 20.4%
  • company 12.7%
  • organisation 8.5%
  • person 7.4%
  • politician 3.4%
NanoJevorganisation_memberwrong12.8%0.584
  • organisation_member 15.7%
  • person 12.8%
  • organisation 11.0%
  • artist 10.6%
  • company 9.8%
  • group 6.5%
Sample 3 of 6 · data-and-operations:dbpedia-588ab13144e8 · 84 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "article_text": "The Apeldoorn–Deventer railway is an important railway line in the Netherlands, running from Apeldoorn to Deventer. The line was opened between 1887 and 1888. It is used by long distance trains from Amsterdam, Rotterdam and The Hague towards Enschede and Berlin, and by local services between Apeldoorn and Enschede."
}

Question l1 · choice · primary (ranked)

Read state.article_text (the opening of a Wikipedia article). Which top-level DBpedia ontology class does the article's subject belong to?

9 options
  • agent — A person, group, organisation, company, team, character or other acting entity
  • place — A geographic or built location: settlement, building, natural feature, route, station, celestial body
  • species — A biological organism: animal, plant, fungus or other eukaryote (incl. named breeds)
  • work — A created work: album, song, book, periodical, software, comic, cartoon
  • event — A dated happening: tournament, race, sports event, societal event, natural event, Olympics
  • sports_season — One season of a sports team or league
  • unit_of_work — A legal case or similar unit of proceedings
  • device — A manufactured device such as an engine
  • topical_concept — An abstract topic such as a genre

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: place

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevplacecorrect100.0%1.000
  • place 100.0%
  • unit_of_work 0.0%
  • device 0.0%
  • event 0.0%
  • work 0.0%
  • topical_concept 0.0%
JevFishplacecorrect87.5%0.989
  • place 87.5%
  • agent 6.8%
  • topical_concept 3.4%
  • device 1.7%
  • work 0.2%
  • species 0.1%
OpenJevplacecorrect99.9%1.000
  • place 99.9%
  • agent 0.1%
  • device 0.0%
  • sports_season 0.0%
  • topical_concept 0.0%
  • event 0.0%
decider-2bplacecorrect84.7%0.986
  • place 84.7%
  • topical_concept 4.7%
  • agent 3.4%
  • device 3.4%
  • event 0.9%
  • sports_season 0.9%
System One Scorerplacecorrect53.7%0.877
  • place 53.7%
  • agent 10.7%
  • event 8.2%
  • work 7.3%
  • topical_concept 5.6%
  • species 5.4%
NanoJevunit_of_workwrong9.7%0.471
  • unit_of_work 38.0%
  • agent 26.4%
  • topical_concept 16.1%
  • place 9.7%
  • species 4.4%
  • event 3.1%

Question l2 · choice

Read state.article_text. Within the top-level class you chose, which second-level DBpedia class fits best? The options are the second-level classes of the article's true top-level class.

15 options
  • amusement_park_attraction — ride or attraction in an amusement park
  • body_of_water — lake, bay, sea or other still water body
  • building — building (house, church, museum, ...)
  • celestial_body — star, planet, asteroid or galaxy
  • clerical_administrative_region — diocese or other church administrative region
  • infrastructure — dam, airport, power station, bridge or similar
  • natural_place — mountain, valley, cave, glacier or other landform
  • route_of_transportation — road, railway line or public transit route
  • satellite — artificial satellite
  • settlement — city, town or village
  • sport_facility — stadium, arena or golf course
  • station — railway or metro station
  • stream — river or stream
  • tower — tower or lighthouse
  • venue — theatre, concert hall or other venue

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: route_of_transportation

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevroute_of_transportationcorrect98.0%1.000
  • route_of_transportation 98.0%
  • infrastructure 2.0%
  • station 0.0%
  • satellite 0.0%
  • celestial_body 0.0%
  • building 0.0%
JevFishroute_of_transportationcorrect96.7%0.999
  • route_of_transportation 96.7%
  • infrastructure 1.8%
  • station 0.7%
  • settlement 0.4%
  • venue 0.1%
  • building 0.1%
OpenJevroute_of_transportationcorrect99.7%1.000
  • route_of_transportation 99.7%
  • infrastructure 0.3%
  • station 0.0%
  • body_of_water 0.0%
  • natural_place 0.0%
  • sport_facility 0.0%
decider-2broute_of_transportationcorrect97.7%1.000
  • route_of_transportation 97.7%
  • infrastructure 1.0%
  • station 0.5%
  • settlement 0.3%
  • venue 0.1%
  • building 0.1%
System One Scorerroute_of_transportationcorrect85.2%0.988
  • route_of_transportation 85.2%
  • settlement 3.6%
  • infrastructure 1.8%
  • station 1.6%
  • clerical_administrative_region 1.2%
  • venue 1.1%
NanoJevstationwrong22.5%0.576
  • station 48.8%
  • route_of_transportation 22.5%
  • settlement 4.9%
  • body_of_water 4.4%
  • natural_place 4.2%
  • celestial_body 3.6%
Sample 4 of 6 · data-and-operations:dbpedia-82cdaa7294bf · 105 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "article_text": "Electroclash, also known as retro electro, nouveau disco, the new new wave, and tech-pop is a genre of music that fuses 1980s electro, new wave and synthpop with 1990s techno, retro-style electropop and electronic dance music. It emerged in the later 1990s, pioneered by and associated with acts such as for example I-F, Miss Kittin & The Hacker or Fischerspooner."
}

Question l1 · choice · primary (ranked)

Read state.article_text (the opening of a Wikipedia article). Which top-level DBpedia ontology class does the article's subject belong to?

9 options
  • agent — A person, group, organisation, company, team, character or other acting entity
  • place — A geographic or built location: settlement, building, natural feature, route, station, celestial body
  • species — A biological organism: animal, plant, fungus or other eukaryote (incl. named breeds)
  • work — A created work: album, song, book, periodical, software, comic, cartoon
  • event — A dated happening: tournament, race, sports event, societal event, natural event, Olympics
  • sports_season — One season of a sports team or league
  • unit_of_work — A legal case or similar unit of proceedings
  • device — A manufactured device such as an engine
  • topical_concept — An abstract topic such as a genre

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: topical_concept

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevtopical_conceptcorrect100.0%1.000
  • topical_concept 100.0%
  • species 0.0%
  • work 0.0%
  • place 0.0%
  • device 0.0%
  • unit_of_work 0.0%
JevFishtopical_conceptcorrect99.8%1.000
  • topical_concept 99.8%
  • work 0.2%
  • agent 0.1%
  • place 0.0%
  • event 0.0%
  • sports_season 0.0%
OpenJevtopical_conceptcorrect99.8%1.000
  • topical_concept 99.8%
  • work 0.2%
  • agent 0.0%
  • event 0.0%
  • place 0.0%
  • species 0.0%
decider-2btopical_conceptcorrect99.7%1.000
  • topical_concept 99.7%
  • work 0.1%
  • agent 0.0%
  • event 0.0%
  • place 0.0%
  • species 0.0%
System One Scorertopical_conceptcorrect50.9%0.858
  • topical_concept 50.9%
  • agent 12.7%
  • work 10.7%
  • event 9.3%
  • place 6.4%
  • species 3.7%
NanoJevtopical_conceptcorrect33.3%0.706
  • topical_concept 33.3%
  • unit_of_work 31.0%
  • agent 20.4%
  • place 5.1%
  • event 3.1%
  • device 3.1%
Sample 5 of 6 · data-and-operations:dbpedia-a00d0a078e2c · 183 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "article_text": "Dame Barbara Mary Quant, Mrs Plunket Greene, DBE, FCSD, RDI (born 11 February 1934) is a Welsh fashion designer and British fashion icon. She became an instrumental figure in the 1960s London-based Mod and youth fashion movements. She was one of the designers who took credit for the miniskirt and hot pants, and by promoting these and other fun fashions she encouraged young people to dress to please themselves and to treat fashion as a game. Ernestine Carter, an authoritative and influential fashion journalist of the 1950s/60s, wrote: \\\"It is given to a fortunate few to be born at the right time, in the right place, with the right talents. In recent fashion there are three: Chanel, Dior, and Mary Quant.\\\""
}

Question l1 · choice · primary (ranked)

Read state.article_text (the opening of a Wikipedia article). Which top-level DBpedia ontology class does the article's subject belong to?

9 options
  • agent — A person, group, organisation, company, team, character or other acting entity
  • place — A geographic or built location: settlement, building, natural feature, route, station, celestial body
  • species — A biological organism: animal, plant, fungus or other eukaryote (incl. named breeds)
  • work — A created work: album, song, book, periodical, software, comic, cartoon
  • event — A dated happening: tournament, race, sports event, societal event, natural event, Olympics
  • sports_season — One season of a sports team or league
  • unit_of_work — A legal case or similar unit of proceedings
  • device — A manufactured device such as an engine
  • topical_concept — An abstract topic such as a genre

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: agent

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevagentcorrect100.0%1.000
  • agent 100.0%
  • species 0.0%
  • sports_season 0.0%
  • unit_of_work 0.0%
  • device 0.0%
  • work 0.0%
JevFishagentcorrect99.7%1.000
  • agent 99.7%
  • topical_concept 0.2%
  • work 0.1%
  • species 0.0%
  • place 0.0%
  • event 0.0%
OpenJevagentcorrect100.0%1.000
  • agent 100.0%
  • place 0.0%
  • topical_concept 0.0%
  • event 0.0%
  • work 0.0%
  • unit_of_work 0.0%
decider-2bagentcorrect99.0%1.000
  • agent 99.0%
  • topical_concept 0.4%
  • work 0.1%
  • place 0.1%
  • species 0.1%
  • event 0.1%
System One Scoreragentcorrect82.3%0.982
  • agent 82.3%
  • species 3.9%
  • event 3.3%
  • place 3.3%
  • topical_concept 2.4%
  • work 2.1%
NanoJevagentcorrect26.0%0.660
  • agent 26.0%
  • unit_of_work 25.5%
  • topical_concept 22.4%
  • place 9.2%
  • species 6.5%
  • event 5.0%

Question l2 · choice

Read state.article_text. Within the top-level class you chose, which second-level DBpedia class fits best? The options are the second-level classes of the article's true top-level class.

26 options
  • actor — film, television or stage actor
  • artist — visual or performing artist not covered by a more specific class
  • athlete — sportsperson (generic, when no sport-specific class applies)
  • boxer — boxer
  • british_royalty — member of the British royal family
  • broadcaster — radio or television station or network
  • cleric — religious office holder
  • coach — sports coach
  • company — commercial company
  • educational_institution — school, college or university
  • fictional_character — fictional character (non-comics)
  • gridiron_football_player — American or Canadian football player
  • group — band or other group of people
  • motorcycle_rider — motorcycle racer
  • organisation — organisation not covered by a more specific class
  • organisation_member — organisation member
  • person — person not covered by a more specific class
  • politician — politician or office holder
  • racing_driver — motor racing driver
  • scientist — scientist or academic
  • sports_league — sports league
  • sports_manager — sports manager
  • sports_team — sports team or club
  • winter_sport_player — winter-sport athlete (ice hockey, skiing, ...)
  • wrestler — wrestler
  • writer — author, poet or journalist

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: artist

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevartistcorrect80.0%0.960
  • artist 80.0%
  • person 20.0%
  • writer 0.0%
  • gridiron_football_player 0.0%
  • actor 0.0%
  • fictional_character 0.0%
JevFishpersonwrong23.6%0.512
  • person 62.1%
  • artist 23.6%
  • writer 8.1%
  • organisation 2.0%
  • organisation_member 0.6%
  • fictional_character 0.3%
OpenJevartistcorrect99.7%1.000
  • artist 99.7%
  • person 0.3%
  • actor 0.0%
  • writer 0.0%
  • company 0.0%
  • organisation 0.0%
decider-2bpersonwrong20.2%0.433
  • person 70.4%
  • artist 20.2%
  • organisation 2.4%
  • organisation_member 0.8%
  • wrestler 0.8%
  • writer 0.7%
System One Scorerartistcorrect38.1%0.785
  • artist 38.1%
  • person 17.3%
  • writer 7.9%
  • organisation 7.0%
  • organisation_member 3.6%
  • company 3.2%
NanoJevorganisation_memberwrong11.0%0.569
  • organisation_member 15.3%
  • artist 11.0%
  • british_royalty 10.2%
  • person 10.2%
  • group 9.2%
  • organisation 8.9%
Sample 6 of 6 · data-and-operations:dbpedia-cfd65071fb78 · 190 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "article_text": "City Sentral was a planned major new retail and leisure development in the heart of the Staffordshire city of Stoke-on-Trent. It was proposed by Realis Estates, and was a planned 650,000 sq ft regional shopping centre which was due to open in 2016. The centre was to include a Marks & Spencer department store, a wide range of new stores and shops, cafés and restaurants, 'vibrant public spaces', a Cineworld cinema complex, an 80-room hotel, parking for 1,000 cars and a new bus station. City Sentral's proposed catchment comprised 870,000 people with more than 360,000 people living within 20 minutes drive of the city centre. The architects for the bus station were Grimshaw Architects and Benoy was working on City Sentral."
}

Question l1 · choice · primary (ranked)

Read state.article_text (the opening of a Wikipedia article). Which top-level DBpedia ontology class does the article's subject belong to?

9 options
  • agent — A person, group, organisation, company, team, character or other acting entity
  • place — A geographic or built location: settlement, building, natural feature, route, station, celestial body
  • species — A biological organism: animal, plant, fungus or other eukaryote (incl. named breeds)
  • work — A created work: album, song, book, periodical, software, comic, cartoon
  • event — A dated happening: tournament, race, sports event, societal event, natural event, Olympics
  • sports_season — One season of a sports team or league
  • unit_of_work — A legal case or similar unit of proceedings
  • device — A manufactured device such as an engine
  • topical_concept — An abstract topic such as a genre

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: place

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevplacecorrect100.0%1.000
  • place 100.0%
  • work 0.0%
  • agent 0.0%
  • topical_concept 0.0%
  • species 0.0%
  • unit_of_work 0.0%
JevFishplacecorrect88.6%0.988
  • place 88.6%
  • agent 10.8%
  • work 0.2%
  • topical_concept 0.2%
  • event 0.0%
  • species 0.0%
OpenJevplacecorrect99.9%1.000
  • place 99.9%
  • agent 0.1%
  • topical_concept 0.0%
  • species 0.0%
  • work 0.0%
  • event 0.0%
decider-2bplacecorrect93.7%0.997
  • place 93.7%
  • agent 3.2%
  • topical_concept 0.8%
  • species 0.5%
  • work 0.4%
  • event 0.4%
System One Scorerplacecorrect49.7%0.850
  • place 49.7%
  • agent 16.1%
  • event 9.7%
  • work 7.8%
  • topical_concept 5.9%
  • species 3.7%
NanoJevunit_of_workwrong14.1%0.541
  • unit_of_work 27.8%
  • agent 27.2%
  • topical_concept 15.1%
  • place 14.1%
  • species 5.5%
  • event 4.8%

Question l2 · choice

Read state.article_text. Within the top-level class you chose, which second-level DBpedia class fits best? The options are the second-level classes of the article's true top-level class.

15 options
  • amusement_park_attraction — ride or attraction in an amusement park
  • body_of_water — lake, bay, sea or other still water body
  • building — building (house, church, museum, ...)
  • celestial_body — star, planet, asteroid or galaxy
  • clerical_administrative_region — diocese or other church administrative region
  • infrastructure — dam, airport, power station, bridge or similar
  • natural_place — mountain, valley, cave, glacier or other landform
  • route_of_transportation — road, railway line or public transit route
  • satellite — artificial satellite
  • settlement — city, town or village
  • sport_facility — stadium, arena or golf course
  • station — railway or metro station
  • stream — river or stream
  • tower — tower or lighthouse
  • venue — theatre, concert hall or other venue

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: building

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevbuildingcorrect79.0%0.969
  • building 79.0%
  • venue 10.0%
  • infrastructure 9.0%
  • settlement 1.0%
  • station 1.0%
  • route_of_transportation 0.0%
JevFishbuildingcorrect54.7%0.867
  • building 54.7%
  • settlement 20.8%
  • venue 10.1%
  • infrastructure 9.1%
  • station 1.5%
  • route_of_transportation 0.9%
OpenJevbuildingcorrect99.8%1.000
  • building 99.8%
  • settlement 0.1%
  • venue 0.0%
  • body_of_water 0.0%
  • amusement_park_attraction 0.0%
  • sport_facility 0.0%
decider-2bbuildingcorrect56.9%0.863
  • building 56.9%
  • settlement 29.0%
  • infrastructure 4.5%
  • venue 2.2%
  • station 1.9%
  • route_of_transportation 1.5%
System One Scorersettlementwrong16.4%0.589
  • settlement 28.8%
  • building 16.4%
  • venue 14.3%
  • station 9.3%
  • infrastructure 5.4%
  • route_of_transportation 4.4%
NanoJevsettlementwrong5.1%0.491
  • settlement 24.6%
  • route_of_transportation 14.4%
  • satellite 10.4%
  • body_of_water 8.3%
  • amusement_park_attraction 6.8%
  • celestial_body 6.5%
  • building 5.1%