Skip to samples
Blobfish AI
BenchmarksTalk to usOpen app
Typed Decision Bench · tasks · Workflow control

hwu64-tool-select

typed tool dispatch · consumer assistant · 200 items · primary question type choice. Source deutsche-telekom/NLU-Evaluation-Data-en-de (CC-BY-4.0); labels are crowd labels, never an LLM judge. Download these samples (JSON).

SystemDecisionScoreAccuracy (top pick)Answered
Jev93.788.0%200 / 200
JevFish91.888.0%200 / 200
OpenJev91.587.0%200 / 200
decider-2b91.286.0%200 / 200
System One Scorer83.480.5%200 / 200
NanoJev67.836.0%200 / 200
Sample 1 of 6 · workflow-control:hwu-18d4d0349a5b · 12 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "utterance": "you have gotten my instructions perfectly."
}

Question tool · choice · primary (ranked)

Read the spoken command in `utterance`. Choose the single assistant tool that should handle it.

64 options
  • alarm_query — Ask what alarms are set or when an alarm will ring
  • alarm_remove — Cancel or delete an alarm
  • alarm_set — Create a new alarm or wake-up call
  • audio_volume_down — Lower the playback volume
  • audio_volume_mute — Mute or silence the audio
  • audio_volume_up — Raise the playback volume
  • calendar_query — Ask about existing calendar events, appointments or schedule
  • calendar_remove — Delete or cancel a calendar event
  • calendar_set — Add a calendar event, meeting or reminder
  • cooking_recipe — Ask for a recipe or how to cook a dish
  • datetime_convert — Convert a time between time zones
  • datetime_query — Ask the current time or date, or what day a date falls on
  • email_addcontact — Add a new contact to the address book
  • email_query — Check the inbox, read or search emails
  • email_querycontact — Look up a contact's details such as their email or phone number
  • email_sendemail — Compose and send an email or reply
  • general_affirm — Say yes or agree with the assistant
  • general_commandstop — Tell the assistant to stop what it is doing
  • general_confirm — Ask the assistant to confirm it did or understood something
  • general_dontcare — Express indifference, any option is fine
  • general_explain — Ask the assistant to explain or clarify what it said
  • general_joke — Ask the assistant to tell a joke
  • general_negate — Say no, deny or correct the assistant
  • general_praise — Praise or thank the assistant
  • general_quirky — Small talk or a quirky personal question to the assistant
  • general_repeat — Ask the assistant to repeat what it said
  • iot_cleaning — Start or control the robot vacuum cleaner
  • iot_coffee — Make coffee with the connected coffee machine
  • iot_hue_lightchange — Change the colour of the smart lights
  • iot_hue_lightdim — Dim the smart lights
  • iot_hue_lightoff — Turn the smart lights off
  • iot_hue_lighton — Turn the smart lights on
  • iot_hue_lightup — Brighten the smart lights
  • iot_wemo_off — Switch off a smart plug or socket
  • iot_wemo_on — Switch on a smart plug or socket
  • lists_createoradd — Create a list or add an item to a list
  • lists_query — Read out or check what is on a list
  • lists_remove — Remove an item from a list or delete a list
  • music_likeness — Say you like or dislike the current song or artist, or save it
  • music_query — Ask what song or artist is playing or about music details
  • music_settings — Change music playback settings such as shuffle or repeat
  • news_query — Ask for news or headlines on a topic
  • play_audiobook — Play or resume an audiobook
  • play_game — Start or play a game
  • play_music — Play a song, artist, album, genre or playlist
  • play_podcasts — Play or resume a podcast episode
  • play_radio — Play a radio station
  • qa_currency — Ask about a currency exchange rate or conversion
  • qa_definition — Ask the definition or meaning of a word
  • qa_factoid — Ask a general factual question
  • qa_maths — Ask for an arithmetic calculation
  • qa_stock — Ask about a stock price or share
  • recommendation_events — Ask for events happening nearby or recommendations of things to do
  • recommendation_locations — Ask for a recommended place such as a restaurant, shop or venue
  • recommendation_movies — Ask for a movie or show recommendation
  • social_post — Post, tweet or share something on social media
  • social_query — Check social media notifications, messages or feeds
  • takeaway_order — Order takeaway food or a delivery
  • takeaway_query — Ask about a takeaway order status or whether a place delivers
  • transport_query — Ask about train, bus or flight times, routes or directions
  • transport_taxi — Book or call a taxi or ride
  • transport_ticket — Book or buy a travel ticket
  • transport_traffic — Ask about traffic conditions on a route
  • weather_query — Ask about the weather or forecast

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: general_affirm

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevgeneral_praisewrong1.0%0.124
  • general_praise 87.0%
  • general_confirm 12.0%
  • general_affirm 1.0%
  • iot_hue_lightup 0.0%
  • transport_query 0.0%
  • social_post 0.0%
JevFishgeneral_affirmcorrect54.3%0.850
  • general_affirm 54.3%
  • general_praise 26.5%
  • general_confirm 14.6%
  • email_sendemail 0.7%
  • general_dontcare 0.6%
  • qa_factoid 0.5%
OpenJevgeneral_praisewrong0.0%0.001
  • general_praise 99.9%
  • qa_factoid 0.1%
  • play_audiobook 0.0%
  • general_quirky 0.0%
  • general_repeat 0.0%
  • alarm_query 0.0%
  • general_affirm 0.0%
decider-2bgeneral_affirmcorrect66.5%0.919
  • general_affirm 66.5%
  • general_praise 17.3%
  • general_confirm 13.6%
  • general_repeat 0.3%
  • general_commandstop 0.2%
  • calendar_query 0.2%
System One Scorergeneral_confirmwrong9.9%0.537
  • general_confirm 27.0%
  • general_praise 18.9%
  • general_affirm 9.9%
  • transport_taxi 2.7%
  • general_commandstop 2.6%
  • lists_query 2.1%
NanoJevgeneral_affirmcorrect46.4%0.836
  • general_affirm 46.4%
  • general_explain 14.6%
  • general_confirm 10.8%
  • qa_factoid 5.0%
  • recommendation_locations 4.1%
  • general_repeat 3.4%

Question date_mentioned · noul

For the same command: Answer yes if the utterance explicitly states a day or calendar date for the `date` argument, such as 'tomorrow', 'on friday', 'next week', 'the 3rd of march', 'this weekend', 'in two days'. Answer no if it gives no day or date, including when it only gives a clock time ('at 5am') or a part of the day ('tonight', 'this morning').

2 options
  • yes
  • no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnocorrect95.0%0.998
  • yes 5.0%
  • no 95.0%
JevFishnocorrect99.0%1.000
  • yes 1.0%
  • no 99.0%
OpenJevnocorrect100.0%1.000
  • yes 0.0%
  • no 100.0%
decider-2bnocorrect98.7%1.000
  • yes 1.3%
  • no 98.7%
System One Scorernocorrect74.7%0.936
  • yes 25.3%
  • no 74.7%
NanoJevyeswrong49.5%0.745
  • yes 50.5%
  • no 49.5%
Sample 2 of 6 · workflow-control:hwu-324dea76367b · 12 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "utterance": "get me new difintion."
}

Question tool · choice · primary (ranked)

Read the spoken command in `utterance`. Choose the single assistant tool that should handle it.

64 options
  • alarm_query — Ask what alarms are set or when an alarm will ring
  • alarm_remove — Cancel or delete an alarm
  • alarm_set — Create a new alarm or wake-up call
  • audio_volume_down — Lower the playback volume
  • audio_volume_mute — Mute or silence the audio
  • audio_volume_up — Raise the playback volume
  • calendar_query — Ask about existing calendar events, appointments or schedule
  • calendar_remove — Delete or cancel a calendar event
  • calendar_set — Add a calendar event, meeting or reminder
  • cooking_recipe — Ask for a recipe or how to cook a dish
  • datetime_convert — Convert a time between time zones
  • datetime_query — Ask the current time or date, or what day a date falls on
  • email_addcontact — Add a new contact to the address book
  • email_query — Check the inbox, read or search emails
  • email_querycontact — Look up a contact's details such as their email or phone number
  • email_sendemail — Compose and send an email or reply
  • general_affirm — Say yes or agree with the assistant
  • general_commandstop — Tell the assistant to stop what it is doing
  • general_confirm — Ask the assistant to confirm it did or understood something
  • general_dontcare — Express indifference, any option is fine
  • general_explain — Ask the assistant to explain or clarify what it said
  • general_joke — Ask the assistant to tell a joke
  • general_negate — Say no, deny or correct the assistant
  • general_praise — Praise or thank the assistant
  • general_quirky — Small talk or a quirky personal question to the assistant
  • general_repeat — Ask the assistant to repeat what it said
  • iot_cleaning — Start or control the robot vacuum cleaner
  • iot_coffee — Make coffee with the connected coffee machine
  • iot_hue_lightchange — Change the colour of the smart lights
  • iot_hue_lightdim — Dim the smart lights
  • iot_hue_lightoff — Turn the smart lights off
  • iot_hue_lighton — Turn the smart lights on
  • iot_hue_lightup — Brighten the smart lights
  • iot_wemo_off — Switch off a smart plug or socket
  • iot_wemo_on — Switch on a smart plug or socket
  • lists_createoradd — Create a list or add an item to a list
  • lists_query — Read out or check what is on a list
  • lists_remove — Remove an item from a list or delete a list
  • music_likeness — Say you like or dislike the current song or artist, or save it
  • music_query — Ask what song or artist is playing or about music details
  • music_settings — Change music playback settings such as shuffle or repeat
  • news_query — Ask for news or headlines on a topic
  • play_audiobook — Play or resume an audiobook
  • play_game — Start or play a game
  • play_music — Play a song, artist, album, genre or playlist
  • play_podcasts — Play or resume a podcast episode
  • play_radio — Play a radio station
  • qa_currency — Ask about a currency exchange rate or conversion
  • qa_definition — Ask the definition or meaning of a word
  • qa_factoid — Ask a general factual question
  • qa_maths — Ask for an arithmetic calculation
  • qa_stock — Ask about a stock price or share
  • recommendation_events — Ask for events happening nearby or recommendations of things to do
  • recommendation_locations — Ask for a recommended place such as a restaurant, shop or venue
  • recommendation_movies — Ask for a movie or show recommendation
  • social_post — Post, tweet or share something on social media
  • social_query — Check social media notifications, messages or feeds
  • takeaway_order — Order takeaway food or a delivery
  • takeaway_query — Ask about a takeaway order status or whether a place delivers
  • transport_query — Ask about train, bus or flight times, routes or directions
  • transport_taxi — Book or call a taxi or ride
  • transport_ticket — Book or buy a travel ticket
  • transport_traffic — Ask about traffic conditions on a route
  • weather_query — Ask about the weather or forecast

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: qa_definition

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevqa_definitioncorrect95.0%0.998
  • qa_definition 95.0%
  • general_explain 4.0%
  • general_repeat 1.0%
  • calendar_remove 0.0%
  • recommendation_locations 0.0%
  • qa_factoid 0.0%
JevFishqa_definitioncorrect96.2%0.999
  • qa_definition 96.2%
  • general_explain 1.0%
  • general_negate 0.5%
  • play_radio 0.4%
  • qa_stock 0.3%
  • general_repeat 0.2%
OpenJevqa_definitioncorrect99.9%1.000
  • qa_definition 99.9%
  • general_negate 0.1%
  • play_audiobook 0.0%
  • alarm_remove 0.0%
  • qa_currency 0.0%
  • email_sendemail 0.0%
decider-2bqa_definitioncorrect97.6%1.000
  • qa_definition 97.6%
  • general_explain 0.7%
  • qa_stock 0.4%
  • play_radio 0.4%
  • qa_currency 0.2%
  • play_audiobook 0.1%
System One Scorerqa_definitioncorrect64.1%0.932
  • qa_definition 64.1%
  • general_explain 5.9%
  • qa_factoid 4.1%
  • transport_taxi 2.3%
  • lists_query 1.5%
  • general_confirm 1.1%
NanoJevqa_definitioncorrect46.3%0.837
  • qa_definition 46.3%
  • general_explain 12.6%
  • qa_factoid 9.8%
  • general_affirm 8.4%
  • recommendation_locations 5.8%
  • news_query 4.0%

Question date_mentioned · noul

For the same command: Answer yes if the utterance explicitly states a day or calendar date for the `date` argument, such as 'tomorrow', 'on friday', 'next week', 'the 3rd of march', 'this weekend', 'in two days'. Answer no if it gives no day or date, including when it only gives a clock time ('at 5am') or a part of the day ('tonight', 'this morning').

2 options
  • yes
  • no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnocorrect94.0%0.996
  • yes 6.0%
  • no 94.0%
JevFishnocorrect98.9%1.000
  • yes 1.1%
  • no 98.9%
OpenJevnocorrect100.0%1.000
  • yes 0.0%
  • no 100.0%
decider-2bnocorrect99.2%1.000
  • yes 0.8%
  • no 99.2%
System One Scorernocorrect80.9%0.964
  • yes 19.1%
  • no 80.9%
NanoJevyeswrong49.8%0.748
  • yes 50.2%
  • no 49.8%
Sample 3 of 6 · workflow-control:hwu-34ce521e9e59 · 15 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "utterance": "I'd like to continue to the next episode."
}

Question tool · choice · primary (ranked)

Read the spoken command in `utterance`. Choose the single assistant tool that should handle it.

64 options
  • alarm_query — Ask what alarms are set or when an alarm will ring
  • alarm_remove — Cancel or delete an alarm
  • alarm_set — Create a new alarm or wake-up call
  • audio_volume_down — Lower the playback volume
  • audio_volume_mute — Mute or silence the audio
  • audio_volume_up — Raise the playback volume
  • calendar_query — Ask about existing calendar events, appointments or schedule
  • calendar_remove — Delete or cancel a calendar event
  • calendar_set — Add a calendar event, meeting or reminder
  • cooking_recipe — Ask for a recipe or how to cook a dish
  • datetime_convert — Convert a time between time zones
  • datetime_query — Ask the current time or date, or what day a date falls on
  • email_addcontact — Add a new contact to the address book
  • email_query — Check the inbox, read or search emails
  • email_querycontact — Look up a contact's details such as their email or phone number
  • email_sendemail — Compose and send an email or reply
  • general_affirm — Say yes or agree with the assistant
  • general_commandstop — Tell the assistant to stop what it is doing
  • general_confirm — Ask the assistant to confirm it did or understood something
  • general_dontcare — Express indifference, any option is fine
  • general_explain — Ask the assistant to explain or clarify what it said
  • general_joke — Ask the assistant to tell a joke
  • general_negate — Say no, deny or correct the assistant
  • general_praise — Praise or thank the assistant
  • general_quirky — Small talk or a quirky personal question to the assistant
  • general_repeat — Ask the assistant to repeat what it said
  • iot_cleaning — Start or control the robot vacuum cleaner
  • iot_coffee — Make coffee with the connected coffee machine
  • iot_hue_lightchange — Change the colour of the smart lights
  • iot_hue_lightdim — Dim the smart lights
  • iot_hue_lightoff — Turn the smart lights off
  • iot_hue_lighton — Turn the smart lights on
  • iot_hue_lightup — Brighten the smart lights
  • iot_wemo_off — Switch off a smart plug or socket
  • iot_wemo_on — Switch on a smart plug or socket
  • lists_createoradd — Create a list or add an item to a list
  • lists_query — Read out or check what is on a list
  • lists_remove — Remove an item from a list or delete a list
  • music_likeness — Say you like or dislike the current song or artist, or save it
  • music_query — Ask what song or artist is playing or about music details
  • music_settings — Change music playback settings such as shuffle or repeat
  • news_query — Ask for news or headlines on a topic
  • play_audiobook — Play or resume an audiobook
  • play_game — Start or play a game
  • play_music — Play a song, artist, album, genre or playlist
  • play_podcasts — Play or resume a podcast episode
  • play_radio — Play a radio station
  • qa_currency — Ask about a currency exchange rate or conversion
  • qa_definition — Ask the definition or meaning of a word
  • qa_factoid — Ask a general factual question
  • qa_maths — Ask for an arithmetic calculation
  • qa_stock — Ask about a stock price or share
  • recommendation_events — Ask for events happening nearby or recommendations of things to do
  • recommendation_locations — Ask for a recommended place such as a restaurant, shop or venue
  • recommendation_movies — Ask for a movie or show recommendation
  • social_post — Post, tweet or share something on social media
  • social_query — Check social media notifications, messages or feeds
  • takeaway_order — Order takeaway food or a delivery
  • takeaway_query — Ask about a takeaway order status or whether a place delivers
  • transport_query — Ask about train, bus or flight times, routes or directions
  • transport_taxi — Book or call a taxi or ride
  • transport_ticket — Book or buy a travel ticket
  • transport_traffic — Ask about traffic conditions on a route
  • weather_query — Ask about the weather or forecast

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: play_podcasts

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevplay_podcastscorrect100.0%1.000
  • play_podcasts 100.0%
  • transport_query 0.0%
  • audio_volume_mute 0.0%
  • general_confirm 0.0%
  • transport_traffic 0.0%
  • iot_hue_lightchange 0.0%
JevFishplay_podcastscorrect95.1%0.999
  • play_podcasts 95.1%
  • play_audiobook 1.7%
  • general_confirm 0.8%
  • general_dontcare 0.4%
  • general_affirm 0.4%
  • play_radio 0.4%
OpenJevplay_audiobookwrong0.1%0.002
  • play_audiobook 99.9%
  • play_podcasts 0.1%
  • iot_cleaning 0.0%
  • general_affirm 0.0%
  • play_game 0.0%
  • music_query 0.0%
decider-2bplay_podcastscorrect97.3%1.000
  • play_podcasts 97.3%
  • play_audiobook 1.9%
  • general_confirm 0.2%
  • general_affirm 0.2%
  • play_music 0.1%
  • general_dontcare 0.1%
System One Scorerplay_podcastscorrect36.5%0.784
  • play_podcasts 36.5%
  • play_audiobook 15.1%
  • transport_taxi 3.1%
  • general_confirm 3.0%
  • lists_query 2.8%
  • play_game 1.7%
NanoJevgeneral_affirmwrong3.5%0.489
  • general_affirm 22.4%
  • general_explain 12.4%
  • qa_factoid 8.1%
  • general_repeat 6.1%
  • recommendation_events 5.5%
  • recommendation_locations 5.4%
  • play_podcasts 3.5%

Question date_mentioned · noul

For the same command: Answer yes if the utterance explicitly states a day or calendar date for the `date` argument, such as 'tomorrow', 'on friday', 'next week', 'the 3rd of march', 'this weekend', 'in two days'. Answer no if it gives no day or date, including when it only gives a clock time ('at 5am') or a part of the day ('tonight', 'this morning').

2 options
  • yes
  • no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnocorrect93.0%0.995
  • yes 7.0%
  • no 93.0%
JevFishnocorrect99.6%1.000
  • yes 0.4%
  • no 99.6%
OpenJevnocorrect100.0%1.000
  • yes 0.0%
  • no 100.0%
decider-2bnocorrect99.3%1.000
  • yes 0.7%
  • no 99.3%
System One Scorernocorrect82.1%0.968
  • yes 17.9%
  • no 82.1%
NanoJevnocorrect51.4%0.764
  • yes 48.5%
  • no 51.4%
Sample 4 of 6 · workflow-control:hwu-9c506f68148c · 13 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "utterance": "that does not really matter with me."
}

Question tool · choice · primary (ranked)

Read the spoken command in `utterance`. Choose the single assistant tool that should handle it.

64 options
  • alarm_query — Ask what alarms are set or when an alarm will ring
  • alarm_remove — Cancel or delete an alarm
  • alarm_set — Create a new alarm or wake-up call
  • audio_volume_down — Lower the playback volume
  • audio_volume_mute — Mute or silence the audio
  • audio_volume_up — Raise the playback volume
  • calendar_query — Ask about existing calendar events, appointments or schedule
  • calendar_remove — Delete or cancel a calendar event
  • calendar_set — Add a calendar event, meeting or reminder
  • cooking_recipe — Ask for a recipe or how to cook a dish
  • datetime_convert — Convert a time between time zones
  • datetime_query — Ask the current time or date, or what day a date falls on
  • email_addcontact — Add a new contact to the address book
  • email_query — Check the inbox, read or search emails
  • email_querycontact — Look up a contact's details such as their email or phone number
  • email_sendemail — Compose and send an email or reply
  • general_affirm — Say yes or agree with the assistant
  • general_commandstop — Tell the assistant to stop what it is doing
  • general_confirm — Ask the assistant to confirm it did or understood something
  • general_dontcare — Express indifference, any option is fine
  • general_explain — Ask the assistant to explain or clarify what it said
  • general_joke — Ask the assistant to tell a joke
  • general_negate — Say no, deny or correct the assistant
  • general_praise — Praise or thank the assistant
  • general_quirky — Small talk or a quirky personal question to the assistant
  • general_repeat — Ask the assistant to repeat what it said
  • iot_cleaning — Start or control the robot vacuum cleaner
  • iot_coffee — Make coffee with the connected coffee machine
  • iot_hue_lightchange — Change the colour of the smart lights
  • iot_hue_lightdim — Dim the smart lights
  • iot_hue_lightoff — Turn the smart lights off
  • iot_hue_lighton — Turn the smart lights on
  • iot_hue_lightup — Brighten the smart lights
  • iot_wemo_off — Switch off a smart plug or socket
  • iot_wemo_on — Switch on a smart plug or socket
  • lists_createoradd — Create a list or add an item to a list
  • lists_query — Read out or check what is on a list
  • lists_remove — Remove an item from a list or delete a list
  • music_likeness — Say you like or dislike the current song or artist, or save it
  • music_query — Ask what song or artist is playing or about music details
  • music_settings — Change music playback settings such as shuffle or repeat
  • news_query — Ask for news or headlines on a topic
  • play_audiobook — Play or resume an audiobook
  • play_game — Start or play a game
  • play_music — Play a song, artist, album, genre or playlist
  • play_podcasts — Play or resume a podcast episode
  • play_radio — Play a radio station
  • qa_currency — Ask about a currency exchange rate or conversion
  • qa_definition — Ask the definition or meaning of a word
  • qa_factoid — Ask a general factual question
  • qa_maths — Ask for an arithmetic calculation
  • qa_stock — Ask about a stock price or share
  • recommendation_events — Ask for events happening nearby or recommendations of things to do
  • recommendation_locations — Ask for a recommended place such as a restaurant, shop or venue
  • recommendation_movies — Ask for a movie or show recommendation
  • social_post — Post, tweet or share something on social media
  • social_query — Check social media notifications, messages or feeds
  • takeaway_order — Order takeaway food or a delivery
  • takeaway_query — Ask about a takeaway order status or whether a place delivers
  • transport_query — Ask about train, bus or flight times, routes or directions
  • transport_taxi — Book or call a taxi or ride
  • transport_ticket — Book or buy a travel ticket
  • transport_traffic — Ask about traffic conditions on a route
  • weather_query — Ask about the weather or forecast

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: general_dontcare

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevgeneral_dontcarecorrect100.0%1.000
  • general_dontcare 100.0%
  • calendar_set 0.0%
  • qa_factoid 0.0%
  • general_negate 0.0%
  • transport_ticket 0.0%
  • general_quirky 0.0%
JevFishgeneral_dontcarecorrect87.8%0.990
  • general_dontcare 87.8%
  • general_negate 6.9%
  • general_explain 2.2%
  • general_confirm 0.9%
  • general_affirm 0.5%
  • general_repeat 0.1%
OpenJevgeneral_dontcarecorrect99.6%1.000
  • general_dontcare 99.6%
  • play_audiobook 0.1%
  • iot_cleaning 0.1%
  • alarm_query 0.0%
  • play_podcasts 0.0%
  • email_query 0.0%
decider-2bgeneral_dontcarecorrect94.5%0.998
  • general_dontcare 94.5%
  • general_negate 1.4%
  • general_explain 1.2%
  • general_affirm 0.7%
  • general_confirm 0.4%
  • alarm_remove 0.2%
System One Scorergeneral_dontcarecorrect20.3%0.673
  • general_dontcare 20.3%
  • general_commandstop 6.2%
  • general_confirm 4.5%
  • transport_taxi 3.6%
  • music_likeness 3.6%
  • general_affirm 3.3%
NanoJevgeneral_dontcarecorrect42.4%0.818
  • general_dontcare 42.4%
  • general_negate 10.7%
  • lists_remove 7.9%
  • general_explain 6.1%
  • qa_factoid 5.9%
  • audio_volume_mute 4.8%

Question date_mentioned · noul

For the same command: Answer yes if the utterance explicitly states a day or calendar date for the `date` argument, such as 'tomorrow', 'on friday', 'next week', 'the 3rd of march', 'this weekend', 'in two days'. Answer no if it gives no day or date, including when it only gives a clock time ('at 5am') or a part of the day ('tonight', 'this morning').

2 options
  • yes
  • no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnocorrect93.0%0.995
  • yes 7.0%
  • no 93.0%
JevFishnocorrect98.3%1.000
  • yes 1.7%
  • no 98.3%
OpenJevnocorrect100.0%1.000
  • yes 0.0%
  • no 100.0%
decider-2bnocorrect97.6%0.999
  • yes 2.4%
  • no 97.6%
System One Scorernocorrect87.3%0.984
  • yes 12.7%
  • no 87.3%
NanoJevnocorrect59.6%0.837
  • yes 40.4%
  • no 59.6%
Sample 5 of 6 · workflow-control:hwu-e09f8df2d611 · 11 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "utterance": "could you confirm that please."
}

Question tool · choice · primary (ranked)

Read the spoken command in `utterance`. Choose the single assistant tool that should handle it.

64 options
  • alarm_query — Ask what alarms are set or when an alarm will ring
  • alarm_remove — Cancel or delete an alarm
  • alarm_set — Create a new alarm or wake-up call
  • audio_volume_down — Lower the playback volume
  • audio_volume_mute — Mute or silence the audio
  • audio_volume_up — Raise the playback volume
  • calendar_query — Ask about existing calendar events, appointments or schedule
  • calendar_remove — Delete or cancel a calendar event
  • calendar_set — Add a calendar event, meeting or reminder
  • cooking_recipe — Ask for a recipe or how to cook a dish
  • datetime_convert — Convert a time between time zones
  • datetime_query — Ask the current time or date, or what day a date falls on
  • email_addcontact — Add a new contact to the address book
  • email_query — Check the inbox, read or search emails
  • email_querycontact — Look up a contact's details such as their email or phone number
  • email_sendemail — Compose and send an email or reply
  • general_affirm — Say yes or agree with the assistant
  • general_commandstop — Tell the assistant to stop what it is doing
  • general_confirm — Ask the assistant to confirm it did or understood something
  • general_dontcare — Express indifference, any option is fine
  • general_explain — Ask the assistant to explain or clarify what it said
  • general_joke — Ask the assistant to tell a joke
  • general_negate — Say no, deny or correct the assistant
  • general_praise — Praise or thank the assistant
  • general_quirky — Small talk or a quirky personal question to the assistant
  • general_repeat — Ask the assistant to repeat what it said
  • iot_cleaning — Start or control the robot vacuum cleaner
  • iot_coffee — Make coffee with the connected coffee machine
  • iot_hue_lightchange — Change the colour of the smart lights
  • iot_hue_lightdim — Dim the smart lights
  • iot_hue_lightoff — Turn the smart lights off
  • iot_hue_lighton — Turn the smart lights on
  • iot_hue_lightup — Brighten the smart lights
  • iot_wemo_off — Switch off a smart plug or socket
  • iot_wemo_on — Switch on a smart plug or socket
  • lists_createoradd — Create a list or add an item to a list
  • lists_query — Read out or check what is on a list
  • lists_remove — Remove an item from a list or delete a list
  • music_likeness — Say you like or dislike the current song or artist, or save it
  • music_query — Ask what song or artist is playing or about music details
  • music_settings — Change music playback settings such as shuffle or repeat
  • news_query — Ask for news or headlines on a topic
  • play_audiobook — Play or resume an audiobook
  • play_game — Start or play a game
  • play_music — Play a song, artist, album, genre or playlist
  • play_podcasts — Play or resume a podcast episode
  • play_radio — Play a radio station
  • qa_currency — Ask about a currency exchange rate or conversion
  • qa_definition — Ask the definition or meaning of a word
  • qa_factoid — Ask a general factual question
  • qa_maths — Ask for an arithmetic calculation
  • qa_stock — Ask about a stock price or share
  • recommendation_events — Ask for events happening nearby or recommendations of things to do
  • recommendation_locations — Ask for a recommended place such as a restaurant, shop or venue
  • recommendation_movies — Ask for a movie or show recommendation
  • social_post — Post, tweet or share something on social media
  • social_query — Check social media notifications, messages or feeds
  • takeaway_order — Order takeaway food or a delivery
  • takeaway_query — Ask about a takeaway order status or whether a place delivers
  • transport_query — Ask about train, bus or flight times, routes or directions
  • transport_taxi — Book or call a taxi or ride
  • transport_ticket — Book or buy a travel ticket
  • transport_traffic — Ask about traffic conditions on a route
  • weather_query — Ask about the weather or forecast

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: general_confirm

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevgeneral_confirmcorrect100.0%1.000
  • general_confirm 100.0%
  • datetime_query 0.0%
  • general_quirky 0.0%
  • cooking_recipe 0.0%
  • music_likeness 0.0%
  • iot_hue_lightoff 0.0%
JevFishgeneral_confirmcorrect77.6%0.967
  • general_confirm 77.6%
  • general_affirm 12.0%
  • general_explain 3.1%
  • general_repeat 2.7%
  • general_dontcare 0.5%
  • calendar_query 0.4%
OpenJevgeneral_confirmcorrect99.8%1.000
  • general_confirm 99.8%
  • play_music 0.0%
  • general_affirm 0.0%
  • play_audiobook 0.0%
  • general_repeat 0.0%
  • alarm_query 0.0%
decider-2bgeneral_confirmcorrect81.2%0.975
  • general_confirm 81.2%
  • general_affirm 11.9%
  • general_explain 1.9%
  • general_repeat 0.9%
  • calendar_query 0.7%
  • alarm_remove 0.3%
System One Scorergeneral_confirmcorrect38.3%0.802
  • general_confirm 38.3%
  • lists_query 8.0%
  • general_affirm 5.0%
  • general_explain 3.8%
  • general_repeat 2.7%
  • takeaway_query 2.7%
NanoJevgeneral_confirmcorrect25.9%0.685
  • general_confirm 25.9%
  • general_affirm 18.3%
  • qa_factoid 13.2%
  • general_explain 11.3%
  • qa_definition 10.8%
  • recommendation_locations 5.8%

Question date_mentioned · noul

For the same command: Answer yes if the utterance explicitly states a day or calendar date for the `date` argument, such as 'tomorrow', 'on friday', 'next week', 'the 3rd of march', 'this weekend', 'in two days'. Answer no if it gives no day or date, including when it only gives a clock time ('at 5am') or a part of the day ('tonight', 'this morning').

2 options
  • yes
  • no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnocorrect94.0%0.996
  • yes 6.0%
  • no 94.0%
JevFishnocorrect99.0%1.000
  • yes 1.0%
  • no 99.0%
OpenJevnocorrect100.0%1.000
  • yes 0.0%
  • no 100.0%
decider-2bnocorrect99.0%1.000
  • yes 1.0%
  • no 99.0%
System One Scorernocorrect72.5%0.924
  • yes 27.5%
  • no 72.5%
NanoJevyeswrong48.6%0.736
  • yes 51.4%
  • no 48.6%
Sample 6 of 6 · workflow-control:hwu-fdbbf272f48a · 7 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "utterance": "Lights off"
}

Question tool · choice · primary (ranked)

Read the spoken command in `utterance`. Choose the single assistant tool that should handle it.

64 options
  • alarm_query — Ask what alarms are set or when an alarm will ring
  • alarm_remove — Cancel or delete an alarm
  • alarm_set — Create a new alarm or wake-up call
  • audio_volume_down — Lower the playback volume
  • audio_volume_mute — Mute or silence the audio
  • audio_volume_up — Raise the playback volume
  • calendar_query — Ask about existing calendar events, appointments or schedule
  • calendar_remove — Delete or cancel a calendar event
  • calendar_set — Add a calendar event, meeting or reminder
  • cooking_recipe — Ask for a recipe or how to cook a dish
  • datetime_convert — Convert a time between time zones
  • datetime_query — Ask the current time or date, or what day a date falls on
  • email_addcontact — Add a new contact to the address book
  • email_query — Check the inbox, read or search emails
  • email_querycontact — Look up a contact's details such as their email or phone number
  • email_sendemail — Compose and send an email or reply
  • general_affirm — Say yes or agree with the assistant
  • general_commandstop — Tell the assistant to stop what it is doing
  • general_confirm — Ask the assistant to confirm it did or understood something
  • general_dontcare — Express indifference, any option is fine
  • general_explain — Ask the assistant to explain or clarify what it said
  • general_joke — Ask the assistant to tell a joke
  • general_negate — Say no, deny or correct the assistant
  • general_praise — Praise or thank the assistant
  • general_quirky — Small talk or a quirky personal question to the assistant
  • general_repeat — Ask the assistant to repeat what it said
  • iot_cleaning — Start or control the robot vacuum cleaner
  • iot_coffee — Make coffee with the connected coffee machine
  • iot_hue_lightchange — Change the colour of the smart lights
  • iot_hue_lightdim — Dim the smart lights
  • iot_hue_lightoff — Turn the smart lights off
  • iot_hue_lighton — Turn the smart lights on
  • iot_hue_lightup — Brighten the smart lights
  • iot_wemo_off — Switch off a smart plug or socket
  • iot_wemo_on — Switch on a smart plug or socket
  • lists_createoradd — Create a list or add an item to a list
  • lists_query — Read out or check what is on a list
  • lists_remove — Remove an item from a list or delete a list
  • music_likeness — Say you like or dislike the current song or artist, or save it
  • music_query — Ask what song or artist is playing or about music details
  • music_settings — Change music playback settings such as shuffle or repeat
  • news_query — Ask for news or headlines on a topic
  • play_audiobook — Play or resume an audiobook
  • play_game — Start or play a game
  • play_music — Play a song, artist, album, genre or playlist
  • play_podcasts — Play or resume a podcast episode
  • play_radio — Play a radio station
  • qa_currency — Ask about a currency exchange rate or conversion
  • qa_definition — Ask the definition or meaning of a word
  • qa_factoid — Ask a general factual question
  • qa_maths — Ask for an arithmetic calculation
  • qa_stock — Ask about a stock price or share
  • recommendation_events — Ask for events happening nearby or recommendations of things to do
  • recommendation_locations — Ask for a recommended place such as a restaurant, shop or venue
  • recommendation_movies — Ask for a movie or show recommendation
  • social_post — Post, tweet or share something on social media
  • social_query — Check social media notifications, messages or feeds
  • takeaway_order — Order takeaway food or a delivery
  • takeaway_query — Ask about a takeaway order status or whether a place delivers
  • transport_query — Ask about train, bus or flight times, routes or directions
  • transport_taxi — Book or call a taxi or ride
  • transport_ticket — Book or buy a travel ticket
  • transport_traffic — Ask about traffic conditions on a route
  • weather_query — Ask about the weather or forecast

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: iot_hue_lightoff

SystemTop pickGradeP(expected)Proper scoreDistribution
Jeviot_hue_lightoffcorrect100.0%1.000
  • iot_hue_lightoff 100.0%
  • email_query 0.0%
  • qa_currency 0.0%
  • iot_wemo_off 0.0%
  • transport_traffic 0.0%
  • alarm_set 0.0%
JevFishiot_hue_lightoffcorrect99.6%1.000
  • iot_hue_lightoff 99.6%
  • iot_hue_lightup 0.1%
  • iot_hue_lightdim 0.1%
  • iot_wemo_off 0.0%
  • general_negate 0.0%
  • iot_hue_lightchange 0.0%
OpenJeviot_hue_lightoffcorrect100.0%1.000
  • iot_hue_lightoff 100.0%
  • audio_volume_mute 0.0%
  • play_audiobook 0.0%
  • iot_hue_lighton 0.0%
  • datetime_convert 0.0%
  • lists_query 0.0%
decider-2biot_hue_lightoffcorrect100.0%1.000
  • iot_hue_lightoff 100.0%
  • alarm_query 0.0%
  • alarm_remove 0.0%
  • alarm_set 0.0%
  • audio_volume_down 0.0%
  • audio_volume_mute 0.0%
System One Scoreriot_hue_lightoffcorrect51.3%0.872
  • iot_hue_lightoff 51.3%
  • iot_wemo_off 12.1%
  • general_confirm 3.4%
  • general_commandstop 2.3%
  • iot_hue_lightdim 1.6%
  • transport_taxi 1.6%
NanoJevgeneral_affirmwrong9.0%0.538
  • general_affirm 22.0%
  • general_explain 13.5%
  • iot_hue_lightoff 9.0%
  • qa_definition 9.0%
  • qa_factoid 9.0%
  • general_confirm 6.4%

Question date_mentioned · noul

For the same command: Answer yes if the utterance explicitly states a day or calendar date for the `date` argument, such as 'tomorrow', 'on friday', 'next week', 'the 3rd of march', 'this weekend', 'in two days'. Answer no if it gives no day or date, including when it only gives a clock time ('at 5am') or a part of the day ('tonight', 'this morning').

2 options
  • yes
  • no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnocorrect92.0%0.994
  • yes 8.0%
  • no 92.0%
JevFishnocorrect99.6%1.000
  • yes 0.4%
  • no 99.6%
OpenJevnocorrect100.0%1.000
  • yes 0.0%
  • no 100.0%
decider-2bnocorrect99.3%1.000
  • yes 0.7%
  • no 99.3%
System One Scorernocorrect87.0%0.983
  • yes 13.0%
  • no 87.0%
NanoJevyeswrong49.8%0.748
  • yes 50.2%
  • no 49.8%