Skip to samples
Blobfish AI
BenchmarksTalk to usOpen app
Typed Decision Bench · tasks · Workflow control

hwu64-tool-select

typed tool dispatch · consumer assistant · 200 items · primary question type choice. Source deutsche-telekom/NLU-Evaluation-Data-en-de (CC-BY-4.0); labels are crowd labels, never an LLM judge. Download these samples (JSON).

SystemDecisionScoreAccuracy (top pick)Answered
Jev93.788.0%200 / 200
OpenJev91.587.0%200 / 200
decider-2b91.286.0%200 / 200
JevFish90.285.0%200 / 200
System One Scorer83.480.5%200 / 200
NanoJev67.836.0%200 / 200
Sample 1 of 6 · workflow-control:hwu-18d4d0349a5b · 12 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "utterance": "you have gotten my instructions perfectly."
}

Question tool · choice · primary (ranked)

Read the spoken command in `utterance`. Choose the single assistant tool that should handle it.

64 options
  • alarm_query — Ask what alarms are set or when an alarm will ring
  • alarm_remove — Cancel or delete an alarm
  • alarm_set — Create a new alarm or wake-up call
  • audio_volume_down — Lower the playback volume
  • audio_volume_mute — Mute or silence the audio
  • audio_volume_up — Raise the playback volume
  • calendar_query — Ask about existing calendar events, appointments or schedule
  • calendar_remove — Delete or cancel a calendar event
  • calendar_set — Add a calendar event, meeting or reminder
  • cooking_recipe — Ask for a recipe or how to cook a dish
  • datetime_convert — Convert a time between time zones
  • datetime_query — Ask the current time or date, or what day a date falls on
  • email_addcontact — Add a new contact to the address book
  • email_query — Check the inbox, read or search emails
  • email_querycontact — Look up a contact's details such as their email or phone number
  • email_sendemail — Compose and send an email or reply
  • general_affirm — Say yes or agree with the assistant
  • general_commandstop — Tell the assistant to stop what it is doing
  • general_confirm — Ask the assistant to confirm it did or understood something
  • general_dontcare — Express indifference, any option is fine
  • general_explain — Ask the assistant to explain or clarify what it said
  • general_joke — Ask the assistant to tell a joke
  • general_negate — Say no, deny or correct the assistant
  • general_praise — Praise or thank the assistant
  • general_quirky — Small talk or a quirky personal question to the assistant
  • general_repeat — Ask the assistant to repeat what it said
  • iot_cleaning — Start or control the robot vacuum cleaner
  • iot_coffee — Make coffee with the connected coffee machine
  • iot_hue_lightchange — Change the colour of the smart lights
  • iot_hue_lightdim — Dim the smart lights
  • iot_hue_lightoff — Turn the smart lights off
  • iot_hue_lighton — Turn the smart lights on
  • iot_hue_lightup — Brighten the smart lights
  • iot_wemo_off — Switch off a smart plug or socket
  • iot_wemo_on — Switch on a smart plug or socket
  • lists_createoradd — Create a list or add an item to a list
  • lists_query — Read out or check what is on a list
  • lists_remove — Remove an item from a list or delete a list
  • music_likeness — Say you like or dislike the current song or artist, or save it
  • music_query — Ask what song or artist is playing or about music details
  • music_settings — Change music playback settings such as shuffle or repeat
  • news_query — Ask for news or headlines on a topic
  • play_audiobook — Play or resume an audiobook
  • play_game — Start or play a game
  • play_music — Play a song, artist, album, genre or playlist
  • play_podcasts — Play or resume a podcast episode
  • play_radio — Play a radio station
  • qa_currency — Ask about a currency exchange rate or conversion
  • qa_definition — Ask the definition or meaning of a word
  • qa_factoid — Ask a general factual question
  • qa_maths — Ask for an arithmetic calculation
  • qa_stock — Ask about a stock price or share
  • recommendation_events — Ask for events happening nearby or recommendations of things to do
  • recommendation_locations — Ask for a recommended place such as a restaurant, shop or venue
  • recommendation_movies — Ask for a movie or show recommendation
  • social_post — Post, tweet or share something on social media
  • social_query — Check social media notifications, messages or feeds
  • takeaway_order — Order takeaway food or a delivery
  • takeaway_query — Ask about a takeaway order status or whether a place delivers
  • transport_query — Ask about train, bus or flight times, routes or directions
  • transport_taxi — Book or call a taxi or ride
  • transport_ticket — Book or buy a travel ticket
  • transport_traffic — Ask about traffic conditions on a route
  • weather_query — Ask about the weather or forecast

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: general_affirm

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevgeneral_praisewrong1.0%0.124
  • general_praise 87.0%
  • general_confirm 12.0%
  • general_affirm 1.0%
  • iot_hue_lightup 0.0%
  • transport_query 0.0%
  • social_post 0.0%
OpenJevgeneral_praisewrong0.0%0.001
  • general_praise 99.9%
  • qa_factoid 0.1%
  • play_audiobook 0.0%
  • general_quirky 0.0%
  • general_repeat 0.0%
  • alarm_query 0.0%
  • general_affirm 0.0%
decider-2bgeneral_affirmcorrect66.5%0.919
  • general_affirm 66.5%
  • general_praise 17.3%
  • general_confirm 13.6%
  • general_repeat 0.3%
  • general_commandstop 0.2%
  • calendar_query 0.2%
JevFishgeneral_confirmwrong11.3%0.454
  • general_confirm 51.9%
  • general_praise 18.0%
  • general_affirm 11.3%
  • general_repeat 5.9%
  • general_explain 1.4%
  • general_quirky 0.9%
System One Scorergeneral_confirmwrong9.9%0.537
  • general_confirm 27.0%
  • general_praise 18.9%
  • general_affirm 9.9%
  • transport_taxi 2.7%
  • general_commandstop 2.6%
  • lists_query 2.1%
NanoJevgeneral_affirmcorrect46.4%0.836
  • general_affirm 46.4%
  • general_explain 14.6%
  • general_confirm 10.8%
  • qa_factoid 5.0%
  • recommendation_locations 4.1%
  • general_repeat 3.4%

Question date_mentioned · noul

For the same command: Answer yes if the utterance explicitly states a day or calendar date for the `date` argument, such as 'tomorrow', 'on friday', 'next week', 'the 3rd of march', 'this weekend', 'in two days'. Answer no if it gives no day or date, including when it only gives a clock time ('at 5am') or a part of the day ('tonight', 'this morning').

2 options
  • yes
  • no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnocorrect95.0%0.998
  • yes 5.0%
  • no 95.0%
OpenJevnocorrect100.0%1.000
  • yes 0.0%
  • no 100.0%
decider-2bnocorrect98.7%1.000
  • yes 1.3%
  • no 98.7%
JevFishnocorrect86.9%0.983
  • yes 13.1%
  • no 86.9%
System One Scorernocorrect74.7%0.936
  • yes 25.3%
  • no 74.7%
NanoJevyeswrong49.5%0.745
  • yes 50.5%
  • no 49.5%
Sample 2 of 6 · workflow-control:hwu-34ce521e9e59 · 15 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "utterance": "I'd like to continue to the next episode."
}

Question tool · choice · primary (ranked)

Read the spoken command in `utterance`. Choose the single assistant tool that should handle it.

64 options
  • alarm_query — Ask what alarms are set or when an alarm will ring
  • alarm_remove — Cancel or delete an alarm
  • alarm_set — Create a new alarm or wake-up call
  • audio_volume_down — Lower the playback volume
  • audio_volume_mute — Mute or silence the audio
  • audio_volume_up — Raise the playback volume
  • calendar_query — Ask about existing calendar events, appointments or schedule
  • calendar_remove — Delete or cancel a calendar event
  • calendar_set — Add a calendar event, meeting or reminder
  • cooking_recipe — Ask for a recipe or how to cook a dish
  • datetime_convert — Convert a time between time zones
  • datetime_query — Ask the current time or date, or what day a date falls on
  • email_addcontact — Add a new contact to the address book
  • email_query — Check the inbox, read or search emails
  • email_querycontact — Look up a contact's details such as their email or phone number
  • email_sendemail — Compose and send an email or reply
  • general_affirm — Say yes or agree with the assistant
  • general_commandstop — Tell the assistant to stop what it is doing
  • general_confirm — Ask the assistant to confirm it did or understood something
  • general_dontcare — Express indifference, any option is fine
  • general_explain — Ask the assistant to explain or clarify what it said
  • general_joke — Ask the assistant to tell a joke
  • general_negate — Say no, deny or correct the assistant
  • general_praise — Praise or thank the assistant
  • general_quirky — Small talk or a quirky personal question to the assistant
  • general_repeat — Ask the assistant to repeat what it said
  • iot_cleaning — Start or control the robot vacuum cleaner
  • iot_coffee — Make coffee with the connected coffee machine
  • iot_hue_lightchange — Change the colour of the smart lights
  • iot_hue_lightdim — Dim the smart lights
  • iot_hue_lightoff — Turn the smart lights off
  • iot_hue_lighton — Turn the smart lights on
  • iot_hue_lightup — Brighten the smart lights
  • iot_wemo_off — Switch off a smart plug or socket
  • iot_wemo_on — Switch on a smart plug or socket
  • lists_createoradd — Create a list or add an item to a list
  • lists_query — Read out or check what is on a list
  • lists_remove — Remove an item from a list or delete a list
  • music_likeness — Say you like or dislike the current song or artist, or save it
  • music_query — Ask what song or artist is playing or about music details
  • music_settings — Change music playback settings such as shuffle or repeat
  • news_query — Ask for news or headlines on a topic
  • play_audiobook — Play or resume an audiobook
  • play_game — Start or play a game
  • play_music — Play a song, artist, album, genre or playlist
  • play_podcasts — Play or resume a podcast episode
  • play_radio — Play a radio station
  • qa_currency — Ask about a currency exchange rate or conversion
  • qa_definition — Ask the definition or meaning of a word
  • qa_factoid — Ask a general factual question
  • qa_maths — Ask for an arithmetic calculation
  • qa_stock — Ask about a stock price or share
  • recommendation_events — Ask for events happening nearby or recommendations of things to do
  • recommendation_locations — Ask for a recommended place such as a restaurant, shop or venue
  • recommendation_movies — Ask for a movie or show recommendation
  • social_post — Post, tweet or share something on social media
  • social_query — Check social media notifications, messages or feeds
  • takeaway_order — Order takeaway food or a delivery
  • takeaway_query — Ask about a takeaway order status or whether a place delivers
  • transport_query — Ask about train, bus or flight times, routes or directions
  • transport_taxi — Book or call a taxi or ride
  • transport_ticket — Book or buy a travel ticket
  • transport_traffic — Ask about traffic conditions on a route
  • weather_query — Ask about the weather or forecast

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: play_podcasts

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevplay_podcastscorrect100.0%1.000
  • play_podcasts 100.0%
  • transport_query 0.0%
  • audio_volume_mute 0.0%
  • general_confirm 0.0%
  • transport_traffic 0.0%
  • iot_hue_lightchange 0.0%
OpenJevplay_audiobookwrong0.1%0.002
  • play_audiobook 99.9%
  • play_podcasts 0.1%
  • iot_cleaning 0.0%
  • general_affirm 0.0%
  • play_game 0.0%
  • music_query 0.0%
decider-2bplay_podcastscorrect97.3%1.000
  • play_podcasts 97.3%
  • play_audiobook 1.9%
  • general_confirm 0.2%
  • general_affirm 0.2%
  • play_music 0.1%
  • general_dontcare 0.1%
JevFishplay_podcastscorrect97.8%1.000
  • play_podcasts 97.8%
  • play_audiobook 0.7%
  • general_dontcare 0.2%
  • alarm_query 0.1%
  • social_query 0.1%
  • general_affirm 0.1%
System One Scorerplay_podcastscorrect36.5%0.784
  • play_podcasts 36.5%
  • play_audiobook 15.1%
  • transport_taxi 3.1%
  • general_confirm 3.0%
  • lists_query 2.8%
  • play_game 1.7%
NanoJevgeneral_affirmwrong3.5%0.489
  • general_affirm 22.4%
  • general_explain 12.4%
  • qa_factoid 8.1%
  • general_repeat 6.1%
  • recommendation_events 5.5%
  • recommendation_locations 5.4%
  • play_podcasts 3.5%

Question date_mentioned · noul

For the same command: Answer yes if the utterance explicitly states a day or calendar date for the `date` argument, such as 'tomorrow', 'on friday', 'next week', 'the 3rd of march', 'this weekend', 'in two days'. Answer no if it gives no day or date, including when it only gives a clock time ('at 5am') or a part of the day ('tonight', 'this morning').

2 options
  • yes
  • no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnocorrect93.0%0.995
  • yes 7.0%
  • no 93.0%
OpenJevnocorrect100.0%1.000
  • yes 0.0%
  • no 100.0%
decider-2bnocorrect99.3%1.000
  • yes 0.7%
  • no 99.3%
JevFishnocorrect90.2%0.990
  • yes 9.8%
  • no 90.2%
System One Scorernocorrect82.1%0.968
  • yes 17.9%
  • no 82.1%
NanoJevnocorrect51.4%0.764
  • yes 48.5%
  • no 51.4%
Sample 3 of 6 · workflow-control:hwu-a313550076b2 · 14 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "utterance": "olly that is all, just stop it."
}

Question tool · choice · primary (ranked)

Read the spoken command in `utterance`. Choose the single assistant tool that should handle it.

64 options
  • alarm_query — Ask what alarms are set or when an alarm will ring
  • alarm_remove — Cancel or delete an alarm
  • alarm_set — Create a new alarm or wake-up call
  • audio_volume_down — Lower the playback volume
  • audio_volume_mute — Mute or silence the audio
  • audio_volume_up — Raise the playback volume
  • calendar_query — Ask about existing calendar events, appointments or schedule
  • calendar_remove — Delete or cancel a calendar event
  • calendar_set — Add a calendar event, meeting or reminder
  • cooking_recipe — Ask for a recipe or how to cook a dish
  • datetime_convert — Convert a time between time zones
  • datetime_query — Ask the current time or date, or what day a date falls on
  • email_addcontact — Add a new contact to the address book
  • email_query — Check the inbox, read or search emails
  • email_querycontact — Look up a contact's details such as their email or phone number
  • email_sendemail — Compose and send an email or reply
  • general_affirm — Say yes or agree with the assistant
  • general_commandstop — Tell the assistant to stop what it is doing
  • general_confirm — Ask the assistant to confirm it did or understood something
  • general_dontcare — Express indifference, any option is fine
  • general_explain — Ask the assistant to explain or clarify what it said
  • general_joke — Ask the assistant to tell a joke
  • general_negate — Say no, deny or correct the assistant
  • general_praise — Praise or thank the assistant
  • general_quirky — Small talk or a quirky personal question to the assistant
  • general_repeat — Ask the assistant to repeat what it said
  • iot_cleaning — Start or control the robot vacuum cleaner
  • iot_coffee — Make coffee with the connected coffee machine
  • iot_hue_lightchange — Change the colour of the smart lights
  • iot_hue_lightdim — Dim the smart lights
  • iot_hue_lightoff — Turn the smart lights off
  • iot_hue_lighton — Turn the smart lights on
  • iot_hue_lightup — Brighten the smart lights
  • iot_wemo_off — Switch off a smart plug or socket
  • iot_wemo_on — Switch on a smart plug or socket
  • lists_createoradd — Create a list or add an item to a list
  • lists_query — Read out or check what is on a list
  • lists_remove — Remove an item from a list or delete a list
  • music_likeness — Say you like or dislike the current song or artist, or save it
  • music_query — Ask what song or artist is playing or about music details
  • music_settings — Change music playback settings such as shuffle or repeat
  • news_query — Ask for news or headlines on a topic
  • play_audiobook — Play or resume an audiobook
  • play_game — Start or play a game
  • play_music — Play a song, artist, album, genre or playlist
  • play_podcasts — Play or resume a podcast episode
  • play_radio — Play a radio station
  • qa_currency — Ask about a currency exchange rate or conversion
  • qa_definition — Ask the definition or meaning of a word
  • qa_factoid — Ask a general factual question
  • qa_maths — Ask for an arithmetic calculation
  • qa_stock — Ask about a stock price or share
  • recommendation_events — Ask for events happening nearby or recommendations of things to do
  • recommendation_locations — Ask for a recommended place such as a restaurant, shop or venue
  • recommendation_movies — Ask for a movie or show recommendation
  • social_post — Post, tweet or share something on social media
  • social_query — Check social media notifications, messages or feeds
  • takeaway_order — Order takeaway food or a delivery
  • takeaway_query — Ask about a takeaway order status or whether a place delivers
  • transport_query — Ask about train, bus or flight times, routes or directions
  • transport_taxi — Book or call a taxi or ride
  • transport_ticket — Book or buy a travel ticket
  • transport_traffic — Ask about traffic conditions on a route
  • weather_query — Ask about the weather or forecast

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: general_commandstop

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevgeneral_commandstopcorrect100.0%1.000
  • general_commandstop 100.0%
  • play_podcasts 0.0%
  • calendar_set 0.0%
  • iot_wemo_on 0.0%
  • play_radio 0.0%
  • audio_volume_mute 0.0%
OpenJevgeneral_commandstopcorrect99.9%1.000
  • general_commandstop 99.9%
  • play_audiobook 0.0%
  • play_game 0.0%
  • alarm_query 0.0%
  • iot_cleaning 0.0%
  • general_confirm 0.0%
decider-2bgeneral_commandstopcorrect95.8%0.999
  • general_commandstop 95.8%
  • audio_volume_mute 2.3%
  • alarm_remove 0.7%
  • audio_volume_down 0.1%
  • general_confirm 0.1%
  • play_game 0.1%
JevFishgeneral_commandstopcorrect84.0%0.986
  • general_commandstop 84.0%
  • general_dontcare 3.2%
  • general_negate 1.8%
  • calendar_remove 0.8%
  • email_sendemail 0.7%
  • general_praise 0.7%
System One Scorergeneral_commandstopcorrect53.6%0.888
  • general_commandstop 53.6%
  • audio_volume_mute 5.3%
  • general_negate 3.5%
  • iot_hue_lightoff 3.3%
  • iot_wemo_off 2.4%
  • music_likeness 1.9%
NanoJevgeneral_commandstopcorrect62.0%0.913
  • general_commandstop 62.0%
  • lists_remove 15.7%
  • general_explain 3.8%
  • audio_volume_mute 3.0%
  • general_affirm 3.0%
  • qa_factoid 2.1%

Question date_mentioned · noul

For the same command: Answer yes if the utterance explicitly states a day or calendar date for the `date` argument, such as 'tomorrow', 'on friday', 'next week', 'the 3rd of march', 'this weekend', 'in two days'. Answer no if it gives no day or date, including when it only gives a clock time ('at 5am') or a part of the day ('tonight', 'this morning').

2 options
  • yes
  • no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnocorrect94.0%0.996
  • yes 6.0%
  • no 94.0%
OpenJevnocorrect100.0%1.000
  • yes 0.0%
  • no 100.0%
decider-2bnocorrect98.7%1.000
  • yes 1.3%
  • no 98.7%
JevFishnocorrect88.1%0.986
  • yes 11.9%
  • no 88.1%
System One Scorernocorrect83.1%0.971
  • yes 16.9%
  • no 83.1%
NanoJevnocorrect52.9%0.778
  • yes 47.1%
  • no 52.9%
Sample 4 of 6 · workflow-control:hwu-bc4a35642d8c · 13 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "utterance": "What will be the weather day after?"
}

Question tool · choice · primary (ranked)

Read the spoken command in `utterance`. Choose the single assistant tool that should handle it.

64 options
  • alarm_query — Ask what alarms are set or when an alarm will ring
  • alarm_remove — Cancel or delete an alarm
  • alarm_set — Create a new alarm or wake-up call
  • audio_volume_down — Lower the playback volume
  • audio_volume_mute — Mute or silence the audio
  • audio_volume_up — Raise the playback volume
  • calendar_query — Ask about existing calendar events, appointments or schedule
  • calendar_remove — Delete or cancel a calendar event
  • calendar_set — Add a calendar event, meeting or reminder
  • cooking_recipe — Ask for a recipe or how to cook a dish
  • datetime_convert — Convert a time between time zones
  • datetime_query — Ask the current time or date, or what day a date falls on
  • email_addcontact — Add a new contact to the address book
  • email_query — Check the inbox, read or search emails
  • email_querycontact — Look up a contact's details such as their email or phone number
  • email_sendemail — Compose and send an email or reply
  • general_affirm — Say yes or agree with the assistant
  • general_commandstop — Tell the assistant to stop what it is doing
  • general_confirm — Ask the assistant to confirm it did or understood something
  • general_dontcare — Express indifference, any option is fine
  • general_explain — Ask the assistant to explain or clarify what it said
  • general_joke — Ask the assistant to tell a joke
  • general_negate — Say no, deny or correct the assistant
  • general_praise — Praise or thank the assistant
  • general_quirky — Small talk or a quirky personal question to the assistant
  • general_repeat — Ask the assistant to repeat what it said
  • iot_cleaning — Start or control the robot vacuum cleaner
  • iot_coffee — Make coffee with the connected coffee machine
  • iot_hue_lightchange — Change the colour of the smart lights
  • iot_hue_lightdim — Dim the smart lights
  • iot_hue_lightoff — Turn the smart lights off
  • iot_hue_lighton — Turn the smart lights on
  • iot_hue_lightup — Brighten the smart lights
  • iot_wemo_off — Switch off a smart plug or socket
  • iot_wemo_on — Switch on a smart plug or socket
  • lists_createoradd — Create a list or add an item to a list
  • lists_query — Read out or check what is on a list
  • lists_remove — Remove an item from a list or delete a list
  • music_likeness — Say you like or dislike the current song or artist, or save it
  • music_query — Ask what song or artist is playing or about music details
  • music_settings — Change music playback settings such as shuffle or repeat
  • news_query — Ask for news or headlines on a topic
  • play_audiobook — Play or resume an audiobook
  • play_game — Start or play a game
  • play_music — Play a song, artist, album, genre or playlist
  • play_podcasts — Play or resume a podcast episode
  • play_radio — Play a radio station
  • qa_currency — Ask about a currency exchange rate or conversion
  • qa_definition — Ask the definition or meaning of a word
  • qa_factoid — Ask a general factual question
  • qa_maths — Ask for an arithmetic calculation
  • qa_stock — Ask about a stock price or share
  • recommendation_events — Ask for events happening nearby or recommendations of things to do
  • recommendation_locations — Ask for a recommended place such as a restaurant, shop or venue
  • recommendation_movies — Ask for a movie or show recommendation
  • social_post — Post, tweet or share something on social media
  • social_query — Check social media notifications, messages or feeds
  • takeaway_order — Order takeaway food or a delivery
  • takeaway_query — Ask about a takeaway order status or whether a place delivers
  • transport_query — Ask about train, bus or flight times, routes or directions
  • transport_taxi — Book or call a taxi or ride
  • transport_ticket — Book or buy a travel ticket
  • transport_traffic — Ask about traffic conditions on a route
  • weather_query — Ask about the weather or forecast

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: weather_query

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevweather_querycorrect100.0%1.000
  • weather_query 100.0%
  • email_querycontact 0.0%
  • iot_wemo_on 0.0%
  • cooking_recipe 0.0%
  • calendar_set 0.0%
  • lists_createoradd 0.0%
OpenJevweather_querycorrect99.9%1.000
  • weather_query 99.9%
  • play_audiobook 0.1%
  • transport_taxi 0.0%
  • general_affirm 0.0%
  • social_post 0.0%
  • datetime_query 0.0%
decider-2bweather_querycorrect92.8%0.997
  • weather_query 92.8%
  • datetime_query 3.2%
  • iot_coffee 0.5%
  • general_dontcare 0.4%
  • alarm_remove 0.4%
  • qa_factoid 0.3%
JevFishweather_querycorrect94.8%0.999
  • weather_query 94.8%
  • datetime_query 0.5%
  • lists_remove 0.3%
  • takeaway_query 0.3%
  • takeaway_order 0.2%
  • transport_traffic 0.2%
System One Scorerweather_querycorrect58.3%0.911
  • weather_query 58.3%
  • qa_factoid 2.9%
  • transport_taxi 2.4%
  • datetime_query 2.2%
  • general_confirm 1.7%
  • lists_query 1.6%
NanoJevweather_querycorrect52.5%0.871
  • weather_query 52.5%
  • qa_factoid 12.2%
  • general_affirm 9.6%
  • general_explain 7.0%
  • general_confirm 2.9%
  • qa_definition 2.7%

Question date_mentioned · noul

For the same command: Answer yes if the utterance explicitly states a day or calendar date for the `date` argument, such as 'tomorrow', 'on friday', 'next week', 'the 3rd of march', 'this weekend', 'in two days'. Answer no if it gives no day or date, including when it only gives a clock time ('at 5am') or a part of the day ('tonight', 'this morning').

2 options
  • yes
  • no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevyescorrect75.0%0.938
  • yes 75.0%
  • no 25.0%
OpenJevnowrong27.3%0.471
  • yes 27.3%
  • no 72.7%
decider-2bnowrong16.3%0.300
  • yes 16.3%
  • no 83.7%
JevFishnowrong39.4%0.632
  • yes 39.4%
  • no 60.7%
System One Scoreryescorrect58.4%0.827
  • yes 58.4%
  • no 41.6%
NanoJevyescorrect52.7%0.776
  • yes 52.7%
  • no 47.3%
Sample 5 of 6 · workflow-control:hwu-e3dc5c4d7d18 · 12 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "utterance": "Read out my to do list."
}

Question tool · choice · primary (ranked)

Read the spoken command in `utterance`. Choose the single assistant tool that should handle it.

64 options
  • alarm_query — Ask what alarms are set or when an alarm will ring
  • alarm_remove — Cancel or delete an alarm
  • alarm_set — Create a new alarm or wake-up call
  • audio_volume_down — Lower the playback volume
  • audio_volume_mute — Mute or silence the audio
  • audio_volume_up — Raise the playback volume
  • calendar_query — Ask about existing calendar events, appointments or schedule
  • calendar_remove — Delete or cancel a calendar event
  • calendar_set — Add a calendar event, meeting or reminder
  • cooking_recipe — Ask for a recipe or how to cook a dish
  • datetime_convert — Convert a time between time zones
  • datetime_query — Ask the current time or date, or what day a date falls on
  • email_addcontact — Add a new contact to the address book
  • email_query — Check the inbox, read or search emails
  • email_querycontact — Look up a contact's details such as their email or phone number
  • email_sendemail — Compose and send an email or reply
  • general_affirm — Say yes or agree with the assistant
  • general_commandstop — Tell the assistant to stop what it is doing
  • general_confirm — Ask the assistant to confirm it did or understood something
  • general_dontcare — Express indifference, any option is fine
  • general_explain — Ask the assistant to explain or clarify what it said
  • general_joke — Ask the assistant to tell a joke
  • general_negate — Say no, deny or correct the assistant
  • general_praise — Praise or thank the assistant
  • general_quirky — Small talk or a quirky personal question to the assistant
  • general_repeat — Ask the assistant to repeat what it said
  • iot_cleaning — Start or control the robot vacuum cleaner
  • iot_coffee — Make coffee with the connected coffee machine
  • iot_hue_lightchange — Change the colour of the smart lights
  • iot_hue_lightdim — Dim the smart lights
  • iot_hue_lightoff — Turn the smart lights off
  • iot_hue_lighton — Turn the smart lights on
  • iot_hue_lightup — Brighten the smart lights
  • iot_wemo_off — Switch off a smart plug or socket
  • iot_wemo_on — Switch on a smart plug or socket
  • lists_createoradd — Create a list or add an item to a list
  • lists_query — Read out or check what is on a list
  • lists_remove — Remove an item from a list or delete a list
  • music_likeness — Say you like or dislike the current song or artist, or save it
  • music_query — Ask what song or artist is playing or about music details
  • music_settings — Change music playback settings such as shuffle or repeat
  • news_query — Ask for news or headlines on a topic
  • play_audiobook — Play or resume an audiobook
  • play_game — Start or play a game
  • play_music — Play a song, artist, album, genre or playlist
  • play_podcasts — Play or resume a podcast episode
  • play_radio — Play a radio station
  • qa_currency — Ask about a currency exchange rate or conversion
  • qa_definition — Ask the definition or meaning of a word
  • qa_factoid — Ask a general factual question
  • qa_maths — Ask for an arithmetic calculation
  • qa_stock — Ask about a stock price or share
  • recommendation_events — Ask for events happening nearby or recommendations of things to do
  • recommendation_locations — Ask for a recommended place such as a restaurant, shop or venue
  • recommendation_movies — Ask for a movie or show recommendation
  • social_post — Post, tweet or share something on social media
  • social_query — Check social media notifications, messages or feeds
  • takeaway_order — Order takeaway food or a delivery
  • takeaway_query — Ask about a takeaway order status or whether a place delivers
  • transport_query — Ask about train, bus or flight times, routes or directions
  • transport_taxi — Book or call a taxi or ride
  • transport_ticket — Book or buy a travel ticket
  • transport_traffic — Ask about traffic conditions on a route
  • weather_query — Ask about the weather or forecast

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: lists_query

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevlists_querycorrect100.0%1.000
  • lists_query 100.0%
  • qa_currency 0.0%
  • play_radio 0.0%
  • music_query 0.0%
  • calendar_remove 0.0%
  • iot_wemo_on 0.0%
OpenJevlists_querycorrect100.0%1.000
  • lists_query 100.0%
  • play_audiobook 0.0%
  • datetime_convert 0.0%
  • lists_createoradd 0.0%
  • music_query 0.0%
  • lists_remove 0.0%
decider-2blists_querycorrect98.2%1.000
  • lists_query 98.2%
  • lists_createoradd 0.4%
  • calendar_query 0.2%
  • iot_hue_lightchange 0.2%
  • play_audiobook 0.1%
  • qa_factoid 0.1%
JevFishlists_querycorrect98.0%1.000
  • lists_query 98.0%
  • play_audiobook 0.7%
  • lists_createoradd 0.1%
  • qa_factoid 0.1%
  • qa_stock 0.1%
  • iot_hue_lightchange 0.1%
System One Scorerlists_querycorrect58.2%0.910
  • lists_query 58.2%
  • lists_createoradd 2.9%
  • transport_taxi 2.9%
  • general_confirm 2.0%
  • recommendation_events 1.7%
  • takeaway_query 1.6%
NanoJevlists_querycorrect54.4%0.883
  • lists_query 54.4%
  • general_affirm 12.3%
  • general_explain 5.7%
  • recommendation_locations 4.8%
  • qa_factoid 3.9%
  • recommendation_events 3.6%

Question date_mentioned · noul

For the same command: Answer yes if the utterance explicitly states a day or calendar date for the `date` argument, such as 'tomorrow', 'on friday', 'next week', 'the 3rd of march', 'this weekend', 'in two days'. Answer no if it gives no day or date, including when it only gives a clock time ('at 5am') or a part of the day ('tonight', 'this morning').

2 options
  • yes
  • no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnocorrect94.0%0.996
  • yes 6.0%
  • no 94.0%
OpenJevnocorrect100.0%1.000
  • yes 0.0%
  • no 100.0%
decider-2bnocorrect99.1%1.000
  • yes 0.9%
  • no 99.1%
JevFishnocorrect90.0%0.990
  • yes 10.0%
  • no 90.0%
System One Scorernocorrect76.6%0.945
  • yes 23.4%
  • no 76.6%
NanoJevnocorrect51.1%0.761
  • yes 48.9%
  • no 51.1%
Sample 6 of 6 · workflow-control:hwu-f9002068b8ff · 15 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)
{
  "utterance": "Please paly for me Elvis Preley song again"
}

Question tool · choice · primary (ranked)

Read the spoken command in `utterance`. Choose the single assistant tool that should handle it.

64 options
  • alarm_query — Ask what alarms are set or when an alarm will ring
  • alarm_remove — Cancel or delete an alarm
  • alarm_set — Create a new alarm or wake-up call
  • audio_volume_down — Lower the playback volume
  • audio_volume_mute — Mute or silence the audio
  • audio_volume_up — Raise the playback volume
  • calendar_query — Ask about existing calendar events, appointments or schedule
  • calendar_remove — Delete or cancel a calendar event
  • calendar_set — Add a calendar event, meeting or reminder
  • cooking_recipe — Ask for a recipe or how to cook a dish
  • datetime_convert — Convert a time between time zones
  • datetime_query — Ask the current time or date, or what day a date falls on
  • email_addcontact — Add a new contact to the address book
  • email_query — Check the inbox, read or search emails
  • email_querycontact — Look up a contact's details such as their email or phone number
  • email_sendemail — Compose and send an email or reply
  • general_affirm — Say yes or agree with the assistant
  • general_commandstop — Tell the assistant to stop what it is doing
  • general_confirm — Ask the assistant to confirm it did or understood something
  • general_dontcare — Express indifference, any option is fine
  • general_explain — Ask the assistant to explain or clarify what it said
  • general_joke — Ask the assistant to tell a joke
  • general_negate — Say no, deny or correct the assistant
  • general_praise — Praise or thank the assistant
  • general_quirky — Small talk or a quirky personal question to the assistant
  • general_repeat — Ask the assistant to repeat what it said
  • iot_cleaning — Start or control the robot vacuum cleaner
  • iot_coffee — Make coffee with the connected coffee machine
  • iot_hue_lightchange — Change the colour of the smart lights
  • iot_hue_lightdim — Dim the smart lights
  • iot_hue_lightoff — Turn the smart lights off
  • iot_hue_lighton — Turn the smart lights on
  • iot_hue_lightup — Brighten the smart lights
  • iot_wemo_off — Switch off a smart plug or socket
  • iot_wemo_on — Switch on a smart plug or socket
  • lists_createoradd — Create a list or add an item to a list
  • lists_query — Read out or check what is on a list
  • lists_remove — Remove an item from a list or delete a list
  • music_likeness — Say you like or dislike the current song or artist, or save it
  • music_query — Ask what song or artist is playing or about music details
  • music_settings — Change music playback settings such as shuffle or repeat
  • news_query — Ask for news or headlines on a topic
  • play_audiobook — Play or resume an audiobook
  • play_game — Start or play a game
  • play_music — Play a song, artist, album, genre or playlist
  • play_podcasts — Play or resume a podcast episode
  • play_radio — Play a radio station
  • qa_currency — Ask about a currency exchange rate or conversion
  • qa_definition — Ask the definition or meaning of a word
  • qa_factoid — Ask a general factual question
  • qa_maths — Ask for an arithmetic calculation
  • qa_stock — Ask about a stock price or share
  • recommendation_events — Ask for events happening nearby or recommendations of things to do
  • recommendation_locations — Ask for a recommended place such as a restaurant, shop or venue
  • recommendation_movies — Ask for a movie or show recommendation
  • social_post — Post, tweet or share something on social media
  • social_query — Check social media notifications, messages or feeds
  • takeaway_order — Order takeaway food or a delivery
  • takeaway_query — Ask about a takeaway order status or whether a place delivers
  • transport_query — Ask about train, bus or flight times, routes or directions
  • transport_taxi — Book or call a taxi or ride
  • transport_ticket — Book or buy a travel ticket
  • transport_traffic — Ask about traffic conditions on a route
  • weather_query — Ask about the weather or forecast

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: play_music

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevplay_musiccorrect100.0%1.000
  • play_music 100.0%
  • general_praise 0.0%
  • iot_coffee 0.0%
  • recommendation_locations 0.0%
  • iot_wemo_off 0.0%
  • qa_definition 0.0%
OpenJevplay_musiccorrect99.8%1.000
  • play_music 99.8%
  • general_confirm 0.1%
  • play_audiobook 0.1%
  • music_likeness 0.0%
  • news_query 0.0%
  • music_query 0.0%
decider-2bplay_musiccorrect97.2%1.000
  • play_music 97.2%
  • general_confirm 0.7%
  • qa_stock 0.4%
  • music_query 0.3%
  • general_repeat 0.2%
  • play_game 0.2%
JevFishmusic_settingswrong2.8%0.139
  • music_settings 88.0%
  • general_repeat 5.2%
  • qa_stock 2.8%
  • play_music 2.8%
  • email_querycontact 0.2%
  • play_audiobook 0.1%
System One Scorerplay_musiccorrect22.5%0.692
  • play_music 22.5%
  • general_confirm 4.9%
  • music_query 4.0%
  • transport_taxi 3.7%
  • music_likeness 3.5%
  • takeaway_query 2.8%
NanoJevplay_musiccorrect35.0%0.743
  • play_music 35.0%
  • music_query 27.3%
  • general_affirm 9.3%
  • general_repeat 6.4%
  • general_explain 4.2%
  • music_likeness 3.0%

Question date_mentioned · noul

For the same command: Answer yes if the utterance explicitly states a day or calendar date for the `date` argument, such as 'tomorrow', 'on friday', 'next week', 'the 3rd of march', 'this weekend', 'in two days'. Answer no if it gives no day or date, including when it only gives a clock time ('at 5am') or a part of the day ('tonight', 'this morning').

2 options
  • yes
  • no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

SystemTop pickGradeP(expected)Proper scoreDistribution
Jevnocorrect93.0%0.995
  • yes 7.0%
  • no 93.0%
OpenJevnocorrect100.0%1.000
  • yes 0.0%
  • no 100.0%
decider-2bnocorrect99.4%1.000
  • yes 0.6%
  • no 99.4%
JevFishnocorrect91.6%0.993
  • yes 8.4%
  • no 91.6%
System One Scorernocorrect86.5%0.982
  • yes 13.5%
  • no 86.5%
NanoJevnocorrect52.3%0.773
  • yes 47.7%
  • no 52.3%