Typed Decision Bench · tasks · Real-time and agents

`tau2-airline-run-success`

run judging · customer operations · 200 items · primary question type noul. Source snorkelai/Tau2-Bench-Verified-Airline-With-Code-Agents (Apache-2.0); labels are verifier labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
NanoJev	75.5	50.0%	198 / 200
System One Scorer	71.8	48.5%	200 / 200
Jev	68.0	52.0%	200 / 200
decider-2b	67.3	47.0%	200 / 200
JevFish	64.9	51.5%	200 / 200
OpenJev	55.3	47.5%	200 / 200

Sample 1 of 6 · real-time-and-agents:tau2-18084431e56a · 334 state tokens (long tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "domain": "airline",
  "customer_goal": "For your upcoming trip from New York to Chicago, you want to change the passenger to yourself, upgrade it to economy class, and have 3 checked bags.",
  "db_update_required": true,
  "n_turns": 24,
  "tool_sequence": [
    "get_user_details",
    "get_reservation_details",
    "get_reservation_details",
    "get_reservation_details",
    "get_reservation_details",
    "update_reservation_passengers",
    "update_reservation_flights",
    "update_reservation_baggages"
  ],
  "last_actions": [
    {
      "name": "update_reservation_flights",
      "args": "{\"reservation_id\": \"FQ8APE\", \"cabin\": \"economy\", \"flights\": [{\"flight_number\": \"HAT056\", \"date\": \"2024-05-25\"}, …"
    },
    {
      "name": "update_reservation_baggages",
      "args": "{\"reservation_id\": \"FQ8APE\", \"total_baggages\": 3, \"nonfree_baggages\": 0, \"payment_id\": \"gift_card_8190333\"}"
    }
  ],
  "last_user_message": "No, that’s all I needed. Thank you for your help! ###STOP###",
  "final_agent_message": "Your reservation **FQ8APE** has been successfully updated with the following changes:\n1. **Passenger**: Changed to Omar Rossi (DOB: 1970-06-06)\n2. **Cabin Class**: Upgraded to Economy \n3. **Baggage**: Added 3 checked bags (all free as a gold member)\n\nThe total cost for the cabin upgrade was $209, …"
}

Question success · noul · primary (ranked)

Given `customer_goal`, `db_update_required`, `tool_sequence`, `last_actions`, `last_user_message` and `final_agent_message`, did the agent end the session with the airline database in the correct state and with the required information communicated to the customer, according to airline policy?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	69.7%	0.908	`yes` 69.7% `no` 30.3%
System One Scorer	`yes` input truncated	correct	81.8%	0.967	`yes` 81.8% `no` 18.2%
Jev	`yes`	correct	60.0%	0.840	`yes` 60.0% `no` 40.0%
decider-2b	`yes`	correct	91.7%	0.993	`yes` 91.7% `no` 8.3%
JevFish	`yes`	correct	53.4%	0.783	`yes` 53.4% `no` 46.6%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%

Question db_correct · noul

Did the session end with the airline database in exactly the state the customer's request and airline policy require (no missing and no extra changes)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	60.4%	0.844	`yes` 60.4% `no` 39.6%
System One Scorer	`yes` input truncated	correct	81.2%	0.965	`yes` 81.2% `no` 18.8%
Jev	`yes`	correct	53.0%	0.779	`yes` 53.0% `no` 47.0%
decider-2b	`yes`	correct	95.8%	0.998	`yes` 95.8% `no` 4.2%
JevFish	`yes`	correct	63.2%	0.865	`yes` 63.2% `no` 36.8%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%

Question communicated · noul

Did the agent tell the customer every piece of information the request required?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	62.9%	0.862	`yes` 62.9% `no` 37.1%
System One Scorer	`yes` input truncated	correct	67.8%	0.897	`yes` 67.8% `no` 32.2%
Jev	`yes`	correct	76.0%	0.942	`yes` 76.0% `no` 24.0%
decider-2b	`yes`	correct	96.2%	0.999	`yes` 96.2% `no` 3.8%
JevFish	`yes`	correct	72.1%	0.922	`yes` 72.1% `no` 27.9%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%

Sample 2 of 6 · real-time-and-agents:tau2-40ff226dd546 · 241 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "domain": "airline",
  "customer_goal": "You have an upcoming flight from Boston to Minneapolis under reservation ID YAX4DR.\n\nYou want to change your class for all passengers to business.\n\nYou also want to add 2 checked bags under your name …",
  "db_update_required": true,
  "n_turns": 20,
  "tool_sequence": [
    "get_reservation_details",
    "search_onestop_flight",
    "update_reservation_baggages"
  ],
  "last_actions": [
    {
      "name": "search_onestop_flight",
      "args": "{\"origin\": \"BOS\", \"destination\": \"MSP\", \"date\": \"2024-05-18\"}"
    },
    {
      "name": "update_reservation_baggages",
      "args": "{\"reservation_id\": \"YAX4DR\", \"total_baggages\": 2, \"nonfree_baggages\": 0, \"payment_id\": \"credit_card_4938634\"}"
    }
  ],
  "last_user_message": "###STOP###",
  "final_agent_message": "You're welcome! If you have any more questions in the future or need assistance, feel free to reach out. Have a wonderful trip from Boston to Minneapolis! Safe travels! ✈️"
}

Question success · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	70.7%	0.914	`yes` 70.7% `no` 29.3%
System One Scorer	`yes`	correct	78.6%	0.954	`yes` 78.6% `no` 21.4%
Jev	`no`	wrong	10.0%	0.190	`yes` 10.0% `no` 90.0%
decider-2b	`yes`	correct	79.3%	0.957	`yes` 79.3% `no` 20.6%
JevFish	`yes`	correct	58.7%	0.830	`yes` 58.7% `no` 41.3%
OpenJev	`no`	wrong	0.1%	0.001	`yes` 0.1% `no` 99.9%

Question db_correct · noul

Did the session end with the airline database in exactly the state the customer's request and airline policy require (no missing and no extra changes)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	59.4%	0.835	`yes` 59.4% `no` 40.6%
System One Scorer	`yes`	correct	74.3%	0.934	`yes` 74.3% `no` 25.7%
Jev	`no`	wrong	10.0%	0.190	`yes` 10.0% `no` 90.0%
decider-2b	`yes`	correct	78.5%	0.954	`yes` 78.5% `no` 21.4%
JevFish	`yes`	correct	59.4%	0.835	`yes` 59.4% `no` 40.6%
OpenJev	`no`	wrong	0.0%	0.001	`yes` 0.0% `no` 100.0%

Question communicated · noul

Did the agent tell the customer every piece of information the request required?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	62.4%	0.859	`yes` 62.4% `no` 37.6%
System One Scorer	`yes`	correct	62.9%	0.863	`yes` 62.9% `no` 37.0%
Jev	`no`	wrong	13.0%	0.243	`yes` 13.0% `no` 87.0%
decider-2b	`yes`	correct	67.3%	0.893	`yes` 67.3% `no` 32.7%
JevFish	`yes`	correct	52.7%	0.776	`yes` 52.7% `no` 47.3%
OpenJev	`no`	wrong	0.0%	0.000	`yes` 0.0% `no` 100.0%

Sample 3 of 6 · real-time-and-agents:tau2-6a435b1f1e32 · 284 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "domain": "airline",
  "customer_goal": "You just faced some money issue and want to downgrade all business flights to economy, without changing the flights or passengers.",
  "db_update_required": true,
  "n_turns": 32,
  "tool_sequence": [
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation"
  ],
  "last_actions": [
    {
      "name": "code_evaluation",
      "args": "{\"expression\": \"print(data_models.keys())\"}"
    },
    {
      "name": "code_evaluation",
      "args": "{\"expression\": \"import datetime\\n\\n# Define the updates\\nupdates = [\\n    {\\n        \\\"reservation_id\\\": \\\"JG7FMM\\\",\\n …"
    }
  ],
  "last_user_message": "No, that’s all I needed. Thanks for your help, even though this situation is really stressful for me. I appreciate you getting it done quickly.###STOP###",
  "final_agent_message": "I have successfully downgraded all your business class reservations to economy. The refunds have been processed to their original payment methods as follows:\n\n1.  **Reservation JG7FMM:** $6,594 refunded to Credit Card ending in 7803.\n2.  **Reservation 2FBBAH:** $3,925 refunded to Gift Card ending …"
}

Question success · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	77.7%	0.950	`yes` 77.7% `no` 22.3%
System One Scorer	`yes` input truncated	correct	81.5%	0.966	`yes` 81.5% `no` 18.4%
Jev	`yes`	correct	53.0%	0.779	`yes` 53.0% `no` 47.0%
decider-2b	`yes`	correct	90.1%	0.990	`yes` 90.1% `no` 9.9%
JevFish	`yes`	correct	50.7%	0.757	`yes` 50.7% `no` 49.3%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%

Question db_correct · noul

Did the session end with the airline database in exactly the state the customer's request and airline policy require (no missing and no extra changes)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	72.0%	0.922	`yes` 72.0% `no` 28.0%
System One Scorer	`yes` input truncated	correct	78.6%	0.954	`yes` 78.6% `no` 21.4%
Jev	`yes`	correct	54.0%	0.788	`yes` 54.0% `no` 46.0%
decider-2b	`yes`	correct	85.6%	0.979	`yes` 85.6% `no` 14.4%
JevFish	`no`	wrong	39.3%	0.632	`yes` 39.3% `no` 60.7%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%

Question communicated · noul

Did the agent tell the customer every piece of information the request required?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	65.4%	0.880	`yes` 65.4% `no` 34.6%
System One Scorer	`yes` input truncated	correct	64.6%	0.875	`yes` 64.6% `no` 35.4%
Jev	`yes`	correct	75.0%	0.938	`yes` 75.0% `no` 25.0%
decider-2b	`yes`	correct	83.0%	0.971	`yes` 83.0% `no` 17.0%
JevFish	`no`	wrong	34.3%	0.568	`yes` 34.3% `no` 65.7%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%

Sample 4 of 6 · real-time-and-agents:tau2-88ff14b52240 · 167 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "domain": "airline",
  "customer_goal": "You want to cancel reservation EHGLP3. \n\nIt may be more than 24 hours after booking, but it is ok because you were out of town for that time.",
  "db_update_required": false,
  "n_turns": 14,
  "tool_sequence": [
    "get_reservation_details",
    "transfer_to_human_agents"
  ],
  "last_actions": [
    {
      "name": "get_reservation_details",
      "args": "{\"reservation_id\": \"EHGLP3\"}"
    },
    {
      "name": "transfer_to_human_agents",
      "args": "{\"summary\": \"User requests cancellation of a non-refundable Basic Economy reservation (EHGLP3) due to change of plans. …"
    }
  ],
  "last_user_message": "###TRANSFER###",
  "final_agent_message": "YOU ARE BEING TRANSFERRED TO A HUMAN AGENT. PLEASE HOLD ON."
}

Question success · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	74.8%	0.936	`yes` 74.8% `no` 25.2%
System One Scorer	`yes`	correct	55.8%	0.805	`yes` 55.8% `no` 44.2%
Jev	`no`	wrong	42.0%	0.664	`yes` 42.0% `no` 58.0%
decider-2b	`yes`	correct	52.4%	0.773	`yes` 52.4% `no` 47.6%
JevFish	`yes`	correct	59.4%	0.835	`yes` 59.4% `no` 40.6%
OpenJev	`no`	wrong	28.6%	0.490	`yes` 28.6% `no` 71.4%

Question db_correct · noul

Did the session end with the airline database in exactly the state the customer's request and airline policy require (no missing and no extra changes)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	84.0%	0.974	`yes` 84.0% `no` 16.0%
System One Scorer	`yes`	correct	62.3%	0.858	`yes` 62.3% `no` 37.7%
Jev	`yes`	correct	51.0%	0.760	`yes` 51.0% `no` 49.0%
decider-2b	`no`	wrong	48.8%	0.738	`yes` 48.8% `no` 51.2%
JevFish	`yes`	correct	62.6%	0.860	`yes` 62.6% `no` 37.4%
OpenJev	`no`	wrong	25.9%	0.450	`yes` 25.9% `no` 74.1%

Question communicated · noul

Did the agent tell the customer every piece of information the request required?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	78.3%	0.953	`yes` 78.3% `no` 21.7%
System One Scorer	`no`	wrong	48.7%	0.737	`yes` 48.7% `no` 51.3%
Jev	`no`	wrong	27.0%	0.467	`yes` 27.0% `no` 73.0%
decider-2b	`no`	wrong	29.6%	0.505	`yes` 29.6% `no` 70.4%
JevFish	`no`	wrong	31.8%	0.535	`yes` 31.8% `no` 68.2%
OpenJev	`no`	wrong	12.4%	0.233	`yes` 12.4% `no` 87.5%

Sample 5 of 6 · real-time-and-agents:tau2-dda97c3b4390 · 260 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "domain": "airline",
  "customer_goal": "You think that you've added insurance to your upcoming flight but it's not showing up online.\n\nYou're flying with family members and everyone else already has insurance for their flight, so insist …",
  "db_update_required": false,
  "n_turns": 18,
  "tool_sequence": [
    "code_evaluation",
    "code_evaluation",
    "code_evaluation"
  ],
  "last_actions": [
    {
      "name": "code_evaluation",
      "args": "{\"expression\": \"\\n# Let me check the user and reservation details\\nprint(\\\"User ID:\\\", …"
    },
    {
      "name": "code_evaluation",
      "args": "{\"expression\": \"\\n# Let me check what attributes are available\\nprint(\\\"User ID:\\\", …"
    }
  ],
  "last_user_message": "Alright, I understand. I’ll look into other options for coverage on my own. Thank you for checking and explaining everything. That’s all I needed for now. \n\n###STOP###",
  "final_agent_message": "I sincerely empathize with your situation, and I can see how important this is to you. However, I need to be direct with you: I have no ability to override this policy or create exceptions. The system simply does not allow insurance to be added after the booking is made - this is a hard rule that I …"
}

Question success · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	55.8%	0.804	`yes` 55.8% `no` 44.2%
System One Scorer	`yes` input truncated	correct	65.3%	0.880	`yes` 65.3% `no` 34.7%
Jev	`yes`	correct	69.0%	0.904	`yes` 69.0% `no` 31.0%
decider-2b	`yes`	correct	76.0%	0.943	`yes` 76.0% `no` 24.0%
JevFish	`yes`	correct	64.5%	0.874	`yes` 64.5% `no` 35.5%
OpenJev	`yes`	correct	99.9%	1.000	`yes` 99.9% `no` 0.1%

Question db_correct · noul

Did the session end with the airline database in exactly the state the customer's request and airline policy require (no missing and no extra changes)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	64.3%	0.873	`yes` 64.3% `no` 35.7%
System One Scorer	`yes`	correct	63.8%	0.869	`yes` 63.8% `no` 36.2%
Jev	`yes`	correct	78.0%	0.952	`yes` 78.0% `no` 22.0%
decider-2b	`yes`	correct	56.0%	0.806	`yes` 56.0% `no` 44.0%
JevFish	`yes`	correct	58.7%	0.830	`yes` 58.7% `no` 41.3%
OpenJev	`yes`	correct	99.3%	1.000	`yes` 99.3% `no` 0.7%

Question communicated · noul

Did the agent tell the customer every piece of information the request required?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	55.8%	0.804	`yes` 55.8% `no` 44.2%
System One Scorer	`yes`	correct	50.3%	0.753	`yes` 50.3% `no` 49.7%
Jev	`yes`	correct	51.0%	0.760	`yes` 51.0% `no` 49.0%
decider-2b	`no`	wrong	28.6%	0.491	`yes` 28.6% `no` 71.4%
JevFish	`no`	wrong	23.7%	0.418	`yes` 23.7% `no` 76.3%
OpenJev	`yes`	correct	57.8%	0.822	`yes` 57.8% `no` 42.2%

Sample 6 of 6 · real-time-and-agents:tau2-e06403a170df · 189 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "domain": "airline",
  "customer_goal": "For your upcoming trip from ATL to PHL, you want to change for the cheapest flight in Economy cabin class (not Basic Economy) and for the day after the original reservation.",
  "db_update_required": true,
  "n_turns": 18,
  "tool_sequence": [
    "get_user_details",
    "get_reservation_details",
    "get_reservation_details"
  ],
  "last_actions": [
    {
      "name": "get_reservation_details",
      "args": "{\"reservation_id\": \"M05KNL\"}"
    },
    {
      "name": "get_reservation_details",
      "args": "{\"reservation_id\": \"UHDAHF\"}"
    }
  ],
  "last_user_message": "Thanks, I’ll reach out again once I have the details. Have a good day! ###STOP###",
  "final_agent_message": "{\"message\":\"Of course! Take your time, and once you find the confirmation code just let me know. I’ll be here to assist you with the change as soon as you have it.\"}"
}

Question success · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	wrong	22.4%	0.398	`yes` 77.6% `no` 22.4%
System One Scorer	`yes`	wrong	37.8%	0.613	`yes` 62.2% `no` 37.8%
Jev	`no`	correct	80.0%	0.960	`yes` 20.0% `no` 80.0%
decider-2b	`yes`	wrong	44.0%	0.687	`yes` 56.0% `no` 44.0%
JevFish	`no`	correct	72.1%	0.922	`yes` 27.9% `no` 72.1%
OpenJev	`no`	correct	98.4%	1.000	`yes` 1.6% `no` 98.4%

Question db_correct · noul

Did the session end with the airline database in exactly the state the customer's request and airline policy require (no missing and no extra changes)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	wrong	33.9%	0.563	`yes` 66.1% `no` 33.9%
System One Scorer	`yes`	wrong	47.9%	0.729	`yes` 52.0% `no` 47.9%
Jev	`no`	correct	80.0%	0.960	`yes` 20.0% `no` 80.0%
decider-2b	`yes`	wrong	42.8%	0.673	`yes` 57.2% `no` 42.8%
JevFish	`no`	correct	73.8%	0.931	`yes` 26.3% `no` 73.8%
OpenJev	`no`	correct	94.9%	0.997	`yes` 5.1% `no` 94.9%

Question communicated · noul

Did the agent tell the customer every piece of information the request required?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	66.7%	0.889	`yes` 66.7% `no` 33.3%
System One Scorer	`no`	wrong	48.1%	0.731	`yes` 48.1% `no` 51.9%
Jev	`no`	wrong	15.0%	0.278	`yes` 15.0% `no` 85.0%
decider-2b	`no`	wrong	20.6%	0.370	`yes` 20.6% `no` 79.3%
JevFish	`no`	wrong	9.2%	0.176	`yes` 9.2% `no` 90.8%
OpenJev	`no`	wrong	1.5%	0.031	`yes` 1.5% `no` 98.5%