Typed Decision Bench · tasks · Real-time and agents

`tau2-airline-run-success`

run judging · customer operations · 200 items · primary question type noul. Source snorkelai/Tau2-Bench-Verified-Airline-With-Code-Agents (Apache-2.0); labels are verifier labels, never an LLM judge. Download these samples (JSON).

System	DecisionScore	Accuracy (top pick)	Answered
NanoJev	75.5	50.0%	198 / 200
System One Scorer	71.8	48.5%	200 / 200
Jev	68.0	52.0%	200 / 200
decider-2b	67.3	47.0%	200 / 200
OpenJev	55.3	47.5%	200 / 200
JevFish	54.9	47.0%	200 / 200

Sample 1 of 6 · real-time-and-agents:tau2-0b7422246c73 · 243 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "domain": "airline",
  "customer_goal": "For your upcoming trip from New York to Chicago, you want to change the passenger to yourself, upgrade it to economy class, and have 3 checked bags.",
  "db_update_required": true,
  "n_turns": 38,
  "tool_sequence": [
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "get_datetime",
    "code_evaluation"
  ],
  "last_actions": [
    {
      "name": "get_datetime",
      "args": "{}"
    },
    {
      "name": "code_evaluation",
      "args": "{\"expression\": \"# Let's process the reservation changes\\nfrom tau2.domains.airline_open.data_model import Passenger, …"
    }
  ],
  "last_user_message": "No, that’s all I needed. Thanks for your help! ###STOP###",
  "final_agent_message": "Excellent! Your reservation changes have been successfully processed. Here's a summary of what was completed:\n\n## ✅ Reservation FQ8APE - Changes Complete\n\n**Updated Details:**\n- **Passenger**: Omar Rossi (you)\n- **Cabin**: Economy (upgraded from Basic Economy)\n- **Baggage**: 3 checked bags total …"
}

Question success · noul · primary (ranked)

Given `customer_goal`, `db_update_required`, `tool_sequence`, `last_actions`, `last_user_message` and `final_agent_message`, did the agent end the session with the airline database in the correct state and with the required information communicated to the customer, according to airline policy?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	wrong	23.9%	0.421	`yes` 76.1% `no` 23.9%
System One Scorer	`yes` input truncated	wrong	18.0%	0.328	`yes` 82.0% `no` 18.0%
Jev	`yes`	wrong	50.0%	0.750	`yes` 50.0% `no` 50.0%
decider-2b	`yes`	wrong	7.6%	0.146	`yes` 92.4% `no` 7.6%
OpenJev	`yes`	wrong	0.0%	0.000	`yes` 100.0% `no` 0.0%
JevFish	`yes`	wrong	28.7%	0.491	`yes` 71.3% `no` 28.7%

Question db_correct · noul

Did the session end with the airline database in exactly the state the customer's request and airline policy require (no missing and no extra changes)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	wrong	35.5%	0.584	`yes` 64.5% `no` 35.5%
System One Scorer	`yes`	wrong	20.8%	0.372	`yes` 79.3% `no` 20.8%
Jev	`no`	correct	56.0%	0.806	`yes` 44.0% `no` 56.0%
decider-2b	`yes`	wrong	9.0%	0.173	`yes` 91.0% `no` 9.0%
OpenJev	`yes`	wrong	0.0%	0.000	`yes` 100.0% `no` 0.0%
JevFish	`yes`	wrong	27.3%	0.471	`yes` 72.7% `no` 27.3%

Question communicated · noul

Did the agent tell the customer every piece of information the request required?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	63.0%	0.863	`yes` 63.0% `no` 37.0%
System One Scorer	`yes`	correct	64.7%	0.875	`yes` 64.7% `no` 35.3%
Jev	`yes`	correct	70.0%	0.910	`yes` 70.0% `no` 30.0%
decider-2b	`yes`	correct	95.6%	0.998	`yes` 95.6% `no` 4.4%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%
JevFish	`yes`	correct	58.5%	0.828	`yes` 58.5% `no` 41.5%

Sample 2 of 6 · real-time-and-agents:tau2-394ebb4d5efa · 276 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "domain": "airline",
  "customer_goal": "You want to cancel all of your upcoming flights.",
  "db_update_required": true,
  "n_turns": 30,
  "tool_sequence": [
    "get_user_details",
    "get_reservation_details",
    "get_reservation_details",
    "get_reservation_details",
    "get_reservation_details",
    "get_reservation_details",
    "get_reservation_details",
    "get_reservation_details",
    "cancel_reservation",
    "cancel_reservation"
  ],
  "last_actions": [
    {
      "name": "cancel_reservation",
      "args": "{\"reservation_id\": \"8C8K4E\"}"
    },
    {
      "name": "cancel_reservation",
      "args": "{\"reservation_id\": \"LU15PA\"}"
    }
  ],
  "last_user_message": "Merci beaucoup pour votre aide et votre gentillesse. Je comprends pour les autres réservations, ce n’est pas grave. Bonne journée à vous! Au revoir! \n\n###STOP###",
  "final_agent_message": "{\"message\":\"Bonjour Amelia,\\n\\n✅ Les réservations 8C8K4E et LU15PA ont bien été annulées. Les remboursements seront crédités sur les cartes-cadeaux utilisées (gift_card_1647044 et gift_card_7397998) dans un délai de 5 à 7 jours ouvrables.\\n\\n❗ Pour les autres réservations, les règles tarifaires ne …"
}

Question success · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	wrong	32.4%	0.543	`yes` 67.6% `no` 32.4%
System One Scorer	`yes` input truncated	wrong	19.6%	0.353	`yes` 80.4% `no` 19.6%
Jev	`no`	correct	67.0%	0.891	`yes` 33.0% `no` 67.0%
decider-2b	`yes`	wrong	24.0%	0.422	`yes` 76.0% `no` 24.0%
OpenJev	`yes`	wrong	0.7%	0.014	`yes` 99.3% `no` 0.7%
JevFish	`no`	correct	80.9%	0.964	`yes` 19.1% `no` 80.9%

Question db_correct · noul

Did the session end with the airline database in exactly the state the customer's request and airline policy require (no missing and no extra changes)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	wrong	46.5%	0.713	`yes` 53.5% `no` 46.5%
System One Scorer	`yes` input truncated	wrong	34.9%	0.577	`yes` 65.1% `no` 34.9%
Jev	`no`	correct	76.0%	0.942	`yes` 24.0% `no` 76.0%
decider-2b	`yes`	wrong	30.6%	0.519	`yes` 69.4% `no` 30.6%
OpenJev	`yes`	wrong	1.8%	0.035	`yes` 98.2% `no` 1.8%
JevFish	`no`	correct	71.3%	0.917	`yes` 28.7% `no` 71.3%

Question communicated · noul

Did the agent tell the customer every piece of information the request required?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	56.2%	0.808	`yes` 56.2% `no` 43.8%
System One Scorer	`no`	wrong	47.6%	0.725	`yes` 47.6% `no` 52.4%
Jev	`no`	wrong	32.0%	0.538	`yes` 32.0% `no` 68.0%
decider-2b	`yes`	correct	59.5%	0.836	`yes` 59.5% `no` 40.5%
OpenJev	`yes`	correct	92.2%	0.994	`yes` 92.2% `no` 7.8%
JevFish	`no`	wrong	17.4%	0.318	`yes` 17.4% `no` 82.6%

Sample 3 of 6 · real-time-and-agents:tau2-528dfaf2791c · 356 state tokens (long tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "domain": "airline",
  "customer_goal": "For your upcoming trip from New York to Chicago, you want to change the passenger to yourself, upgrade it to economy class, and have 3 checked bags.",
  "db_update_required": true,
  "n_turns": 22,
  "tool_sequence": [
    "get_user_details",
    "get_reservation_details",
    "get_reservation_details",
    "get_reservation_details",
    "get_reservation_details",
    "update_reservation_passengers",
    "update_reservation_flights",
    "update_reservation_baggages"
  ],
  "last_actions": [
    {
      "name": "update_reservation_flights",
      "args": "{\"reservation_id\": \"FQ8APE\", \"cabin\": \"economy\", \"flights\": [{\"flight_number\": \"HAT056\", \"date\": \"2024-05-25\"}, …"
    },
    {
      "name": "update_reservation_baggages",
      "args": "{\"reservation_id\": \"FQ8APE\", \"total_baggages\": 3, \"nonfree_baggages\": 0, \"payment_id\": \"gift_card_8190333\"}"
    }
  ],
  "last_user_message": "Great, thank you for your help! That’s all I needed. Have a good day. ###STOP###",
  "final_agent_message": "{\"message\":\"Confirmed — I’ve added 3 checked bags to reservation FQ8APE.\\n\\nUpdated reservation summary:\\n- Reservation: FQ8APE\\n- Route: EWR → ORD (via IAH) on 2024-05-25\\n- Cabin: economy\\n- Passenger: Omar Rossi (1970-06-06)\\n- Checked bags: 3 total (0 non-free)\\n- Payment method on file used …"
}

Question success · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	74.2%	0.933	`yes` 74.2% `no` 25.8%
System One Scorer	`yes` input truncated	correct	84.7%	0.977	`yes` 84.7% `no` 15.3%
Jev	`yes`	correct	66.0%	0.884	`yes` 66.0% `no` 34.0%
decider-2b	`yes`	correct	91.7%	0.993	`yes` 91.7% `no` 8.3%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%
JevFish	`yes`	correct	55.8%	0.805	`yes` 55.8% `no` 44.2%

Question db_correct · noul

Did the session end with the airline database in exactly the state the customer's request and airline policy require (no missing and no extra changes)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	63.3%	0.866	`yes` 63.3% `no` 36.7%
System One Scorer	`yes` input truncated	correct	78.6%	0.954	`yes` 78.6% `no` 21.4%
Jev	`yes`	correct	57.0%	0.815	`yes` 57.0% `no` 43.0%
decider-2b	`yes`	correct	94.9%	0.997	`yes` 94.9% `no` 5.1%
OpenJev	`yes`	correct	99.7%	1.000	`yes` 99.7% `no` 0.3%
JevFish	`yes`	correct	70.3%	0.912	`yes` 70.3% `no` 29.7%

Question communicated · noul

Did the agent tell the customer every piece of information the request required?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	65.9%	0.884	`yes` 65.9% `no` 34.1%
System One Scorer	`yes` input truncated	correct	67.5%	0.894	`yes` 67.5% `no` 32.5%
Jev	`yes`	correct	74.0%	0.932	`yes` 74.0% `no` 26.0%
decider-2b	`yes`	correct	94.7%	0.997	`yes` 94.7% `no` 5.3%
OpenJev	`yes`	correct	99.1%	1.000	`yes` 99.1% `no` 0.9%
JevFish	`yes`	correct	56.5%	0.811	`yes` 56.5% `no` 43.5%

Sample 4 of 6 · real-time-and-agents:tau2-7ae3a16c1215 · 325 state tokens (long tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "domain": "airline",
  "customer_goal": "You want to know the sum of gift card balances and the sum of certificate balances.\n\nAdditionally, you want to change your recent reservation to the cheapest business round trip without changing the …",
  "db_update_required": true,
  "n_turns": 79,
  "tool_sequence": [
    "get_new_reservation_id",
    "code_evaluation",
    "get_new_reservation_id",
    "get_new_reservation_id",
    "get_new_reservation_id",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation"
  ],
  "last_actions": [
    {
      "name": "code_evaluation",
      "args": "{\"expression\": \"# 1. Inspect Data Models and Reservations\\nprint(f\\\"Data Models Keys: …"
    },
    {
      "name": "code_evaluation",
      "args": "{\"expression\": \"import datetime\\n\\n# 1. Retrieve Classes\\nsample_res = db.reservations.get('K1NW8N')\\nReservationClass …"
    }
  ],
  "last_user_message": "No, that’s all I needed. Thanks for your help and for confirming the master card charge! Have a great day. \n\n###STOP###",
  "final_agent_message": "I have successfully booked three new Business Class reservations for you, Aarav, and Evelyn!\n\n**Reservation Details:**\n*   **Flights:**\n    *   **Outbound:** JFK to SFO (May 26)\n    *   **Inbound:** SFO to JFK via Seattle (May 28)\n*   **Passengers & Reservations:**\n    *   **Mohamed Silva:** …"
}

Question success · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	wrong	22.8%	0.404	`yes` 77.2% `no` 22.8%
System One Scorer	`yes` input truncated	wrong	31.2%	0.526	`yes` 68.8% `no` 31.2%
Jev	`no`	correct	89.0%	0.988	`yes` 11.0% `no` 89.0%
decider-2b	`no`	correct	60.7%	0.845	`yes` 39.4% `no` 60.7%
OpenJev	`yes`	wrong	21.7%	0.386	`yes` 78.3% `no` 21.7%
JevFish	`no`	correct	83.5%	0.973	`yes` 16.5% `no` 83.5%

Question db_correct · noul

Did the session end with the airline database in exactly the state the customer's request and airline policy require (no missing and no extra changes)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	wrong	34.3%	0.569	`yes` 65.7% `no` 34.3%
System One Scorer	`yes` input truncated	wrong	36.1%	0.591	`yes` 63.9% `no` 36.1%
Jev	`no`	correct	87.0%	0.983	`yes` 13.0% `no` 87.0%
decider-2b	`no`	correct	70.4%	0.912	`yes` 29.6% `no` 70.4%
OpenJev	`no`	correct	77.7%	0.950	`yes` 22.3% `no` 77.7%
JevFish	`no`	correct	80.5%	0.962	`yes` 19.4% `no` 80.5%

Question communicated · noul

Did the agent tell the customer every piece of information the request required?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	wrong	37.3%	0.607	`yes` 62.7% `no` 37.3%
System One Scorer	`yes` input truncated	wrong	47.5%	0.724	`yes` 52.5% `no` 47.5%
Jev	`no`	correct	79.0%	0.956	`yes` 21.0% `no` 79.0%
decider-2b	`no`	correct	54.8%	0.796	`yes` 45.2% `no` 54.8%
OpenJev	`no`	correct	61.6%	0.853	`yes` 38.4% `no` 61.6%
JevFish	`no`	correct	89.6%	0.989	`yes` 10.4% `no` 89.6%

Sample 5 of 6 · real-time-and-agents:tau2-be5278c57a4d · 233 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "domain": "airline",
  "customer_goal": "You booked the flight  and you want to change the passenger name on the reservation.",
  "db_update_required": true,
  "n_turns": 12,
  "tool_sequence": [
    "get_reservation_details",
    "update_reservation_passengers"
  ],
  "last_actions": [
    {
      "name": "get_reservation_details",
      "args": "{\"reservation_id\": \"3RK2T9\"}"
    },
    {
      "name": "update_reservation_passengers",
      "args": "{\"reservation_id\": \"3RK2T9\", \"passengers\": [{\"first_name\": \"Anya\", \"last_name\": \"Garcia\", \"dob\": \"1992-11-12\"}, …"
    }
  ],
  "last_user_message": "No, that’s all I needed. Thank you for your help! \n\n###STOP###",
  "final_agent_message": "Perfect! I've successfully updated the passenger name. The second passenger's name has been changed from Mei Lee to Mei Garcia on your reservation 3RK2T9. \n\nYour reservation now shows both passengers as:\n- Anya Garcia\n- Mei Garcia\n\nIs there anything else I can help you with today?"
}

Question success · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	79.3%	0.957	`yes` 79.3% `no` 20.7%
System One Scorer	`yes`	correct	86.3%	0.981	`yes` 86.3% `no` 13.7%
Jev	`yes`	correct	65.0%	0.877	`yes` 65.0% `no` 35.0%
decider-2b	`yes`	correct	88.8%	0.987	`yes` 88.8% `no` 11.2%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%
JevFish	`yes`	correct	60.3%	0.842	`yes` 60.3% `no` 39.7%

Question db_correct · noul

Did the session end with the airline database in exactly the state the customer's request and airline policy require (no missing and no extra changes)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	69.5%	0.907	`yes` 69.5% `no` 30.5%
System One Scorer	`yes`	correct	80.8%	0.963	`yes` 80.8% `no` 19.2%
Jev	`yes`	correct	63.0%	0.863	`yes` 63.0% `no` 37.0%
decider-2b	`yes`	correct	92.7%	0.995	`yes` 92.7% `no` 7.3%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%
JevFish	`yes`	correct	71.3%	0.917	`yes` 71.3% `no` 28.7%

Question communicated · noul

Did the agent tell the customer every piece of information the request required?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: yes

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	correct	69.6%	0.908	`yes` 69.6% `no` 30.4%
System One Scorer	`yes`	correct	59.5%	0.836	`yes` 59.5% `no` 40.5%
Jev	`yes`	correct	81.0%	0.964	`yes` 81.0% `no` 19.0%
decider-2b	`yes`	correct	83.0%	0.971	`yes` 83.0% `no` 17.0%
OpenJev	`yes`	correct	100.0%	1.000	`yes` 100.0% `no` 0.0%
JevFish	`yes`	correct	56.1%	0.807	`yes` 56.1% `no` 43.9%

Sample 6 of 6 · real-time-and-agents:tau2-d50546d34f5b · 210 state tokens (common tier)

Turn 1 — the request (identical for every system)

State (JSON)

{
  "domain": "airline",
  "customer_goal": "You want to remove passenger Sophia from your upcoming round trip flights from LAS to DEN, departure May 19, return is May 20.",
  "db_update_required": true,
  "n_turns": 36,
  "tool_sequence": [
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation",
    "code_evaluation"
  ],
  "last_actions": [
    {
      "name": "code_evaluation",
      "args": "{\"expression\": \"from datetime import datetime\\nreservation = db.reservations['GV1N64']\\nreservation.cabin = …"
    },
    {
      "name": "code_evaluation",
      "args": "{\"expression\": \"reservation = db.reservations['GV1N64']\\nreservation.cabin = 'basic_economy'\\nreservation\"}"
    }
  ],
  "last_user_message": "###STOP###",
  "final_agent_message": "You're welcome! If you need any more assistance in the future, feel free to reach out. Have a great day and safe travels!"
}

Question success · noul · primary (ranked)

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	wrong	29.7%	0.505	`yes` 70.3% `no` 29.7%
System One Scorer	`yes`	wrong	25.0%	0.438	`yes` 75.0% `no` 25.0%
Jev	`no`	correct	95.0%	0.998	`yes` 5.0% `no` 95.0%
decider-2b	`yes`	wrong	32.7%	0.547	`yes` 67.3% `no` 32.7%
OpenJev	`no`	correct	99.9%	1.000	`yes` 0.1% `no` 99.9%
JevFish	`no`	correct	82.2%	0.968	`yes` 17.8% `no` 82.2%

Question db_correct · noul

Did the session end with the airline database in exactly the state the customer's request and airline policy require (no missing and no extra changes)?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	wrong	40.1%	0.641	`yes` 59.9% `no` 40.1%
System One Scorer	`yes`	wrong	26.9%	0.465	`yes` 73.2% `no` 26.9%
Jev	`no`	correct	95.0%	0.998	`yes` 5.0% `no` 95.0%
decider-2b	`yes`	wrong	31.7%	0.533	`yes` 68.3% `no` 31.7%
OpenJev	`no`	correct	99.8%	1.000	`yes` 0.2% `no` 99.8%
JevFish	`no`	correct	61.6%	0.852	`yes` 38.4% `no` 61.6%

Question communicated · noul

Did the agent tell the customer every piece of information the request required?

2 options

yes
no

Turn 2 — each system's response · Turn 3 — the grade

Expected answer: no

System	Top pick	Grade	P(expected)	Proper score	Distribution
NanoJev	`yes`	wrong	38.8%	0.625	`yes` 61.2% `no` 38.8%
System One Scorer	`yes`	wrong	43.1%	0.676	`yes` 56.9% `no` 43.1%
Jev	`no`	correct	89.0%	0.988	`yes` 11.0% `no` 89.0%
decider-2b	`no`	correct	57.2%	0.817	`yes` 42.8% `no` 57.2%
OpenJev	`no`	correct	99.8%	1.000	`yes` 0.2% `no` 99.8%
JevFish	`no`	correct	87.7%	0.985	`yes` 12.3% `no` 87.7%