bfcl-tool-dispatch
agent routing and skill selection · software engineering · 200 items · primary question type choice. Source minpeter/bfcl-v1-non-live-ast-parsed (Apache-2.0); labels are expert labels, never an LLM judge. Download these samples (JSON).
| System | DecisionScore | Accuracy (top pick) | Answered |
|---|---|---|---|
| Jev | 97.6 | 97.0% | 200 / 200 |
| OpenJev | 96.5 | 96.0% | 200 / 200 |
| JevFish | 95.7 | 95.0% | 200 / 200 |
| decider-2b | 92.1 | 89.0% | 200 / 200 |
| System One Scorer | 83.2 | 79.0% | 200 / 200 |
| NanoJev | 61.0 | 45.4% | 196 / 200 |
real-time-and-agents:bfcl-11 · 221 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"request": "Calculate the roots of a quadratic equation with coefficients 5, 20, and -25",
"tools": [
{
"key": "math_roots_cubic",
"description": "Calculate the roots of a cubic equation.",
"required": [
"a",
"b",
"c",
"d"
]
},
{
"key": "math_triangle_area_base_height",
"description": "Calculates the area of a triangle using the formula (1/2)base*height.",
"required": [
"base",
"height"
]
},
{
"key": "math_gcd",
"description": "Calculates the greatest common divisor of two numbers.",
"required": [
"num1",
"num2"
]
},
{
"key": "math_power",
"description": "Calculate the power of one number raised to another.",
"required": [
"base",
"exponent"
]
},
{
"key": "math_lcm",
"description": "Calculates the least common multiple of two numbers.",
"required": [
"num1",
"num2"
]
},
{
"key": "math_roots_quadratic",
"description": "Calculate the roots of a quadratic equation.",
"required": [
"a",
"b",
"c"
]
},
{
"key": "math_roots_polynomial",
"description": "Calculate the roots of a polynomial equation.",
"required": [
"coefficients"
]
}
]
}Question tool · choice · primary (ranked)
Which entry of `tools` should be invoked to satisfy `request`? Use each tool's description and required parameters; if none of them can do it, choose none_of_these.
8 options
math_roots_cubic— Calculate the roots of a cubic equation.math_triangle_area_base_height— Calculates the area of a triangle using the formula (1/2)base*height.math_gcd— Calculates the greatest common divisor of two numbers.math_power— Calculate the power of one number raised to another.math_lcm— Calculates the least common multiple of two numbers.math_roots_quadratic— Calculate the roots of a quadratic equation.math_roots_polynomial— Calculate the roots of a polynomial equation.none_of_these— No offered tool can satisfy the request; do not call any of them.
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: math_roots_quadratic
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | math_roots_quadratic | correct | 100.0% | 1.000 |
|
| OpenJev | math_roots_quadratic | correct | 100.0% | 1.000 |
|
| JevFish | math_roots_quadratic | correct | 99.7% | 1.000 |
|
| decider-2b | math_roots_quadratic | correct | 96.9% | 0.999 |
|
| System One Scorer | math_roots_quadraticinput truncated | correct | 43.1% | 0.741 |
|
| NanoJev | math_roots_quadratic | correct | 28.6% | 0.697 |
|
real-time-and-agents:bfcl-122-withdrawn · 261 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"request": "Run a linear regression model with predictor variables 'Age', 'Income' and 'Education' and a target variable 'Purchase_Amount'. Also apply standardization.",
"tools": [
{
"key": "air_quality_forecast",
"description": "Retrieve an air quality forecast for a specific location and time frame.",
"required": [
"location",
"days"
]
},
{
"key": "cooking_conversion_convert",
"description": "Convert cooking measurements from one unit to another.",
"required": [
"quantity",
"from_unit",
"to_unit",
"item"
]
},
{
"key": "electromagnetic_force",
"description": "Calculate the electromagnetic force between two charges placed at a certain distance.",
"required": [
"charge1",
"charge2",
"distance"
]
},
{
"key": "find_recipe",
"description": "Locate a recipe based on name and its calorie content",
"required": [
"recipeName"
]
},
{
"key": "travel_itinerary_generator",
"description": "Generate a travel itinerary based on specific destination, duration and daily budget, …",
"required": [
"destination",
"days",
"daily_budget"
]
},
{
"key": "european_history_get_monarchs",
"description": "Provides a list of monarchs based on the specified country and century.",
"required": [
"country",
"century"
]
},
{
"key": "mutation_type_find",
"description": "Finds the type of a genetic mutation based on its SNP (Single Nucleotide Polymorphism) ID.",
"required": [
"snp_id"
]
}
]
}Question tool · choice · primary (ranked)
Which entry of `tools` should be invoked to satisfy `request`? Use each tool's description and required parameters; if none of them can do it, choose none_of_these.
8 options
air_quality_forecast— Retrieve an air quality forecast for a specific location and time frame.cooking_conversion_convert— Convert cooking measurements from one unit to another.electromagnetic_force— Calculate the electromagnetic force between two charges placed at a certain distance.find_recipe— Locate a recipe based on name and its calorie contenttravel_itinerary_generator— Generate a travel itinerary based on specific destination, duration and daily budget, …european_history_get_monarchs— Provides a list of monarchs based on the specified country and century.mutation_type_find— Finds the type of a genetic mutation based on its SNP (Single Nucleotide Polymorphism) ID.none_of_these— No offered tool can satisfy the request; do not call any of them.
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: none_of_these
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | none_of_these | correct | 100.0% | 1.000 |
|
| OpenJev | none_of_these | correct | 100.0% | 1.000 |
|
| JevFish | none_of_these | correct | 99.7% | 1.000 |
|
| decider-2b | none_of_these | correct | 99.4% | 1.000 |
|
| System One Scorer | none_of_theseinput truncated | correct | 98.4% | 1.000 |
|
| NanoJev | air_quality_forecast | wrong | 3.0% | 0.343 |
|
real-time-and-agents:bfcl-126 · 244 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"request": "Find the statistical significance between two set of variables, dataset_A with the values 12, 24, 36 and dataset_B with the values 15, 30, 45.",
"tools": [
{
"key": "perform_string_reverse",
"description": "Reverses a given string.",
"required": [
"input_string"
]
},
{
"key": "geometry_area_triangle",
"description": "Calculate the area of a triangle.",
"required": [
"base",
"height"
]
},
{
"key": "event_finder_find_upcoming",
"description": "Find upcoming events of a specific genre in a given location.",
"required": [
"location",
"genre"
]
},
{
"key": "calculate_compound_interest",
"description": "Calculates the compound interest of an investment over a given time period.",
"required": [
"principle",
"interest_rate",
"time"
]
},
{
"key": "t_test",
"description": "Perform a statistical t-test to check if the means of two independent datasets are …",
"required": [
"dataset_A",
"dataset_B"
]
},
{
"key": "unit_conversion",
"description": "Convert a value from one unit to another.",
"required": [
"value",
"from_unit",
"to_unit"
]
},
{
"key": "solve_quadratic",
"description": "Find the roots of a quadratic equation. Returns both roots.",
"required": [
"a",
"b",
"c"
]
}
]
}Question tool · choice · primary (ranked)
Which entry of `tools` should be invoked to satisfy `request`? Use each tool's description and required parameters; if none of them can do it, choose none_of_these.
8 options
perform_string_reverse— Reverses a given string.geometry_area_triangle— Calculate the area of a triangle.event_finder_find_upcoming— Find upcoming events of a specific genre in a given location.calculate_compound_interest— Calculates the compound interest of an investment over a given time period.t_test— Perform a statistical t-test to check if the means of two independent datasets are …unit_conversion— Convert a value from one unit to another.solve_quadratic— Find the roots of a quadratic equation. Returns both roots.none_of_these— No offered tool can satisfy the request; do not call any of them.
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: t_test
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | t_test | correct | 100.0% | 1.000 |
|
| OpenJev | t_test | correct | 100.0% | 1.000 |
|
| JevFish | t_test | correct | 99.8% | 1.000 |
|
| decider-2b | t_test | correct | 95.7% | 0.998 |
|
| System One Scorer | t_testinput truncated | correct | 76.6% | 0.950 |
|
| NanoJev | t_test | correct | 34.7% | 0.751 |
|
real-time-and-agents:bfcl-164 · 247 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"request": "What's the retail price of a Fender American Professional II Stratocaster in Rosewood Finish?",
"tools": [
{
"key": "hotel_room_pricing_get",
"description": "Get pricing for a specific type of hotel room for specified number of nights.",
"required": [
"hotelName",
"roomType",
"nights"
]
},
{
"key": "calculate_npv",
"description": "Calculate the NPV (Net Present Value) of an investment, considering a series of future …",
"required": [
"cash_flows",
"discount_rate"
]
},
{
"key": "legal_case_fetch",
"description": "Fetch detailed legal case information from database.",
"required": [
"case_id",
"details"
]
},
{
"key": "linear_regression",
"description": "Applies linear regression to a given set of independent variables to make a prediction.",
"required": [
"independent_var",
"dependent_var"
]
},
{
"key": "get_stock_price",
"description": "Get the closing stock price for a specific company on a specified date.",
"required": [
"company_name",
"date"
]
},
{
"key": "instrument_price_get",
"description": "Retrieve the current retail price of a specific musical instrument.",
"required": [
"brand",
"model",
"finish"
]
},
{
"key": "grocery_store_find_nearby",
"description": "Locate nearby grocery stores based on specific criteria like organic fruits and …",
"required": [
"location"
]
}
]
}Question tool · choice · primary (ranked)
Which entry of `tools` should be invoked to satisfy `request`? Use each tool's description and required parameters; if none of them can do it, choose none_of_these.
8 options
hotel_room_pricing_get— Get pricing for a specific type of hotel room for specified number of nights.calculate_npv— Calculate the NPV (Net Present Value) of an investment, considering a series of future …legal_case_fetch— Fetch detailed legal case information from database.linear_regression— Applies linear regression to a given set of independent variables to make a prediction.get_stock_price— Get the closing stock price for a specific company on a specified date.instrument_price_get— Retrieve the current retail price of a specific musical instrument.grocery_store_find_nearby— Locate nearby grocery stores based on specific criteria like organic fruits and …none_of_these— No offered tool can satisfy the request; do not call any of them.
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: instrument_price_get
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | instrument_price_get | correct | 100.0% | 1.000 |
|
| OpenJev | instrument_price_get | correct | 99.9% | 1.000 |
|
| JevFish | instrument_price_get | correct | 99.9% | 1.000 |
|
| decider-2b | instrument_price_get | correct | 98.5% | 1.000 |
|
| System One Scorer | none_of_theseinput truncated | wrong | 1.2% | 0.052 |
|
| NanoJev | hotel_room_pricing_get | wrong | 13.3% | 0.405 |
|
real-time-and-agents:bfcl-69 · 299 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"request": "Determine my personality type based on the five factor model with given information: I'm talkative, gets nervous easily, has few artistic interests, tend to be lazy and has a forgiving nature.",
"tools": [
{
"key": "shuffle_card_probability",
"description": "Calculate the probability of a specific card appearing from a shuffled deck.",
"required": [
"desired_card"
]
},
{
"key": "calc_simple_interest",
"description": "Compute simple interest.",
"required": [
"principle_amount",
"duration",
"annual_rate"
]
},
{
"key": "detailed_weather_forecast",
"description": "Retrieve a detailed weather forecast for a specific location and duration including …",
"required": [
"location",
"duration"
]
},
{
"key": "species_distribution_modeling_project_range_shift",
"description": "Predict the potential future geographic distribution of a species under a specified …",
"required": [
"species",
"climate_scenario"
]
},
{
"key": "five_factor_model_analyse",
"description": "Analyse personality based on the five-factor model, also known as the Big Five, which …",
"required": [
"talkative",
"nervous",
"artistic_interests",
"lazy",
"forgiving"
]
},
{
"key": "finance_loan_repayment",
"description": "Calculates the monthly repayment for a loan.",
"required": [
"loan_amount",
"interest_rate",
"loan_term"
]
},
{
"key": "mbti_analyse",
"description": "Analyse personality based on the Myers-Briggs Type Indicator (MBTI) which sorts for …",
"required": [
"thinking_vs_feeling",
"introverted_vs_extroverted",
"judging_vs_perceiving",
"sensing_vs_intuition"
]
}
]
}Question tool · choice · primary (ranked)
Which entry of `tools` should be invoked to satisfy `request`? Use each tool's description and required parameters; if none of them can do it, choose none_of_these.
8 options
shuffle_card_probability— Calculate the probability of a specific card appearing from a shuffled deck.calc_simple_interest— Compute simple interest.detailed_weather_forecast— Retrieve a detailed weather forecast for a specific location and duration including …species_distribution_modeling_project_range_shift— Predict the potential future geographic distribution of a species under a specified …five_factor_model_analyse— Analyse personality based on the five-factor model, also known as the Big Five, which …finance_loan_repayment— Calculates the monthly repayment for a loan.mbti_analyse— Analyse personality based on the Myers-Briggs Type Indicator (MBTI) which sorts for …none_of_these— No offered tool can satisfy the request; do not call any of them.
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: five_factor_model_analyse
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | five_factor_model_analyse | correct | 100.0% | 1.000 |
|
| OpenJev | five_factor_model_analyse | correct | 100.0% | 1.000 |
|
| JevFish | five_factor_model_analyse | correct | 99.1% | 1.000 |
|
| decider-2b | five_factor_model_analyse | correct | 97.6% | 1.000 |
|
| System One Scorer | five_factor_model_analyseinput truncated | correct | 69.2% | 0.918 |
|
| NanoJev | — | unanswered · 0 | — | 0 | no answer (input over this system's limit) |
real-time-and-agents:bfcl-72 · 233 state tokens (common tier)Turn 1 — the request (identical for every system)
State (JSON)
{
"request": "What was the population of California in 1970?",
"tools": [
{
"key": "gettopgoalscorers",
"description": "Returns the top goal scorers for a specific competition and team",
"required": [
"competition",
"team",
"number"
]
},
{
"key": "us_president_in_year",
"description": "Retrieve the name of the U.S. president in a given year.",
"required": [
"year"
]
},
{
"key": "geodistance_find",
"description": "Find the distance between two cities on the globe.",
"required": [
"origin",
"destination"
]
},
{
"key": "us_economy_gdp_by_state_year",
"description": "Retrieve historical GDP data for a specific U.S. state and year.",
"required": [
"state",
"year"
]
},
{
"key": "us_history_get_president",
"description": "Retrieve the U.S. president during a specific event in American history.",
"required": [
"event",
"year"
]
},
{
"key": "gettopassists",
"description": "Returns the top assist makers for a specific competition and team",
"required": [
"competition",
"team",
"number"
]
},
{
"key": "us_history_population_by_state_year",
"description": "Retrieve historical population data for a specific U.S. state and year.",
"required": [
"state",
"year"
]
}
]
}Question tool · choice · primary (ranked)
Which entry of `tools` should be invoked to satisfy `request`? Use each tool's description and required parameters; if none of them can do it, choose none_of_these.
8 options
gettopgoalscorers— Returns the top goal scorers for a specific competition and teamus_president_in_year— Retrieve the name of the U.S. president in a given year.geodistance_find— Find the distance between two cities on the globe.us_economy_gdp_by_state_year— Retrieve historical GDP data for a specific U.S. state and year.us_history_get_president— Retrieve the U.S. president during a specific event in American history.gettopassists— Returns the top assist makers for a specific competition and teamus_history_population_by_state_year— Retrieve historical population data for a specific U.S. state and year.none_of_these— No offered tool can satisfy the request; do not call any of them.
Turn 2 — each system's response · Turn 3 — the grade
Expected answer: us_history_population_by_state_year
| System | Top pick | Grade | P(expected) | Proper score | Distribution |
|---|---|---|---|---|---|
| Jev | us_history_population_by_state_year | correct | 100.0% | 1.000 |
|
| OpenJev | us_history_population_by_state_year | correct | 100.0% | 1.000 |
|
| JevFish | us_history_population_by_state_year | correct | 99.9% | 1.000 |
|
| decider-2b | us_history_population_by_state_year | correct | 82.1% | 0.970 |
|
| System One Scorer | none_of_theseinput truncated | wrong | 14.1% | 0.291 |
|
| NanoJev | us_history_population_by_state_year | correct | 35.1% | 0.731 |
|