Skip to benchmark
Blobfish AI
Talk to usOpen Studio
Blobfish ResearchFactoryBench-100 v1.0.0Public

Can an agent run the factory workflow—not just describe the next screen?

FactoryBench-100 measures complete manufacturing work: validate the policy and source records, transact across an Oracle-shaped ERP, preserve lot and approval controls, and leave the factory in the exact verifiable state the task requires.

View measured controls Browse all 100 tasks
Run on Harbor ↗Dataset on Hugging Face ↗Executable world ↗
100factory workflows
36typed tools
33ERP tables
10workflow families
0LLM grading calls
Measured controls first

The rows below are executed qualification controls, not fabricated model scores: a 100/100 strict oracle, an incomplete workflow, a read-only controller, and a control-skipping ablation. Version-pinned model rows stay empty until those runs exist.

Inspect the scoring contract →

Measured performance

Leaderboard

Every displayed row was replayed against all 100 released worlds. These are solvability and diagnostic controls; none is presented as a language-model submission.

RankAgentCoverageFactoryScoreRun
REF
Reference oraclefactorybench.evaluation deterministic replay
100 tasks
100.0%
6.5 calls avg.Measured solvability ceiling from replaying the checked-in reference workflow; not a model submission.
CTL
Incomplete-workflow controlfactorybench.evaluation deterministic replay
100 tasks
81.8%
5.5 calls avg.Measured ablation that omits the final ERP mutation while still submitting the oracle answer.
CTL
Read-only controlfactorybench.evaluation deterministic replay
100 tasks
46.0%
3.1 calls avg.Measured control that performs the required reads but makes no ERP changes or answer submission.
CTL
No-control ablationfactorybench.evaluation deterministic replay
100 tasks
15.3%
3.4 calls avg.Measured ablation that skips all policy and preflight reads; the environment rejects its writes.
No version-pinned language-model submission has run the released dataset yet. The measured controls remain visible above.
100%FactoryScore

100 × deterministic workflow checks passed / checks available, averaged over tasks.

100 distinct tasks

Task browser

Search all 100 executable tasks. Every catalog row publishes its prompt, governing policy, starting ERP records, exact checks, and oracle call count.

100 shown
factorybench-001Order release

Release SO-0001 to manufacturing

Release sales order SO-0001 for 5 FG-PANEL units into production at SEA. Check the governing policy, credit status, effective BOM, and component availability; create and fully reserve the work order. Return the work-order ID, released quantity, and scheduled completion date.

Company
Northstar Controls Manufacturing
As of
2026-01-12
Seeded files
2
Reference path
9 MCP calls

Complete task prompt

7 graded criteria summarized
Release sales order SO-0001 for 5 FG-PANEL units into production at SEA. Check the governing policy, credit status, effective BOM, and component availability; create and fully reserve the work order. Return the work-order ID, released quantity, and scheduled completion date.
Rubric summary
  • A released discrete work order links to the sales order.
  • All three BOM components are reserved.
  • The two-per-assembly relay requirement is fully reserved.
  • Read-before-write control
  • Exact answer fields
  • Write-scope containment
  • Error-free execution
Open full task specification

Seeded evidence

Asset room

Each task exposes the synthetic policy, exact starting-state records, and deterministic verifier contract used by its isolated world.

policy.mdassets/factorybench-001Governing PolicyMarkdown184 B

Release only when the customer is not on credit hold, an effective BOM exists, and every component has enough available stock. Reserve the exact BOM quantity.

Versioned synthetic operating policy retrieved through plant_docs.Open released file
starting-state.jsonassets/factorybench-001PrimaryJSON9.5 KB

37 seeded records across 12 populated ERP tables.

Exact synthetic records used to seed the isolated SQLite world.Open released file
expected-checks.jsonassets/factorybench-001VerifierJSON1.1 KB

7 deterministic workflow checks plus exact answer fields.

Public check contract; no LLM judge participates in scoring.Open released file

Runnable world

Environment and tool contract

A 33-table SQLite world exposes 36 typed operations through oracle_erp, plant_docs, and factory_harness. Writes are transactional and scoped per task.

MCP package pinfactory-agent-simulation@1.0.0
MCP protocol2025-03-26
Worldnorthstar-controls-erp-v1
DatabaseSQLite task snapshot
Harbor base imagepython:3.12-slim@sha256:7a8b475003c4fe15a2cd4e55e5cfc2f3560bdc9333d624f24cdd6d4340fd7a17
Network in reward pathnone
approve_invoiceoracle_erp

oracle_erp: execute approve invoice in the task world.

{
  "additionalProperties": false,
  "properties": {
    "invoice_id": {
      "type": "string"
    }
  },
  "required": [
    "invoice_id"
  ],
  "type": "object"
}
approve_purchase_orderoracle_erp

oracle_erp: execute approve purchase order in the task world.

{
  "additionalProperties": false,
  "properties": {
    "approver_id": {
      "type": "string"
    },
    "purchase_order_id": {
      "type": "string"
    }
  },
  "required": [
    "purchase_order_id",
    "approver_id"
  ],
  "type": "object"
}
approve_requisitionoracle_erp

oracle_erp: execute approve requisition in the task world.

{
  "additionalProperties": false,
  "properties": {
    "approver_id": {
      "type": "string"
    },
    "requisition_id": {
      "type": "string"
    }
  },
  "required": [
    "requisition_id",
    "approver_id"
  ],
  "type": "object"
}
complete_operationoracle_erp

oracle_erp: execute complete operation in the task world.

{
  "additionalProperties": false,
  "properties": {
    "actual_hours": {
      "type": "number"
    },
    "sequence": {
      "type": "integer"
    },
    "work_order_id": {
      "type": "string"
    }
  },
  "required": [
    "work_order_id",
    "sequence",
    "actual_hours"
  ],
  "type": "object"
}
complete_transferoracle_erp

oracle_erp: execute complete transfer in the task world.

{
  "additionalProperties": false,
  "properties": {
    "arrival_date": {
      "type": "string"
    },
    "transfer_id": {
      "type": "string"
    }
  },
  "required": [
    "transfer_id",
    "arrival_date"
  ],
  "type": "object"
}
complete_work_orderoracle_erp

oracle_erp: execute complete work order in the task world.

{
  "additionalProperties": false,
  "properties": {
    "completed_qty": {
      "type": "number"
    },
    "scrap_qty": {
      "type": "number"
    },
    "work_order_id": {
      "type": "string"
    }
  },
  "required": [
    "work_order_id",
    "completed_qty",
    "scrap_qty"
  ],
  "type": "object"
}
create_maintenance_work_orderoracle_erp

oracle_erp: execute create maintenance work order in the task world.

{
  "additionalProperties": false,
  "properties": {
    "expected_finish": {
      "type": "string"
    },
    "failure_code": {
      "type": "string"
    },
    "maintenance_id": {
      "type": "string"
    },
    "priority": {
      "type": "string"
    },
    "scheduled_start": {
      "type": "string"
    },
    "workcenter_id": {
      "type": "string"
    }
  },
  "required": [
    "maintenance_id",
    "workcenter_id",
    "priority",
    "scheduled_start",
    "expected_finish",
    "failure_code"
  ],
  "type": "object"
}
create_nonconformanceoracle_erp

oracle_erp: execute create nonconformance in the task world.

{
  "additionalProperties": false,
  "properties": {
    "disposition": {
      "type": "string"
    },
    "inspection_id": {
      "type": "string"
    },
    "nonconformance_id": {
      "type": "string"
    },
    "owner_id": {
      "type": "string"
    }
  },
  "required": [
    "nonconformance_id",
    "inspection_id",
    "disposition",
    "owner_id"
  ],
  "type": "object"
}
create_purchase_orderoracle_erp

oracle_erp: execute create purchase order in the task world.

{
  "additionalProperties": false,
  "properties": {
    "buyer_id": {
      "type": "string"
    },
    "item_id": {
      "type": "string"
    },
    "promised_date": {
      "type": "string"
    },
    "purchase_order_id": {
      "type": "string"
    },
    "quantity": {
      "type": "number"
    },
    "requisition_id": {
      "type": "string"
    },
    "supplier_id": {
      "type": "string"
    },
    "unit_price": {
      "type": "number"
    }
  },
  "required": [
    "purchase_order_id",
    "requisition_id",
    "supplier_id",
    "buyer_id",
    "item_id",
    "quantity",
    "unit_price",
    "promised_date"
  ],
  "type": "object"
}
create_requisitionoracle_erp

oracle_erp: execute create requisition in the task world.

{
  "additionalProperties": false,
  "properties": {
    "item_id": {
      "type": "string"
    },
    "need_by": {
      "type": "string"
    },
    "quantity": {
      "type": "number"
    },
    "requester_id": {
      "type": "string"
    },
    "requisition_id": {
      "type": "string"
    },
    "supplier_id": {
      "type": "string"
    },
    "unit_price": {
      "type": "number"
    },
    "work_order_id": {
      "type": "string"
    }
  },
  "required": [
    "requisition_id",
    "requester_id",
    "work_order_id",
    "supplier_id",
    "item_id",
    "quantity",
    "unit_price",
    "need_by"
  ],
  "type": "object"
}
create_transferoracle_erp

oracle_erp: execute create transfer in the task world.

{
  "additionalProperties": false,
  "properties": {
    "from_plant": {
      "type": "string"
    },
    "from_subinventory": {
      "type": "string"
    },
    "item_id": {
      "type": "string"
    },
    "lot_number": {
      "type": "string"
    },
    "quantity": {
      "type": "number"
    },
    "to_plant": {
      "type": "string"
    },
    "to_subinventory": {
      "type": "string"
    },
    "transfer_id": {
      "type": "string"
    },
    "transferred_at": {
      "type": "string"
    }
  },
  "required": [
    "transfer_id",
    "item_id",
    "lot_number",
    "from_plant",
    "from_subinventory",
    "to_plant",
    "to_subinventory",
    "quantity",
    "transferred_at"
  ],
  "type": "object"
}
create_work_orderoracle_erp

oracle_erp: execute create work order in the task world.

{
  "additionalProperties": false,
  "properties": {
    "item_id": {
      "type": "string"
    },
    "quantity": {
      "type": "number"
    },
    "sales_order_id": {
      "type": "string"
    },
    "scheduled_completion": {
      "type": "string"
    },
    "scheduled_start": {
      "type": "string"
    },
    "work_order_id": {
      "type": "string"
    },
    "workcenter_id": {
      "type": "string"
    }
  },
  "required": [
    "work_order_id",
    "sales_order_id",
    "item_id",
    "quantity",
    "scheduled_start",
    "scheduled_completion",
    "workcenter_id"
  ],
  "type": "object"
}
get_bomoracle_erp

oracle_erp: execute get bom in the task world.

{
  "additionalProperties": false,
  "properties": {
    "as_of": {
      "type": "string"
    },
    "item_id": {
      "type": "string"
    }
  },
  "required": [
    "item_id",
    "as_of"
  ],
  "type": "object"
}
get_inventoryoracle_erp

oracle_erp: execute get inventory in the task world.

{
  "additionalProperties": false,
  "properties": {
    "item_ids": {
      "items": {
        "type": "string"
      },
      "type": "array"
    },
    "plant_id": {
      "type": "string"
    }
  },
  "required": [
    "plant_id",
    "item_ids"
  ],
  "type": "object"
}
get_invoice_matchoracle_erp

oracle_erp: execute get invoice match in the task world.

{
  "additionalProperties": false,
  "properties": {
    "invoice_id": {
      "type": "string"
    }
  },
  "required": [
    "invoice_id"
  ],
  "type": "object"
}
get_maintenance_contextoracle_erp

oracle_erp: execute get maintenance context in the task world.

{
  "additionalProperties": false,
  "properties": {
    "workcenter_id": {
      "type": "string"
    }
  },
  "required": [
    "workcenter_id"
  ],
  "type": "object"
}
get_purchase_orderoracle_erp

oracle_erp: execute get purchase order in the task world.

{
  "additionalProperties": false,
  "properties": {
    "purchase_order_id": {
      "type": "string"
    }
  },
  "required": [
    "purchase_order_id"
  ],
  "type": "object"
}
get_quality_contextoracle_erp

oracle_erp: execute get quality context in the task world.

{
  "additionalProperties": false,
  "properties": {
    "inspection_id": {
      "type": "string"
    }
  },
  "required": [
    "inspection_id"
  ],
  "type": "object"
}
get_receiptoracle_erp

oracle_erp: execute get receipt in the task world.

{
  "additionalProperties": false,
  "properties": {
    "receipt_id": {
      "type": "string"
    }
  },
  "required": [
    "receipt_id"
  ],
  "type": "object"
}
get_requisitionoracle_erp

oracle_erp: execute get requisition in the task world.

{
  "additionalProperties": false,
  "properties": {
    "requisition_id": {
      "type": "string"
    }
  },
  "required": [
    "requisition_id"
  ],
  "type": "object"
}
get_sales_orderoracle_erp

oracle_erp: execute get sales order in the task world.

{
  "additionalProperties": false,
  "properties": {
    "sales_order_id": {
      "type": "string"
    }
  },
  "required": [
    "sales_order_id"
  ],
  "type": "object"
}
get_scheduleoracle_erp

oracle_erp: execute get schedule in the task world.

{
  "additionalProperties": false,
  "properties": {
    "plant_id": {
      "type": "string"
    },
    "work_order_id": {
      "type": "string"
    }
  },
  "required": [
    "plant_id",
    "work_order_id"
  ],
  "type": "object"
}
get_supplier_quotesoracle_erp

oracle_erp: execute get supplier quotes in the task world.

{
  "additionalProperties": false,
  "properties": {
    "item_id": {
      "type": "string"
    },
    "need_by": {
      "type": "string"
    },
    "task_id": {
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "item_id",
    "need_by"
  ],
  "type": "object"
}
get_work_orderoracle_erp

oracle_erp: execute get work order in the task world.

{
  "additionalProperties": false,
  "properties": {
    "work_order_id": {
      "type": "string"
    }
  },
  "required": [
    "work_order_id"
  ],
  "type": "object"
}
hold_invoiceoracle_erp

oracle_erp: execute hold invoice in the task world.

{
  "additionalProperties": false,
  "properties": {
    "invoice_id": {
      "type": "string"
    },
    "reason": {
      "type": "string"
    }
  },
  "required": [
    "invoice_id",
    "reason"
  ],
  "type": "object"
}
issue_materialoracle_erp

oracle_erp: execute issue material in the task world.

{
  "additionalProperties": false,
  "properties": {
    "item_id": {
      "type": "string"
    },
    "lot_number": {
      "type": "string"
    },
    "occurred_at": {
      "type": "string"
    },
    "plant_id": {
      "type": "string"
    },
    "quantity": {
      "type": "number"
    },
    "subinventory": {
      "type": "string"
    },
    "transaction_id": {
      "type": "string"
    },
    "work_order_id": {
      "type": "string"
    }
  },
  "required": [
    "transaction_id",
    "work_order_id",
    "item_id",
    "plant_id",
    "subinventory",
    "lot_number",
    "quantity",
    "occurred_at"
  ],
  "type": "object"
}
place_quality_holdoracle_erp

oracle_erp: execute place quality hold in the task world.

{
  "additionalProperties": false,
  "properties": {
    "hold_id": {
      "type": "string"
    },
    "inspection_id": {
      "type": "string"
    },
    "item_id": {
      "type": "string"
    },
    "lot_number": {
      "type": "string"
    },
    "reason_code": {
      "type": "string"
    }
  },
  "required": [
    "hold_id",
    "inspection_id",
    "item_id",
    "lot_number",
    "reason_code"
  ],
  "type": "object"
}
receive_purchase_orderoracle_erp

oracle_erp: execute receive purchase order in the task world.

{
  "additionalProperties": false,
  "properties": {
    "item_id": {
      "type": "string"
    },
    "lot_number": {
      "type": "string"
    },
    "purchase_order_id": {
      "type": "string"
    },
    "quantity": {
      "type": "number"
    },
    "receipt_id": {
      "type": "string"
    },
    "received_at": {
      "type": "string"
    },
    "receiver_id": {
      "type": "string"
    }
  },
  "required": [
    "receipt_id",
    "purchase_order_id",
    "receiver_id",
    "item_id",
    "quantity",
    "lot_number",
    "received_at"
  ],
  "type": "object"
}
record_inspectionoracle_erp

oracle_erp: execute record inspection in the task world.

{
  "additionalProperties": false,
  "properties": {
    "accepted_qty": {
      "type": "number"
    },
    "inspected_qty": {
      "type": "number"
    },
    "inspection_id": {
      "type": "string"
    },
    "inspector_id": {
      "type": "string"
    },
    "item_id": {
      "type": "string"
    },
    "lot_number": {
      "type": "string"
    },
    "rejected_qty": {
      "type": "number"
    },
    "result": {
      "type": "string"
    },
    "source_id": {
      "type": "string"
    },
    "source_type": {
      "type": "string"
    }
  },
  "required": [
    "inspection_id",
    "source_type",
    "source_id",
    "item_id",
    "lot_number",
    "inspected_qty",
    "accepted_qty",
    "rejected_qty",
    "result",
    "inspector_id"
  ],
  "type": "object"
}
record_wip_varianceoracle_erp

oracle_erp: execute record wip variance in the task world.

{
  "additionalProperties": false,
  "properties": {
    "labor_variance": {
      "type": "number"
    },
    "material_variance": {
      "type": "number"
    },
    "overhead_variance": {
      "type": "number"
    },
    "variance_id": {
      "type": "string"
    },
    "work_order_id": {
      "type": "string"
    }
  },
  "required": [
    "variance_id",
    "work_order_id",
    "material_variance",
    "labor_variance",
    "overhead_variance"
  ],
  "type": "object"
}
reroute_operationoracle_erp

oracle_erp: execute reroute operation in the task world.

{
  "additionalProperties": false,
  "properties": {
    "sequence": {
      "type": "integer"
    },
    "work_order_id": {
      "type": "string"
    },
    "workcenter_id": {
      "type": "string"
    }
  },
  "required": [
    "work_order_id",
    "sequence",
    "workcenter_id"
  ],
  "type": "object"
}
reschedule_work_orderoracle_erp

oracle_erp: execute reschedule work order in the task world.

{
  "additionalProperties": false,
  "properties": {
    "scheduled_completion": {
      "type": "string"
    },
    "scheduled_start": {
      "type": "string"
    },
    "work_order_id": {
      "type": "string"
    },
    "workcenter_id": {
      "type": "string"
    }
  },
  "required": [
    "work_order_id",
    "scheduled_start",
    "scheduled_completion",
    "workcenter_id"
  ],
  "type": "object"
}
reserve_materialoracle_erp

oracle_erp: execute reserve material in the task world.

{
  "additionalProperties": false,
  "properties": {
    "item_id": {
      "type": "string"
    },
    "lot_number": {
      "type": "string"
    },
    "plant_id": {
      "type": "string"
    },
    "quantity": {
      "type": "number"
    },
    "reservation_id": {
      "type": "string"
    },
    "subinventory": {
      "type": "string"
    },
    "work_order_id": {
      "type": "string"
    }
  },
  "required": [
    "reservation_id",
    "work_order_id",
    "item_id",
    "plant_id",
    "subinventory",
    "lot_number",
    "quantity"
  ],
  "type": "object"
}
search_documentsplant_docs

plant_docs: execute search documents in the task world.

{
  "additionalProperties": false,
  "properties": {
    "category": {
      "type": "string"
    }
  },
  "required": [
    "category"
  ],
  "type": "object"
}
start_operationoracle_erp

oracle_erp: execute start operation in the task world.

{
  "additionalProperties": false,
  "properties": {
    "sequence": {
      "type": "integer"
    },
    "work_order_id": {
      "type": "string"
    }
  },
  "required": [
    "work_order_id",
    "sequence"
  ],
  "type": "object"
}
submit_answerfactory_harness

factory_harness: execute submit answer in the task world.

{
  "additionalProperties": true,
  "properties": {},
  "required": [],
  "type": "object"
}

Architecture comparison

The same evaluation spine, assembled for manufacturing ERP

Archipelago and FactoryBench share the central idea that an agent should act through tools in an isolated world and be graded from observable outcomes. FactoryBench adds a released Oracle-shaped schema, task-level control checks, exact answer fields, and full public oracle replays.

Inspect Archipelago
LayerMercor ArchipelagoBlobfish FactoryBench-100
EnvironmentContainerized task environments with snapshot lifecyclePer-task SQLite snapshots generated from public synthetic seed records
Tool surfaceMCP gateway and environment serversThree MCP namespaces: oracle_erp, plant_docs, factory_harness
Agent runAgent runner records tool interactionHarness records every read, transaction, rejection, and submission
GradingBefore/after snapshot gradersState assertions + read order + exact answer + write scope + errors
DistributionOpen-source framework and role benchmarksGitHub world, Hugging Face rows, Harbor tasks, and website explorer

Replay the work

Trajectories

Replay one real oracle episode from each workflow family, including tool arguments, compact outcomes, and links to the full state diff and verifier report.

AgentReference oraclefactorybench.evaluation deterministic replay
Score100.0%Strict pass
Tool calls9factorybench-001
Run cost$0.00
11 events
Full transcript Verifier report
  1. Agent

    Release sales order SO-0001 for 5 FG-PANEL units into production at SEA. Check the governing policy, credit status, effective BOM, and component availability; create and fully reserve the work order. Return the work-order ID, released quantity, and scheduled completion date.

  2. 1
    Policysearch_documentsok
    {
  "category": "order_release"
}

    {"documents": [{"body": "Release only when the customer is not on credit hold, an effective BOM exists, and every component has enough available stock. Reserve the exact BOM quantity.", "category": "order_release", "doc_id": "POL-01", "sha256": "f02d22e29d62e78093fa85720ae5d9dcd6620f34621e0b615bdc04230e020876", "title": "Order release control"}]}

  3. 2
    Inspectget_sales_orderok
    {
  "sales_order_id": "SO-0001"
}

    {"header": {"credit_hold": 0, "customer_name": "Atlas Systems 1", "priority": "Standard", "requested_date": "2026-01-16", "sales_order_id": "SO-0001", "status": "Booked", "task_id": "factorybench-001"}, "lines": [{"item_id": "FG-PANEL", "line_no": 1, "ordered_qty": 5.0, "sales_order_id": "SO-0001", "shipped_qty": 0.0}]}

  4. 3
    Inspectget_bomok
    {
  "as_of": "2026-01-12",
  "item_id": "FG-PANEL"
}

    {"components": [{"bom_id": "BOM-PANEL-C", "component_item_id": "RM-HOUSING", "operation_sequence": 10, "quantity_per": 1.0, "yield_factor": 1.0}, {"bom_id": "BOM-PANEL-C", "component_item_id": "RM-COPPER", "operation_sequence": 20, "quantity_per": 1.0, "yield_factor": 1.0}, {"bom_id": "BOM-PANEL-C", "component_item_id": "RM-RELAY", "operation_sequence": 20, …

  5. 4
    Inspectget_inventoryok
    {
  "item_ids": [
    "RM-HOUSING",
    "RM-RELAY",
    "RM-COPPER"
  ],
  "plant_id": "SEA"
}

    {"lots": [{"available_qty": 7.0, "expiration_date": null, "item_id": "RM-COPPER", "lot_number": "C-001", "plant_id": "SEA", "quantity": 7.0, "reserved_qty": 0.0, "status": "Unrestricted", "subinventory": "STORES"}, {"available_qty": 8.0, "expiration_date": null, "item_id": "RM-HOUSING", "lot_number": "H-001", "plant_id": "SEA", "quantity": 8.0, "reserved_qty…

  6. 5
    Transactcreate_work_orderok
    {
  "item_id": "FG-PANEL",
  "quantity": 5,
  "sales_order_id": "SO-0001",
  "scheduled_completion": "2026-01-16",
  "scheduled_start": "2026-01-13",
  "work_order_id": "WO-0001",
  "workcenter_id": "WC-ASM-1"
}

    {"requirements_created": 3, "status": "Released", "work_order_id": "WO-0001"}

  7. 6
    Transactreserve_materialok
    {
  "item_id": "RM-HOUSING",
  "lot_number": "H-001",
  "plant_id": "SEA",
  "quantity": 5,
  "reservation_id": "RES-H-001",
  "subinventory": "STORES",
  "work_order_id": "WO-0001"
}

    {"reservation_id": "RES-H-001", "status": "Active"}

  8. 7
    Transactreserve_materialok
    {
  "item_id": "RM-RELAY",
  "lot_number": "R-001",
  "plant_id": "SEA",
  "quantity": 10,
  "reservation_id": "RES-R-001",
  "subinventory": "STORES",
  "work_order_id": "WO-0001"
}

    {"reservation_id": "RES-R-001", "status": "Active"}

  9. 8
    Transactreserve_materialok
    {
  "item_id": "RM-COPPER",
  "lot_number": "C-001",
  "plant_id": "SEA",
  "quantity": 5,
  "reservation_id": "RES-C-001",
  "subinventory": "STORES",
  "work_order_id": "WO-0001"
}

    {"reservation_id": "RES-C-001", "status": "Active"}

  10. 9
    Submitsubmit_answerok
    {
  "completion_date": "2026-01-16",
  "released_quantity": 5,
  "work_order_id": "WO-0001"
}

    {"submitted": {"completion_date": "2026-01-16", "released_quantity": "5", "work_order_id": "WO-0001"}, "task_id": "factorybench-001"}

  11. Agent

    Verifier: 7/7 checks passed; FactoryScore 100.00.

Methodology

How the numbers are made

The benchmark is grounded in production-style ERP controls while remaining synthetic, clean-room, and fully reproducible.

01

One metric, inspectable evidence

FactoryScore is the mean percentage of deterministic checks passed. Each task checks required reads before writes, exact ERP state transitions, exact answer fields, write-scope containment, and tool-call validity. Strict completion is shown as supporting evidence, not a second benchmark metric.

02

Real workflow shape, synthetic company data

The suite is a clean-room simulation shaped by real manufacturing operations: multi-record order intake, SKU and supplier preflight, approval limits, transactional rollback, lot-level inventory, tax-and-amount reconciliation, and production recovery. It contains no Oracle binaries, proprietary screens, customer records, or copied benchmark tasks.

03

Qualification before publication

The release replays every reference workflow, replays a deterministic sample twice, runs three negative controls, and removes every reference mutation one at a time. Qualification passed: 100/100 reference strict passes; 240/240 mutation omissions were detected; impaired controls remain below the oracle.

04

Public inspirations and design boundary

Enterprise-Bench informed noisy multi-system task framing; ERP-Bench informed long-horizon manufacturing coverage; Mercor APEX informed the public task, asset, environment, and trajectory presentation. FactoryBench tasks, records, tools, and verifiers are independently authored and executable under this repository's licenses.

Run it yourself

The tasks, world, trajectories, and verifiers are public.

Download the rows, run the Harbor tasks, inspect every synthetic record, and submit the first version-pinned model result.

Open Harbor dataset ↗Download dataset ↗