Can an agent answer the factory question—and safely carry the decision through?
FactoryBench-100 starts with questions employees actually ask: when can we ship, what blocks an earlier date, and which option is viable? The agent must decide where to look, reconcile Oracle, email, Drive, Sheets, and Slack, calculate the alternatives, then make only the approved changes.
The leaderboard reports a version-pinned Harbor run over 100/100tasks at 76.20 FactoryScore. Qualification controls are disclosed separately and never enter the ranking.
Oracle passes
100/100
Exact replays
100/100
Attacks rejected
1,200/1,200
Write contracts
100/100
1,400 isolated executions covered 12 attack types with 0 false accepts. All writes were authorized, agent-visible, schema-validated, persisted, read back, and contained. Wrong-target and keyword-stuffing attempts were rejected; the verifier required neither hidden reference text nor hidden serialization.
The pinned model row is a real Harbor run over 100/100 released tasks. Oracle and diagnostic replays are disclosed separately in methodology.
Ranking
Model leaderboard
Only complete, version-pinned model runs on this exact release receive a rank.
RankAgentCoverageFactoryScoreRun
1
gpt-5.6-lunaHarbor 0.21.0 · release 3.3.5 · codex 0.151.0 · max reasoning · web search disabled · 100/100 native MCP traces
100 tasks
76.2%
76.96 calls avg.$0.09 / task · Full 100/100-task version-pinned model run derived from the exact Harbor trial artifacts; no oracle or deterministic-control trials are ranked. Scoring observes only released task state and answers, Codex built-in web search was disabled, and OS-level no-egress isolation is not claimed. Provider-native trace provenance is reported per task (100/100 native, 0/100 explicit Harbor agent fallbacks); trials with no recorded MCP operation retain that fallback rather than being presented as provider calls.Inspect run
Qualification evidence
Measured controls
Oracle and adversarial replays prove solvability and verifier discrimination. They are measured, but never ranked as model submissions.
33 calls avg.Measured control that reaches the reference provider state but chooses a rejected operating alternative.
100%FactoryScore
100 × passed task-specific deterministic criterion weight / available criterion weight, averaged over tasks.
100 distinct tasks
Task browser
Search all 100 distinct employee requests. Each gives the agent a business question—not a task list—then leaves it to investigate 28 heterogeneous sources, reconcile identifiers, calculate and compare options, and commit the supported outcome. The published reference is one valid solution, not a required call order.
100 shown
factorybench-001Customer commitment
Commit one pallet of Luma lamps
Sales needs a date for one pallet of Luma lamps. When can we honestly have it finished with today's schedule and stock, what is preventing an earlier run, and do expediting parts or adding a shift change the answer? Commit the best approved plan and prepare the launch-team update for review.
Company
Northstar Controls Manufacturing
As of
2026-01-12
Seeded files
28
Reference solution
33 MCP calls
Complete task prompt
51 graded criteria summarized
Sales needs a date for one pallet of Luma lamps. When can we honestly have it finished with today's schedule and stock, what is preventing an earlier run, and do expediting parts or adding a shift change the answer? Commit the best approved plan and prepare the launch-team update for review.
How the employee outcome is evaluated
Reasoning, persisted state, and the answer must agree.
14 semantic milestones
Reconcile the plan decision for one pallet of Luma lamps; apply material readiness versus the protected assembly calendar; quarantined and reserved bulb units = 60 EA; uncovered or unsupported scope = 120.0 EA, compare the three task-specific alternatives, and persist only the supported scope.
Strict success
Every task-specific semantic milestone must hold over successful evidence reads, the exact final state, post-write readback, answer fields, and write-scope containment.
Ordering policy
The reference trajectory is illustrative, not graded. Material identity, authority, constraint, and ERP discoveries must precede the dependent mutation; their order and valid query shapes are otherwise open.
Inspect the task-specific causal milestones
outcome
milestone-01
Establish the isolated CASE-001 scope and immutable identities across the mailbox, Drive index, and mounted systems before relying on similarly named records.
outcome
milestone-02
Determine the operative C requirement for NS-000001, reconcile the current source material, and distinguish current evidence from superseded or merely contextual records.
outcome
milestone-03
Reconcile the supported operating inputs: Reconciled 420 observed less 60 excluded to 360 supported EA for nettable LMP-BULB-12 units. Applied qualified non-displacing WC-2 slots to derive the three option outcomes without using excluded or protected scope.
outcome
milestone-04
Establish what is actually authorized and what remains conditional: Applied AP-0001 only to standard_plan, NS-000001, and the supported 360 EA; kept broader scope and add_weekend_shift outside current authority.
outcome
milestone-05
Correlate NS-000001 by immutable IDs across the material Oracle records (inventory onhand balances, sales orders, work order materials, work order operations, work orders) and preserve the effective C state.
outcome
milestone-06
Derive the eligible scope, exclusions, unit, and controlling threshold from the joined evidence rather than copying a headline quantity for one pallet of Luma lamps.
outcome
milestone-07
Calculate the feasible timing for standard_plan under the actual operating window and report whether the 2026-01-20 outcome meets the control date.
outcome
milestone-08
Compare all three task-specific alternatives, including timing, cost or exposure, authority, and residual risk; select standard_plan only because the joined evidence supports it.
outcome
milestone-09
Persist exactly the supported 'Production plan committed' transition for NS-000001 through the documented Oracle operation, with no neighboring record or unsupported field changed.
outcome
milestone-10
After the Oracle outcome is known, write the outcome into the existing control-workbook cell; and create the requested email draft without sending it with the supported decision, timing, constraint, and business reference.
outcome
milestone-11
Read NS-000001 back after the primary mutation and verify the persisted provider state rather than trusting the write acknowledgement.
outcome
milestone-12
Return the exact task-supported recommendation, outcome date, immutable business record, decisive operating insight, and timing status for one pallet of Luma lamps.
outcome
milestone-13
Keep all successful changes inside CASE-001's declared Oracle, collaboration, answer, and audit scope.
outcome
milestone-14
Complete without a rejected state-changing call; failed exploratory reads may be recovered from, but an invalid mutation is not accepted.
Decision space (3 grounded options)
standard_plan: outcome 2026-01-20, economic impact USD 0, SUPPORTED_AND_APPROVED — selected: standard plan uses only nettable LMP-BULB-12 units, satisfies material readiness versus the protected assembly calendar, lands 1 day(s) on or ahead of the control date, and carries the documented economic impact of USD 0.
expedite_bulbs: outcome 2026-01-20, economic impact USD 640, FEASIBLE_WITH_INFERIOR_TRADEOFF: expedite bulbs would finish on the same date as standard_plan and adds USD 640 versus standard_plan; it remains controlled but is not the best authorized tradeoff once quarantined and reserved bulb units stays out of scope.
add_weekend_shift: outcome 2026-01-18, economic impact USD 1280, SEPARATE_APPROVAL_OR_POLICY_EXCEPTION_REQUIRED: add weekend shift would finish 2 day(s) earlier than standard_plan and adds USD 1280 versus standard_plan, but current approval does not authorize it and it could expose quarantined and reserved bulb units.
Inspect all 51 deterministic criteria
Investigation: Established the isolated CASE-001 scope, immutable handles, mounted systems, and evidence index before investigating NS-000001. [investigation_01]
Investigation: Located the task-scoped correspondence for CASE-001 before opening any message; did not rely on a filename or a guessed sender. [investigation_02]
Investigation: Enumerated the case-scoped Drive records for CASE-001 and used their immutable file IDs to distinguish the effective specification, approval, and option workbook. [investigation_04]
Investigation: Used the independently confirmed 2026-01-14 readiness input for expedite_bulbs, then separately derived its 2026-01-20 operating outcome under material readiness versus the protected assembly calendar instead of treating an external promise as authorization or a completion date. [investigation_03]
Investigation: Opened the effective control specification and Applied revision C and established 480 EA for revision-C bulbs required for 120 four-lamp kits, with control date 2026-01-21. [investigation_05]
Investigation: Opened the task-scoped source-input workbook and Applied revision C and established 480 EA for revision-C bulbs required for 120 four-lamp kits, with control date 2026-01-21. [investigation_06]
Investigation: Opened the independent reconciliation rows and Reconciled 420 observed less 60 excluded to 360 supported EA for nettable LMP-BULB-12 units. [investigation_07]
Investigation: Opened the raw operating-window calendar and Applied qualified non-displacing WC-2 slots to derive the three option outcomes without using excluded or protected scope. [investigation_08]
Investigation: Reconciled 420 observed less 60 excluded to 360 supported EA for nettable LMP-BULB-12 units. Applied qualified non-displacing WC-2 slots to derive the three option outcomes without using excluded or protected scope. [investigation_10]
Investigation: Opened the task-scoped signed approval record and Applied AP-0001 only to standard_plan, NS-000001, and the supported 360 EA; kept broader scope and add_weekend_shift outside current authority. [investigation_09]
Investigation: Read the scoped operations discussion, confirmed the independently stated stakeholder need date, and separated operational context from formal approval: Applied AP-0001 only to standard_plan, NS-000001, and the supported 360 EA; kept broader scope and add_weekend_shift outside current authority. [investigation_11]
Investigation: Correlated the Oracle sales orders record: Applied revision C and established 480 EA for revision-C bulbs required for 120 four-lamp kits, with control date 2026-01-21. [investigation_12]
Investigation: Correlated the Oracle work orders record: Located NS-000001 for one pallet of Luma lamps using immutable IDs and preserved effective revision C. [investigation_13]
Investigation: Correlated the Oracle work order operations record: Applied qualified non-displacing WC-2 slots to derive the three option outcomes without using excluded or protected scope. [investigation_14]
Investigation: Correlated the Oracle work order materials record: Applied revision C and established 480 EA for revision-C bulbs required for 120 four-lamp kits, with control date 2026-01-21. [investigation_15]
Investigation: Correlated the Oracle inventory onhand balances record: Reconciled 420 observed less 60 excluded to 360 supported EA for nettable LMP-BULB-12 units. [investigation_16]
Analysis: Derived 480 EA for revision-C bulbs required for 120 four-lamp kits at revision C. [derive_plan_requirement]
Analysis: Read 420 EA of gross observed nettable LMP-BULB-12 units. [read_gross_coverage]
Analysis: Excluded 60 EA for quarantined and reserved bulb units. [remove_ineligible_coverage]
Analysis: Kept every planning quantity in EA. [preserve_plan_unit]
Analysis: Calculated standard_plan outcome as 2026-01-20 under qualified non-displacing WC-2 slots. [compare_baseline_plan]
Analysis: Calculated expedite_bulbs outcome as 2026-01-20 using Rainier's confirmed standard and expedite dates. [compare_accelerated_plan]
Analysis: Calculated add_weekend_shift outcome as 2026-01-18 and kept its separate-approval condition. [compare_escalated_plan]
Analysis: Read 2026-01-17 as the independently confirmed standard readiness date for Rainier's confirmed standard and expedite dates. [read_standard_external_readiness]
Analysis: Read 2026-01-14 as the independently confirmed expedited readiness date for Rainier's confirmed standard and expedite dates. [read_expedited_external_readiness]
Analysis: Bound external recovery to the 120 EA uncovered requirement rather than ordering the full header quantity. [bound_external_recovery_quantity]
Analysis: Compared the 2026-01-14 expedited bulb readiness with the first non-displacing WC-2 slot on 2026-01-18 and proved that expediting alone still completes on 2026-01-20, saving 0 days. [test_lamp_expedite_against_capacity]
Analysis: Calculated that the 2026-01-17 weekend slot would complete on 2026-01-18, two days earlier than the selected plan, but kept it outside action because the shift lacks approval. [test_lamp_weekend_shift]
Analysis: Preserved 2026-01-21 as the documented date for one pallet of Luma lamps; did not infer urgency from the title. [identify_business_date]
Analysis: Calculated 2026-01-20 as the supported outcome date for standard_plan. [calculate_recommended_outcome]
Analysis: Compared 2026-01-20 with the independent control date 2026-01-21 and calculated a signed variance of -1 day(s). [calculate_outcome_variance]
Analysis: Reported ON_TIME; did not relabel a controlled but late result as on time. [state_honest_timing_status]
Decision: Applied USD 0 as the documented incremental recovery cost for standard_plan; did not invent a premium where spend is not a decision driver. [calculate_selected_cost]
Decision: Recognized that add_weekend_shift remains outside current authority and requires an additional approval. [apply_escalation_authority]
Decision: Compared the timing, economic impact, control status, and consequence of standard_plan, expedite_bulbs, and add_weekend_shift; selected standard_plan because it alone gives the best currently authorized result under material readiness versus the protected assembly calendar. [choose_task_specific_option]
State: Required immutable record NS-000001 to reach business outcome 'Production plan committed' through oracle_fusion.work_orders.update with exact provider-critical values WorkOrderId=100001, requestBody.WorkOrderStatusCode='ORA_RELEASED', requestBody.PlannedStartDate='2026-01-18', requestBody.PlannedCompletionDate='2026-01-20'. The audited transition binds selected option standard_plan, revision C, approval AP-0001, and constraint material readiness versus the protected assembly calendar; quarantined and reserved bulb units = 60 EA; uncovered or unsupported scope = 120.0 EA; no other provider record satisfies this state criterion. [mutation_01]
State: Wrote standard_plan, completion 2026-01-20, and the binding constraint into the existing task-scoped control cell without altering another row. [mutation_02]
State: Created—but did not send—the requested CASE-001 decision draft with completion, constraint, alternatives, and the Oracle business reference. [mutation_03]
Verification: Read NS-000001 back through the documented Oracle work orders.get operation after the mutation and confirmed the resource-specific changed fields ['LastUpdateDate', 'PlannedCompletionDate', 'PlannedStartDate', 'WorkOrderStatusCode'], including the effective update at 2026-01-20, rather than relying on the write acknowledgement. [verify_primary_oracle_state]
Verification: Reopened the task-scoped google_sheets.spreadsheets.values.update result through google_sheets.spreadsheets.values.get and confirmed the agent's actual written value persisted instead of trusting the mutation acknowledgement. [verify_collaboration_01]
Verification: Reopened the task-scoped gmail.drafts.create result through gmail.drafts.get and confirmed the agent's actual written value persisted instead of trusting the mutation acknowledgement. [verify_collaboration_02]
Answer: Reported recommended option as 'standard_plan', tied to NS-000001, revision C, and the selected standard_plan outcome. [answer_recommended_option]
Answer: Reported recommended outcome date as '2026-01-20', tied to NS-000001, revision C, and the selected standard_plan outcome. [answer_recommended_outcome_date]
Answer: Reported coverage item or resource as 'LMP-BULB-12', tied to NS-000001, revision C, and the selected standard_plan outcome. [answer_coverage_item_or_resource]
Answer: Reported shortage quantity as 120, tied to NS-000001, revision C, and the selected standard_plan outcome. [answer_shortage_quantity]
Answer: Reported decision timing status as 'ON_TIME', tied to NS-000001, revision C, and the selected standard_plan outcome. [answer_decision_timing_status]
Containment: Keep all successful changes inside CASE-001's declared Oracle, collaboration, answer, and audit scope. Atomic verifier selector: `write_scope`. [write_scope]
Execution: Complete without a rejected state-changing call; failed exploratory reads may be recovered from, but an invalid mutation is not accepted. Atomic verifier selector: `no_rejected_mutation`. [no_rejected_mutation]
The release contains 2,800 inspectable task files. Every isolated task includes 28 seeded source artifacts—PDFs, Excel workbooks, CSV exports, email, Drive, Slack, specifications, policies, and approvals—plus its starting state and verifier contract.
NameRoleFormatSize
operating-policy.mdassets/factorybench-001PolicyMarkdown810 B
Control rule: material readiness versus the protected assembly calendar. Establish the immutable source record and effective revision, then reconcile revision-C bulbs required for 120 four-lamp kits, nettable LMP-BULB-12 units, and quarantined and reserved bulb units from independent records. Do not treat a header total, filename, similar name, or unapproved
Sales needs a date for one pallet of Luma lamps. When can we honestly have it finished with today's schedule and stock, what is preventing an earlier run, and do expediting parts or adding a shift change the answer? Commit the best approved plan and prepare the launch-team update for review.
Please use the immutable references in the attached source record;
Account team needs one defensible decision for one pallet of Luma lamps by 2026-01-21; a faster or broader action has value only if it remains inside material readiness versus the protected assembly calendar. · I found 480 EA on the header, but the workbook has separate observation and exclusion rows. Please do not net from this message. · Approval AP-0001 p
Case: CASE-001
Document control number: SPEC-0001
Effective revision: C
Superseded revision visible in archive: R9
Subject: one pallet of Luma lamps
Primary measure: revision-C bulbs required for 120 four-lamp kits
Source finished or header quantity: 120
Effective usage per finished or header unit: 4 EA
Unit: EA
Eligibility definition: nettable LMP-BULB-12 u
NORTHSTAR CONTROLS — RELEASED ENGINEERING STRUCTURE · Case: CASE-001 · Controlled record: NS-000001 · Finished item or subject: LMP-BULB-12 · Released revision: C · Organization: SEA · Effective date: 2026-01-12 · Status: CURRENT AND RELEASED · · COMPONENT AND CONTROL BASIS · Primary usage or conversion: 4 EA per finished or header unit · Source quantity st
We matched the supplier response to LMP-BULB-12, revision C, and immutable record NS-000001. The standard and priority readiness dates are inputs only; neither is an ERP completion date.
The current catalog covers no more than 120 EA. It excludes quarantined and reserved bulb units, the archived revision R9, PDX demand, and nearby case CASE-002.
Before rel
A closed, stateful sandbox MCP exposes 94 typed Gmail, Drive, Sheets, Slack, and documented Oracle Fusion operation mappings over 7 private SQLite state tables. It returns JSON-RPC tool responses, persists writes per session, and grades the resulting cross-system state server-side; the release claims only the contract fidelity it publishes.
Live sandbox MCPfactorybench-001
Ready to create an isolated session
Released native runtime
Connect to 94 released FactoryBench-100 tools and mutate a private task snapshot.
The server creates a fresh task database, preserves all writes inside that session, and exposes the same tool implementation used by the Harbor release.Every button sends Streamable HTTP JSON-RPC to the checked-in world, records the native trajectory, and runs the release's deterministic verifier. No prerecorded tool responses are replayed; every result is computed from the session's seeded synthetic state.
The same evaluation spine, assembled for manufacturing ERP
Archipelago and FactoryBench share the central idea that an agent should act through tools in an isolated world and be graded from observable outcomes. FactoryBench adds endpoint-mapped enterprise APIs, heterogeneous evidence, task-level control checks, exact answer fields, and full public oracle replays.
DistributionOpen-source framework and role benchmarksGitHub world, Hugging Face rows, Harbor tasks, and website explorer
Replay the work
Trajectories
Inspect the pinned model trajectory and one valid oracle solution from every workflow family, including exact tool arguments, discoveries, calculations, outcomes, state evidence, and verifier artifacts.
Sales needs a date for one pallet of Luma lamps. When can we honestly have it finished with today's schedule and stock, what is preventing an earlier run, and do expediting parts or adding a shift change the answer? Commit the best approved plan and prepare the launch-team update for review.
{"messages": {"matches": [{"channel": {"id": "C-PROCUREMENT"}, "text": "Account team needs one defensible decision for one pallet of Luma lamps by 2026-01-21; a faster or broader action has value only if it remains inside material readiness versus the protected assembly calendar.", "ts": "1768000001.000100", "user": "U-OPS-LEAD"}, {"channel": {"id": "C-PROCUREMENT"}, "text": "I found 480 EA on the header, but the workbook has separate observation and exclusion rows. Please do not net from this message.", "ts": "1768000001.000150", "user": "U-PLANNER"}, {"channel": {"id": "C-PROCUREMENT"}, "fi…
{"content": "# Business request and control — CASE-001\n\nSubject: one pallet of Luma lamps\nBusiness/control date: 2026-01-21\nRequester reference: NS-000001\nDecision boundary: material readiness versus the protected assembly calendar.\nEscalation boundary: add_weekend_shift requires a separate approval.\n\nThis record states the boundary only. It does not identify the recommended option or compute the supported outcome.\n", "description": "Business request and control", "id": "drive-001", "kind": "drive#file", "md5Checksum": "2bade8fa0dd928e2a447d2f8364bd34a96383a0b1ec4f2397ddda2f79d36ba9d…
Methodology
How the numbers are made
Each task has its own deterministic rubric for required discoveries, calculations, option analysis, approved state changes, exact answers, write containment, and valid tool use. The world remains synthetic, clean-room, and fully reproducible.
01
One metric, inspectable evidence
FactoryScore is the mean percentage of task-specific deterministic criterion weight passed. Discovery and final insights carry weight, calculations and the primary ERP decision carry more, and supporting collaboration writes and containment remain observable. Strict completion is supporting evidence, not a second metric.
02
The request does not reveal the recipe
Each prompt is a short employee request: decide the date, quantity, disposition, or commercial response and explain the constraint and alternatives. It contains no task ID, source list, filenames, answer schema, API names, or ordered steps. The agent must discover the case records and decide which evidence matters.
03
Facts are scattered, not preassembled
Each task mounts 28 retrievable, task-specific sources across Oracle resources, email, Drive, spreadsheets, Slack, vendor PDFs, specifications, inventory eligibility, finite-capacity slots, revisions, and approval limits. Current, superseded, other-case, and other-plant records must be correlated by immutable identity. No read returns an approved payload, operation name, or assembled answer. The clean-room company and all records are synthetic.
04
Qualification before publication
The release executes 1400 canonical trials: 100 reference workflows, 100 exact replays, and 1,200 adversarial runs across 12 controls. It also removes every reference mutation one at a time as a supplemental verifier test. Qualification passed: 100/100 oracle strict passes, 100/100 exact replay matches, and 1200/1,200 negative-control rejections.
05
Public inspirations and design boundary
Enterprise-Bench informed noisy multi-system task framing; ERP-Bench informed long-horizon manufacturing coverage; Mercor APEX informed the public task, asset, environment, and trajectory presentation. FactoryBench tasks, records, tools, and verifiers are independently authored and executable under this repository's licenses.
Qualification controls — excluded from the leaderboard
Reference oracle: 100.0% across 100 tasks — Measured solvability ceiling from replaying the checked-in reference workflow; not a model submission.
Shortcut control: 40.5% across 100 tasks — Measured control that submits a plausible answer and one late write without doing the investigation.
State-only control: 66.0% across 100 tasks — Measured control that reaches the reference provider state but omits the employee handoff.
Wrong-evidence control: 97.6% across 100 tasks — Measured control that substitutes a valid but stale or irrelevant source for one required current record.
Wrong-decision control: 94.0% across 100 tasks — Measured control that reaches the reference provider state but chooses a rejected operating alternative.
Run it yourself
The tasks, world, trajectories, and verifiers are public.
Use the live sandbox MCP, download the rows, rerun the Harbor tasks, and inspect the published model and oracle evidence call by call.
