Blobfish ResearchArc CRM 6 · v0.1.2Public release

Can an agent finish the CRM workflow without breaking the account?

Resolve the latest request. Check authority. Make only the supported change, preserve unrelated records, and verify what actually persisted. Six independently authored CRM workflows run across a CLI, real HTML forms, REST API and MCP tools sharing one isolated mock world.

Explore all six tasks Inspect reference trajectories Source adaptation record →

Run on Harbor ↗Dataset on Hugging Face ↗Executable world on GitHub ↗

6synthetic workflows

33tools per task

4executed interfaces

33evidence files · 4 formats

167negative controls rejected

One source, six new workflows—not 1,200 converted conversations.

We inspected all 1,200 conversations in the pinned Arc CRM source and authored these six synthetic workflows independently. Zero source rows are reproduced. This is partial source-inspired coverage, not upstream API parity or completion of the full Harbor and Hugging Face catalog.

The failed v0.1.0 registry run and v0.1.1’s truncated-PDF history remain preserved. This page describes the separately qualified v0.1.2 packages with complete native evidence. No paid model run or model leaderboard score is claimed.

Browse the classified source catalog →

Measured performance

Leaderboard

Model evaluations belong here only after a complete run on the exact published task set. Executed reference controls are shown separately below.

Ranking

Model leaderboard

Only complete, version-pinned model runs on this exact release receive a rank.

RankAgentCoverageScoreRun

No model has been evaluated on this release yet. The 6/6 reference-oracle passes below establish solvability, not model performance.

Qualification evidence

Measured controls

Oracle and adversarial replays prove solvability and verifier discrimination. They are measured, but never ranked as model submissions.

ClassControlCoverageScoreEvidence

Control

Local Docker reference oracleHarbor 0.21.0 · all six tasks · zero retries

6 tasks

100.0%

Solvability and isolation check, not model performance.Inspect run

Control

Registry Docker reference oracleHarbor 0.21.0 · all six tasks · zero retries

6 tasks

100.0%

Solvability and isolation check, not model performance.Inspect run

20%Evidence

15%Readback

30%Outcome

15%Containment

10%Handoff

10%Answer

6 distinct tasks

Task browser

All six independently authored workflows are inspectable: full prompts, evidence files, deterministic checks and reference executions.

Search tasksFilter by workflow6 shown

arc-crm-001Contact Onboarding

Onboard a buyer without duplicating the account

Finish the contact-onboarding thread for Aster Freight. Determine the verified existing client, create only the newly requested contact, and leave account records and existing contacts unchanged. Record the association in a client handoff.

Company: Aster Freight
As of: 2026-09-05
Seeded files: 4
Reference solution: 12 mixed-interface calls

Complete task prompt

59 graded criteria summarized

Finish the contact-onboarding thread for Aster Freight. Determine the verified existing client, create only the newly requested contact, and leave account records and existing contacts unchanged. Record the association in a client handoff.

Investigate the current request, operative policies, permissions and relevant records before any write. Earlier messages and expired policies are not authority. Preserve every unrelated record and financial field. Persist the requested handoff and read back every changed record plus that handoff before submitting the structured answer. The handoff is graded by its entity, record/evidence references and explicitly requested literal facts, not by exact prose or semantic judgment. All organizations, tools and assets are synthetic; no external action is authorized. This is a partial clean-room Arc CRM adaptation using the HubBench engine and its context/answer controls, not a released HubBench family.

In the handoff content include these literal facts or IDs, resolved from the evidence and persisted records: resolved existing client ID, new contact's verified email, current request ID. Set evidence_ids to the operative request and policy/approval/document asset IDs used for the action.

How the employee outcome is evaluated

Reasoning, persisted state, and the answer must agree.

8 semantic milestones

Strict success: Resolve current evidence and authority, execute only the requested business changes, preserve unrelated state, verify persisted records and submit the task-specific answer.
Ordering policy: Required evidence precedes relevant writes and readbacks follow them. Exact reference order and exact handoff prose are not required; handoff checks cover disclosed IDs and literal facts, not semantic consistency.

Inspect the task-specific causal milestones

Evidence
investigate-current_request
Resolve the owner's current clarification, not the superseded message.
Evidence
investigate-verified_client
Identify the existing verified client and the similarly named account.
Evidence
investigate-operative_policy
Read the operative policy and its scope.
Evidence
investigate-duplicate_contacts
Check existing contacts before creating another.
Evidence
investigate-authority
Read the acting role's specific operation/target grants before writing.
Readback
readback-contact
Read back the persisted contact after the business mutation.
Readback
readback-client_relationship
Read back the persisted client_relationship after the business mutation.
Readback
readback-handoff
Read back the persisted handoff after the business mutation.

Inspect all 59 deterministic criteria

Resolve the owner's current clarification, not the superseded message.
Identify the existing verified client and the similarly named account.
Read the operative policy and its scope.
Check existing contacts before creating another.
Read the acting role's specific operation/target grants before writing.
Read back the persisted contact after the business mutation.
Read back the persisted client_relationship after the business mutation.
Read back the persisted handoff after the business mutation.
Persist the required crm_contacts record and exact structural fields.
Persist the required crm_notes record and exact structural fields.
Preserve users record {'user_id': 'U-11'}, except the explicitly requested changes.
Preserve users record {'user_id': 'U-12'}, except the explicitly requested changes.
No missing or extra records in users.
Preserve evidence_files record {'asset_id': 'E-1-request-1'}, except the explicitly requested changes.
Preserve evidence_files record {'asset_id': 'E-1-request-2'}, except the explicitly requested changes.
Preserve evidence_files record {'asset_id': 'E-1-onboarding-1'}, except the explicitly requested changes.
Preserve evidence_files record {'asset_id': 'E-1-onboarding-2'}, except the explicitly requested changes.
No missing or extra records in evidence_files.
Preserve crm_evidence record {'asset_id': 'E-1-request-1'}, except the explicitly requested changes.
Preserve crm_evidence record {'asset_id': 'E-1-request-2'}, except the explicitly requested changes.
Preserve crm_evidence record {'asset_id': 'E-1-onboarding-1'}, except the explicitly requested changes.
Preserve crm_evidence record {'asset_id': 'E-1-onboarding-2'}, except the explicitly requested changes.
No missing or extra records in crm_evidence.
Preserve crm_clients record {'client_id': 'C-11'}, except the explicitly requested changes.
Preserve crm_clients record {'client_id': 'C-12'}, except the explicitly requested changes.
Preserve crm_clients record {'client_id': 'C-13'}, except the explicitly requested changes.
No missing or extra records in crm_clients.
Preserve crm_contacts record {'contact_id': 'CT-11'}, except the explicitly requested changes.
No missing or extra records in crm_contacts.
Preserve crm_opportunities record {'opportunity_id': 'OP-11'}, except the explicitly requested changes.
Preserve crm_opportunities record {'opportunity_id': 'OP-12'}, except the explicitly requested changes.
Preserve crm_opportunities record {'opportunity_id': 'OP-13'}, except the explicitly requested changes.
No missing or extra records in crm_opportunities.
No missing or extra records in crm_quotes.
No missing or extra records in crm_quote_lines.
No missing or extra records in crm_contracts.
No missing or extra records in crm_documents.
No missing or extra records in crm_attachments.
No missing or extra records in crm_notes.
Preserve crm_requests record {'request_id': 'RQ-11'}, except the explicitly requested changes.
Preserve crm_requests record {'request_id': 'RQ-12'}, except the explicitly requested changes.
No missing or extra records in crm_requests.
Preserve crm_policies record {'policy_id': 'POL-1-onboarding-1'}, except the explicitly requested changes.
Preserve crm_policies record {'policy_id': 'POL-1-onboarding-2'}, except the explicitly requested changes.
No missing or extra records in crm_policies.
No missing or extra records in crm_prices.
No missing or extra records in crm_approvals.
Preserve crm_permissions record {'permission_id': 'PERM-1-1'}, except the explicitly requested changes.
Preserve crm_permissions record {'permission_id': 'PERM-1-2'}, except the explicitly requested changes.
No missing or extra records in crm_permissions.
Exactly the required domain writes, with no hidden extra mutation.
Kept every successful write inside arc-crm-001's declared state, collaboration, answer, and audit tables.
Completed without a rejected state-changing call; failed exploratory reads and retryable provider faults do not erase a correct outcome.
Handoff content contains the disclosed literal facts; punctuation/case/prose may vary. Not a semantic consistency check.
Report the correct client id.
Report the correct contact id.
Report the correct email.
Report the correct source request id.
Report the correct note id.

Open pinned task on Hugging Face

Seeded evidence

Asset room

These are the exact synthetic PDF, XLSX, email and Markdown files in the immutable Hugging Face release, not placeholders.

Search assets

Owner request 1assetsSynthetic evidenceEML247 B

Owner request 1

File SHA256 5257d68ebdf1fae1c588a2632dca03d03b98499f26caa39c4099f0805747be80. Text-content SHA256 5257d68ebdf1fae1c588a2632dca03d03b98499f26caa39c4099f0805747be80. Aster Freight.Open released file

Owner request 2assetsSynthetic evidenceEML444 B

Owner request 2

File SHA256 ff9a31ea119254cc41ff1480aaabd700bc43b0cafe15647e0bdf2da6f9600a87. Text-content SHA256 ff9a31ea119254cc41ff1480aaabd700bc43b0cafe15647e0bdf2da6f9600a87. Aster Freight.Open released file

Onboarding policy revision 1assetsSynthetic evidencePDF706 B

Onboarding policy revision 1

File SHA256 3fbdc93ab9b4fbab0cca952054e22d66d18f17555a25bf1b9f9836f70840f5cf. Text-content SHA256 c7641ec3005adb6771fa01c11469d4595b2056ef9fbe5668a9e515b4730fac0d. Aster Freight.Open released file

Onboarding policy revision 2assetsSynthetic evidencePDF811 B

Onboarding policy revision 2

File SHA256 56669d39c00fee3b9e73bdf3be8aaced977f39e750316e13bd6524c41dc7fd32. Text-content SHA256 2a212435e1a9cd5ab0e3b8567fdb590749d22733ccb929711ff387bf1fb1c931. Aster Freight.Open released file

Runnable world

Environment and tool contract

Each episode exposes 31 domain/evidence tools plus two controls across CLI, HTML forms, REST and MCP. Below: 32 shared schemas and six task-specific answer schemas—38 listings, but 33 tools in each task. This website does not host an Arc sandbox.

MCP package pinblobfishai/arc-crm-6@0.1.2

Reviewed commit090d4028e485

Source dataset revision8efb61f5920812d03d6267ed25f4439acb82cfbf

Frozen package manifest SHA2561daae8b6c5f34f33e014600cea82339fd5b4507c090af8bf08da37727b86e6a3

HF publication commitb53b22d65fa4a82d22fc95b91326b381b1482c44

Harbor immutable registry identifier (opaque)sha256:fc9ca3b80d6a1a3c3e5bd1b975c54f1f3ada21b49668cb5f25e2912d67b69dfa

Independent client manifest digestsha256:9fa081af4ee114df2715a78b8e6ec7d7d69671455687540970c9611496198dd5

Reproduce all six reference tasksharbor run -d blobfishai/arc-crm-6@sha256:fc9ca3b80d6a1a3c3e5bd1b975c54f1f3ada21b49668cb5f25e2912d67b69dfa -a oracle -e docker -n 1 -k 1 -r 0

Shared across all six tasksarccrm.clients.search

arccrm

Read onlyIdempotentClosed sandbox

Literal case-insensitive search, all matches in deterministic ID order. Independently authored adaptation of observed client_search; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "query": {
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}

Shared across all six tasksarccrm.clients.get

arccrm

Read onlyIdempotentClosed sandbox

Read the persisted record by its immutable ID. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "client_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    }
  },
  "required": [
    "client_id"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.contacts.search

arccrm

Read onlyIdempotentClosed sandbox

Literal case-insensitive search, all matches in deterministic ID order. Independently authored adaptation of observed contact_search; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "query": {
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}

Shared across all six tasksarccrm.contacts.get

arccrm

Read onlyIdempotentClosed sandbox

Read the persisted record by its immutable ID. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "contact_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    }
  },
  "required": [
    "contact_id"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.opportunities.search

arccrm

Read onlyIdempotentClosed sandbox

Literal case-insensitive search, all matches in deterministic ID order. Independently authored adaptation of observed opportunity_search; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "query": {
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}

Shared across all six tasksarccrm.opportunities.get

arccrm

Read onlyIdempotentClosed sandbox

Read the persisted record by its immutable ID. Independently authored adaptation of observed view_opportunity_details, opportunity_details; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "opportunity_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    }
  },
  "required": [
    "opportunity_id"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.quotes.search

arccrm

Read onlyIdempotentClosed sandbox

Literal case-insensitive search, all matches in deterministic ID order. Independently authored adaptation of observed quote_search; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "query": {
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}

Shared across all six tasksarccrm.quotes.get

arccrm

Read onlyIdempotentClosed sandbox

Read the persisted record by its immutable ID. Independently authored adaptation of observed quote_details; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "quote_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    }
  },
  "required": [
    "quote_id"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.contracts.search

arccrm

Read onlyIdempotentClosed sandbox

Literal case-insensitive search, all matches in deterministic ID order. Independently authored adaptation of observed contract_search; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "query": {
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}

Shared across all six tasksarccrm.contracts.get

arccrm

Read onlyIdempotentClosed sandbox

Read the persisted record by its immutable ID. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "contract_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    }
  },
  "required": [
    "contract_id"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.documents.search

arccrm

Read onlyIdempotentClosed sandbox

Literal case-insensitive search, all matches in deterministic ID order. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "query": {
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}

Shared across all six tasksarccrm.documents.get

arccrm

Read onlyIdempotentClosed sandbox

Read the persisted record by its immutable ID. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "document_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    }
  },
  "required": [
    "document_id"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.attachments.search

arccrm

Read onlyIdempotentClosed sandbox

Literal case-insensitive search, all matches in deterministic ID order. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "query": {
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}

Shared across all six tasksarccrm.attachments.get

arccrm

Read onlyIdempotentClosed sandbox

Read the persisted record by its immutable ID. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "attachment_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    }
  },
  "required": [
    "attachment_id"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.notes.search

arccrm

Read onlyIdempotentClosed sandbox

Literal case-insensitive search, all matches in deterministic ID order. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "query": {
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}

Shared across all six tasksarccrm.notes.get

arccrm

Read onlyIdempotentClosed sandbox

Read the persisted record by its immutable ID. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "note_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    }
  },
  "required": [
    "note_id"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.contacts.create

arccrm

State changingNon-idempotentClosed sandbox

Create a contact on an authorized active existing client. Verified domain required; email is globally case-insensitive unique. Titles and account creation are not permitted. Independently authored adaptation of observed create_new_contact; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "client_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    },
    "email": {
      "minLength": 1,
      "type": "string"
    },
    "first_name": {
      "minLength": 1,
      "type": "string"
    },
    "last_name": {
      "minLength": 1,
      "type": "string"
    }
  },
  "required": [
    "client_id",
    "first_name",
    "last_name",
    "email"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.opportunities.update

arccrm

State changingNon-idempotentClosed sandbox

Update only stage/probability using the current revision and operative stage policy. All financial/owner fields remain unchanged. Independently authored adaptation of observed modify_opportunity; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "expected_revision": {
      "minimum": 1,
      "type": "integer"
    },
    "opportunity_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    },
    "probability": {
      "maximum": 100,
      "minimum": 0,
      "type": "integer"
    },
    "stage": {
      "enum": [
        "discovery",
        "qualified",
        "proposal",
        "negotiation",
        "won",
        "lost"
      ],
      "type": "string"
    }
  },
  "required": [
    "opportunity_id",
    "stage",
    "probability",
    "expected_revision"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.quotes.create

arccrm

State changingNon-idempotentClosed sandbox

Create only a draft with current SKU prices, bounded approval and integer minor-unit total. Invoice-level discount rounds half up. One open draft/approved quote per opportunity; cancelled predecessor is optional for replacements. Independently authored adaptation of observed create_quote; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "amount_minor": {
      "maximum": 1000000000000,
      "minimum": 0,
      "type": "integer"
    },
    "approval_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    },
    "currency": {
      "enum": [
        "USD",
        "EUR"
      ],
      "type": "string"
    },
    "discount_bps": {
      "maximum": 10000,
      "minimum": 0,
      "type": "integer"
    },
    "lines": {
      "items": {
        "additionalProperties": false,
        "properties": {
          "quantity": {
            "maximum": 10000,
            "minimum": 1,
            "type": "integer"
          },
          "sku": {
            "minLength": 1,
            "pattern": "^[A-Za-z0-9-]+$",
            "type": "string"
          }
        },
        "required": [
          "sku",
          "quantity"
        ],
        "type": "object"
      },
      "type": "array"
    },
    "opportunity_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    },
    "predecessor_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    },
    "status": {
      "enum": [
        "draft",
        "approved",
        "signed"
      ],
      "type": "string"
    },
    "valid_until": {
      "pattern": "^\\d{4}-\\d{2}-\\d{2}$",
      "type": "string"
    }
  },
  "required": [
    "opportunity_id",
    "amount_minor",
    "currency",
    "discount_bps",
    "approval_id",
    "valid_until",
    "status",
    "lines"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.quotes.cancel

arccrm

State changingNon-idempotentClosed sandbox

Cancel the authorized expected draft version. Signed/approved quotes and history cannot be deleted or edited. Independently authored adaptation of observed cancel_quote; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "expected_version": {
      "minimum": 1,
      "type": "integer"
    },
    "quote_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    }
  },
  "required": [
    "quote_id",
    "expected_version"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.quotes.compare

arccrm

Read onlyIdempotentClosed sandbox

Compare persisted same-currency quotes and the amount spread; comparison does not confer approval. Independently authored adaptation of observed compare_quotes; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "quote_ids": {
      "items": {
        "minLength": 1,
        "pattern": "^[A-Za-z0-9-]+$",
        "type": "string"
      },
      "type": "array"
    }
  },
  "required": [
    "quote_ids"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.contracts.create

arccrm

State changingNon-idempotentClosed sandbox

Create an unsigned draft from a current approved quote. Exact client/opportunity/currency/value links and a role limit are enforced. Never signs or activates a contract. Independently authored adaptation of observed create_contract; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "client_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    },
    "currency": {
      "enum": [
        "USD",
        "EUR"
      ],
      "type": "string"
    },
    "opportunity_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    },
    "quote_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    },
    "status": {
      "enum": [
        "draft",
        "signed",
        "active"
      ],
      "type": "string"
    },
    "value_minor": {
      "maximum": 1000000000000,
      "minimum": 0,
      "type": "integer"
    }
  },
  "required": [
    "client_id",
    "opportunity_id",
    "quote_id",
    "value_minor",
    "currency",
    "status"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.documents.attach

arccrm

State changingNon-idempotentClosed sandbox

Associate an approved seeded mock:// document with the authorized opportunity of the same client. Exact filename required. No URL or filesystem fetch ever occurs. Independently authored adaptation of observed upload_document; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "document_url": {
      "minLength": 1,
      "type": "string"
    },
    "entity_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    },
    "entity_type": {
      "enum": [
        "client",
        "opportunity",
        "quote"
      ],
      "type": "string"
    },
    "file_name": {
      "minLength": 1,
      "type": "string"
    }
  },
  "required": [
    "document_url",
    "entity_type",
    "entity_id",
    "file_name"
  ],
  "type": "object"
}

Shared across all six tasksarccrm.notes.create

arccrm

State changingNon-idempotentClosed sandbox

Persist a fact-grounded handoff on an authorized entity. References must exist on that entity; evidence IDs must exist. Literal facts and structured links are graded, not semantic prose quality. Independently authored adaptation of observed add_note; not upstream ABI parity.

{
  "additionalProperties": false,
  "properties": {
    "content": {
      "minLength": 1,
      "type": "string"
    },
    "entity_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    },
    "entity_type": {
      "enum": [
        "client",
        "opportunity"
      ],
      "type": "string"
    },
    "evidence_ids": {
      "items": {
        "minLength": 1,
        "pattern": "^[A-Za-z0-9-]+$",
        "type": "string"
      },
      "type": "array"
    },
    "reference_ids": {
      "items": {
        "minLength": 1,
        "pattern": "^[A-Za-z0-9-]+$",
        "type": "string"
      },
      "type": "array"
    }
  },
  "required": [
    "entity_type",
    "entity_id",
    "content",
    "reference_ids",
    "evidence_ids"
  ],
  "type": "object"
}

Shared across all six tasksdesk.requests.search

desk

Read onlyIdempotentClosed sandbox

Search the full conversation thread, including superseded requests. Resolve sequence/status before changing state. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "query": {
      "type": "string"
    }
  },
  "required": [],
  "type": "object"
}

Shared across all six tasksdesk.requests.get

desk

Read onlyIdempotentClosed sandbox

Read one request; later current messages supersede earlier conflicting messages. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "request_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    }
  },
  "required": [
    "request_id"
  ],
  "type": "object"
}

Shared across all six tasksdesk.policies.get

desk

Read onlyIdempotentClosed sandbox

Read a versioned policy and whether it is operative at the frozen task date. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "policy_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    }
  },
  "required": [
    "policy_id"
  ],
  "type": "object"
}

Shared across all six tasksdesk.permissions.list

desk

Read onlyIdempotentClosed sandbox

List the acting user's current and expired write grants. No grant means no permission. This does not select the requested business outcome. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {},
  "required": [],
  "type": "object"
}

Shared across all six tasksdesk.approvals.get

desk

Read onlyIdempotentClosed sandbox

Read immutable scoped pricing approval; check status, opportunity, currency and expiry. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "approval_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    }
  },
  "required": [
    "approval_id"
  ],
  "type": "object"
}

Shared across all six tasksdesk.prices.list

desk

Read onlyIdempotentClosed sandbox

Return all price revisions for a currency, including expired versions. Use the latest currently effective revision per SKU. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "currency": {
      "enum": [
        "USD",
        "EUR"
      ],
      "type": "string"
    }
  },
  "required": [
    "currency"
  ],
  "type": "object"
}

Shared across all six tasksvault.files.get

vault

Read onlyIdempotentClosed sandbox

Read text extracted from a seeded synthetic asset with its asset metadata. Binary release assets have separate file-byte hashes; no arbitrary paths or URLs are accepted. Blobfish-only supporting evidence/readback extension; not an observed upstream contract.

{
  "additionalProperties": false,
  "properties": {
    "asset_id": {
      "minLength": 1,
      "pattern": "^[A-Za-z0-9-]+$",
      "type": "string"
    }
  },
  "required": [
    "asset_id"
  ],
  "type": "object"
}

Shared across all six taskshubbench.context.get

hubbench

Read onlyIdempotentClosed sandbox

Return the isolated task scope: case reference, immutable record handles, mounted servers, identity, and the evidence index.

{
  "additionalProperties": false,
  "properties": {},
  "required": [],
  "type": "object"
}

arc-crm-001hubbench.submit_answer

hubbench

State changingIdempotentClosed sandbox

arc-crm-001 only. Record the structured decision for the current task. Every field of the task's answer contract is required; resubmission replaces earlier values.

{
  "additionalProperties": false,
  "properties": {
    "client_id": {
      "description": "client id",
      "type": "string"
    },
    "contact_id": {
      "description": "contact id",
      "type": "string"
    },
    "email": {
      "description": "email",
      "type": "string"
    },
    "note_id": {
      "description": "note id",
      "type": "string"
    },
    "source_request_id": {
      "description": "source request id",
      "type": "string"
    }
  },
  "required": [
    "client_id",
    "contact_id",
    "email",
    "source_request_id",
    "note_id"
  ],
  "type": "object"
}

arc-crm-002hubbench.submit_answer

hubbench

State changingIdempotentClosed sandbox

arc-crm-002 only. Record the structured decision for the current task. Every field of the task's answer contract is required; resubmission replaces earlier values.

{
  "additionalProperties": false,
  "properties": {
    "amount_minor": {
      "description": "amount minor",
      "type": "integer"
    },
    "note_id": {
      "description": "note id",
      "type": "string"
    },
    "opportunity_id": {
      "description": "opportunity id",
      "type": "string"
    },
    "probability": {
      "description": "probability",
      "type": "integer"
    },
    "source_request_id": {
      "description": "source request id",
      "type": "string"
    },
    "stage": {
      "description": "stage",
      "type": "string"
    }
  },
  "required": [
    "opportunity_id",
    "stage",
    "probability",
    "amount_minor",
    "source_request_id",
    "note_id"
  ],
  "type": "object"
}

arc-crm-003hubbench.submit_answer

hubbench

State changingIdempotentClosed sandbox

arc-crm-003 only. Record the structured decision for the current task. Every field of the task's answer contract is required; resubmission replaces earlier values.

{
  "additionalProperties": false,
  "properties": {
    "amount_minor": {
      "description": "amount minor",
      "type": "integer"
    },
    "gross_minor": {
      "description": "gross minor",
      "type": "integer"
    },
    "note_id": {
      "description": "note id",
      "type": "string"
    },
    "opportunity_id": {
      "description": "opportunity id",
      "type": "string"
    },
    "premium_vs_Q32_minor": {
      "description": "premium vs Q32 minor",
      "type": "integer"
    },
    "quote_id": {
      "description": "quote id",
      "type": "string"
    },
    "savings_vs_Q31_minor": {
      "description": "savings vs Q31 minor",
      "type": "integer"
    },
    "status": {
      "description": "status",
      "type": "string"
    }
  },
  "required": [
    "quote_id",
    "opportunity_id",
    "gross_minor",
    "amount_minor",
    "savings_vs_Q31_minor",
    "premium_vs_Q32_minor",
    "status",
    "note_id"
  ],
  "type": "object"
}

arc-crm-004hubbench.submit_answer

hubbench

State changingIdempotentClosed sandbox

arc-crm-004 only. Record the structured decision for the current task. Every field of the task's answer contract is required; resubmission replaces earlier values.

{
  "additionalProperties": false,
  "properties": {
    "amount_minor": {
      "description": "amount minor",
      "type": "integer"
    },
    "cancelled_quote_id": {
      "description": "cancelled quote id",
      "type": "string"
    },
    "gross_minor": {
      "description": "gross minor",
      "type": "integer"
    },
    "note_id": {
      "description": "note id",
      "type": "string"
    },
    "protected_quote_id": {
      "description": "protected quote id",
      "type": "string"
    },
    "replacement_quote_id": {
      "description": "replacement quote id",
      "type": "string"
    }
  },
  "required": [
    "cancelled_quote_id",
    "replacement_quote_id",
    "protected_quote_id",
    "amount_minor",
    "gross_minor",
    "note_id"
  ],
  "type": "object"
}

arc-crm-005hubbench.submit_answer

hubbench

State changingIdempotentClosed sandbox

arc-crm-005 only. Record the structured decision for the current task. Every field of the task's answer contract is required; resubmission replaces earlier values.

{
  "additionalProperties": false,
  "properties": {
    "client_id": {
      "description": "client id",
      "type": "string"
    },
    "contract_id": {
      "description": "contract id",
      "type": "string"
    },
    "note_id": {
      "description": "note id",
      "type": "string"
    },
    "opportunity_id": {
      "description": "opportunity id",
      "type": "string"
    },
    "quote_id": {
      "description": "quote id",
      "type": "string"
    },
    "status": {
      "description": "status",
      "type": "string"
    },
    "value_minor": {
      "description": "value minor",
      "type": "integer"
    }
  },
  "required": [
    "contract_id",
    "client_id",
    "opportunity_id",
    "quote_id",
    "value_minor",
    "status",
    "note_id"
  ],
  "type": "object"
}

arc-crm-006hubbench.submit_answer

hubbench

State changingIdempotentClosed sandbox

arc-crm-006 only. Record the structured decision for the current task. Every field of the task's answer contract is required; resubmission replaces earlier values.

{
  "additionalProperties": false,
  "properties": {
    "attachment_id": {
      "description": "attachment id",
      "type": "string"
    },
    "document_id": {
      "description": "document id",
      "type": "string"
    },
    "note_id": {
      "description": "note id",
      "type": "string"
    },
    "opportunity_id": {
      "description": "opportunity id",
      "type": "string"
    },
    "source_request_id": {
      "description": "source request id",
      "type": "string"
    }
  },
  "required": [
    "attachment_id",
    "document_id",
    "opportunity_id",
    "source_request_id",
    "note_id"
  ],
  "type": "object"
}

Replay the work

Trajectories

Actual all-six Harbor registry oracle executions, not model reasoning traces. Tool results are labeled with the interface used; every episode includes an HTML mutation. Open the pinned trace, surface receipt and verifier result to audit them.

AgentReference oracle — not a modelHarbor 0.21.0 · Docker registry round trip

Score100.0%Strict pass

Tool calls12arc-crm-001

Run cost—

14 events

Full transcript Verifier report Source trial artifact

Employee request
Finish the contact-onboarding thread for Aster Freight. Determine the verified existing client, create only the newly requested contact, and leave account records and existing contacts unchanged. Record the association in a client handoff. Investigate the current request, operative policies, permissions and relevant records before any write. Earlier messages and expired policies are not authority. Preserve every unrelated record and financial field. Persist the requested handoff and read back every changed record plus that handoff before submitting the structured answer. The handoff is graded by its entity, record/evidence references and explicitly requested literal facts, not by exact prose or semantic judgment. All organizations, tools and assets are synthetic; no external action is authorized. This is a partial clean-room Arc CRM adaptation using the HubBench engine and its context/answer controls, not a released HubBench family. In the handoff content include these literal facts or IDs, resolved from the evidence and persisted records: resolved existing client ID, new contact's verified email, current request ID. Set evidence_ids to the operative request and policy/approval/document asset IDs used for the action.
1
Evidencehubbench.context.getok
```
{}
```
REST execution { "evidence_index": [ { "asset_id": "E-1-onboarding-1", "kind": "synthetic_evidence", "media_type": "application/pdf", "path": "assets/E-1-onboarding-1.pdf", "sha256": "c7641ec3005adb6771fa01c11469d4595b2056ef9fbe5668a9e515b4730fac0d", "source": "Aster Freight", "title": "Onboarding policy revision 1" }, { "asset_id": "E-1-onboarding-2", "kind": "synthetic_evidence", "media_type": "application/pdf", "path": "assets/E-1-onboarding-2.pdf", "sha256": "2a212435e1a9cd5ab0e3b8567fdb590749d22733ccb929711ff387bf1fb1c931", "source": "Aster Freight", "title": "Onboarding policy revision 2" }, { "asset_id": "E-1-request-1", "kind": "synthetic_evidence", "media_type": "message/rfc822", "path": "assets/E-1-request-1.eml", "sha256": "5257d68ebdf1fae1c588a2632dca03d03b98499f26caa39c4099f0805747be80", "source": "Aster Freight", "title": "Owner request 1" }, { "asset_id": "E-1-request-2", "kind": "synthetic_evidence", "media_type": "message/rfc822", "path": "assets/E-1-request-2.eml", "sha256": "ff9a31ea119254cc41ff1480aaabd700bc43b0cafe15647e0bdf2da6f9600a87", "source": "Aster Freight", "title": "Owner request 2" } ], "identity": [ { "approval_limit_usd": 25000, "display_name": "Alex Rowan", "role": "sales_operations", "user_id": "U-11" }, { "approval_limit_usd": 100000, "display_name": "Morgan Ellis", "role": "commercial_owner", "user_id": "U-12" } ], "organization": { "name": "Aster Freight", "synthetic": true, "systems": [ "arccrm", "desk", "vault" ] }, "reference_records": {}, "starting_records": [ { "client_hint": "Aster Freight", "thread_id": "TH-1" } ], "state": { "network": "closed", "persistence": "episode-local SQLite", "scope": "isolated task snapshot" }, "task": { "as_of": "2026-09-05", "family": "arc-crm", "role": "U-11", "task_id": "arc-crm-001" }, "tool_servers": [ { "description": "Stateful synthetic CRM", "name": "arccrm", "tools": [ "arccrm.attachments.get", "arccrm.attachments.search", "arccrm.clients.get", "arccrm.clients.search", "arccrm.contacts.create", "arccrm.contacts.get", "arccrm.contacts.search", "arccrm.contracts.create", "arccrm.contracts.get", "arccrm.contracts.search", "arccrm.documents.attach", "arccrm.documents.get", "arccrm.documents.search", "arccrm.notes.create", "arccrm.notes.get", "arccrm.notes.search", "arccrm.opportunities.get", "arccrm.opportunities.search", "arccrm.opportunities.update", "arccrm.quotes.cancel", "arccrm.quotes.compare", "arccrm.quotes.create", "arccrm.quotes.get", "arccrm.quotes.search" ] }, { "description": "Mock requests, permissions, approvals and versioned policy/price records", "name": "desk", "tools": [ "desk.approvals.get", "desk.permissions.list", "desk.policies.get", "desk.prices.list", "desk.requests.get", "desk.requests.search" ] }, { "description": "Seeded synthetic evidence; no external file access", "name": "vault", "tools": [ "vault.files.get" ] }, { "description": "Benchmark-only discovery and structured answer submission controls.", "name": "hubbench", "tools": [ "hubbench.context.get", "hubbench.submit_answer" ] } ] }
2
Evidencedesk.requests.searchok
```
{
  "query": "TH-1"
}
```
CLI execution { "query": "th-1", "records": [ { "asset_id": "E-1-request-1", "body": "Please add Robin Vale at Aster Freight Services; create an account if needed.", "client_id": "C-11", "request_id": "RQ-11", "sender_id": "U-12", "sequence": 1, "status": "superseded", "subject": "Onboard a buyer without duplicating the account", "thread_id": "TH-1" }, { "asset_id": "E-1-request-2", "body": "Correction: Robin Vale is the buyer at Aster Freight, verified domain aster-freight.example, not the similarly named Services company. Email robin@aster-freight.example. Use the existing account; do not create an account or assign a title. The current domain policy applies.", "client_id": "C-11", "request_id": "RQ-12", "sender_id": "U-12", "sequence": 2, "status": "current", "subject": "Onboard a buyer without duplicating the account", "thread_id": "TH-1" } ], "total": 2 }
3
Evidencearccrm.clients.searchok
```
{
  "query": "Aster Freight"
}
```
MCP execution { "query": "aster freight", "records": [ { "client_id": "C-11", "domain": "aster-freight.example", "email": "billing@aster-freight.example", "name": "Aster Freight", "revision": 3, "status": "active" }, { "client_id": "C-12", "domain": "services-aster-freight.example", "email": "billing@services-aster-freight.example", "name": "Aster Freight Services", "revision": 2, "status": "active" }, { "client_id": "C-13", "domain": "archive-aster-freight.example", "email": "hold@archive-aster-freight.example", "name": "Aster Freight Archive", "revision": 1, "status": "suspended" } ], "total": 3 }
4
Evidencedesk.policies.getok
```
{
  "policy_id": "POL-1-onboarding-2"
}
```
WEB execution { "asset_id": "E-1-onboarding-2", "effective_from": "2026-09-01", "effective_until": "2026-12-31", "operative": true, "policy_id": "POL-1-onboarding-2", "revision": 2, "rules": { "duplicate_email": "case-insensitive reject", "email_domain": "verified client domain", "existing_client_required": true, "titles": "not authorized" }, "rules_json": "{\"duplicate_email\":\"case-insensitive reject\",\"email_domain\":\"verified client domain\",\"existing_client_required\":true,\"titles\":\"not authorized\"}", "topic": "onboarding" }
5
Evidencearccrm.contacts.searchok
```
{
  "query": "C-11"
}
```
REST execution { "query": "c-11", "records": [ { "client_id": "C-11", "contact_id": "CT-11", "email": "jamie@aster-freight.example", "first_name": "Jamie", "last_name": "Park", "revision": 1, "title": "Procurement lead" } ], "total": 1 }
6
Evidencedesk.permissions.listok
```
{}
```
CLI execution { "permissions": [ { "limit_minor": 2500000, "operation": "contact:create", "permission_id": "PERM-1-1", "target_id": "C-11", "user_id": "U-11", "valid_from": "2026-09-01", "valid_until": "2026-09-30" }, { "limit_minor": 2500000, "operation": "note:create", "permission_id": "PERM-1-2", "target_id": "C-11", "user_id": "U-11", "valid_from": "2026-09-01", "valid_until": "2026-09-30" } ], "user_id": "U-11" }
7
Actionarccrm.contacts.createok
```
{
  "client_id": "C-11",
  "email": "robin@aster-freight.example",
  "first_name": "Robin",
  "last_name": "Vale"
}
```
WEB execution { "client_id": "C-11", "contact_id": "CT-12", "email": "robin@aster-freight.example", "first_name": "Robin", "last_name": "Vale", "mutation_id": "arc-crm-001-mutation-01", "revision": 1, "title": "" }
8
Handoffarccrm.notes.createok
```
{
  "content": "Linked robin@aster-freight.example to verified C-11 per RQ-12; existing account retained.",
  "entity_id": "C-11",
  "entity_type": "client",
  "evidence_ids": [
    "E-1-request-2",
    "E-1-onboarding-2"
  ],
  "reference_ids": [
    "CT-12"
  ]
}
```
MCP execution { "content": "Linked robin@aster-freight.example to verified C-11 per RQ-12; existing account retained.", "entity_id": "C-11", "entity_type": "client", "evidence_ids_json": "[\"E-1-onboarding-2\",\"E-1-request-2\"]", "mutation_id": "arc-crm-001-mutation-02", "note_id": "NT-1", "reference_ids_json": "[\"CT-12\"]" }
9
Readbackarccrm.contacts.getok
```
{
  "contact_id": "CT-12"
}
```
REST execution { "client_id": "C-11", "contact_id": "CT-12", "email": "robin@aster-freight.example", "first_name": "Robin", "last_name": "Vale", "revision": 1, "title": "" }
10
Readbackarccrm.clients.getok
```
{
  "client_id": "C-11"
}
```
CLI execution { "client_id": "C-11", "domain": "aster-freight.example", "email": "billing@aster-freight.example", "name": "Aster Freight", "revision": 3, "status": "active" }
11
Readbackarccrm.notes.getok
```
{
  "note_id": "NT-1"
}
```
MCP execution { "content": "Linked robin@aster-freight.example to verified C-11 per RQ-12; existing account retained.", "entity_id": "C-11", "entity_type": "client", "evidence_ids_json": "[\"E-1-onboarding-2\",\"E-1-request-2\"]", "note_id": "NT-1", "reference_ids_json": "[\"CT-12\"]" }
12
Answerhubbench.submit_answerok
```
{
  "client_id": "C-11",
  "contact_id": "CT-12",
  "email": "robin@aster-freight.example",
  "note_id": "NT-1",
  "source_request_id": "RQ-12"
}
```
WEB execution { "accepted": true, "fields": { "client_id": "C-11", "contact_id": "CT-12", "email": "robin@aster-freight.example", "note_id": "NT-1", "source_request_id": "RQ-12" }, "task_id": "arc-crm-001" }
Verifier receipt
Separate networkless verifier: strict pass, HubScore 100; 59/59 atomic checks. Reference oracle, not a model result.

Methodology

How the numbers are made

Qualification, isolation, provenance and the limits of this partial adaptation.

Partial, source-specific coverage

Six independent synthetic workflows inspired by the pinned source; not 1,200 converted conversations or upstream API parity. All 1,200 source rows were inspected at an immutable revision. No upstream conversations, code or entities are reproduced; the six fixtures are independently authored. This release does not complete the Harbor/Hugging Face catalog.

Real execution, explicit control results

All six clean packages pass local Docker and digest-pinned registry oracle runs with strict HubScore 100. Admission independently regrades saved SQLite snapshots and checks exact task, trace, state, isolation and four-interface evidence. All 167 in-process negative controls reject. These are reference controls, not a model leaderboard.

One world across four interfaces

CLI commands, real HTML forms, REST and MCP mutate the same episode-local SQLite world. Every oracle uses all four, including an HTML mutation. There are 31 domain/evidence tools plus context and answer controls: 33 per task. The explorer lists 32 shared schemas and six task-specific answer schemas (38 listings). These are independent mock contracts, not upstream API/ABI parity. There is no hosted Arc sandbox on this website.

State, evidence, containment and handoff grading

Required current evidence and authority must precede business writes, and readbacks follow them. The verifier checks exact state, write counts, protected rows, entity associations, answer fields and disclosed handoff facts. It does not require the exact oracle call order or prose, does not semantically judge handoff consistency, and uses no LLM judge.

Native evidence completeness

The 33 evidence files comprise 15 PDFs, two XLSX workbooks, 12 EML emails and four Markdown files. The release PDF writer wraps long lines and adds pages instead of clipping source text. Independent extraction tests cover all 15 PDFs; workbook tests check every XLSX source cell. Frozen admission also checks exact full-text PDF rendering. File SHA256 binds the download bytes; text-content SHA256 binds the logical evidence returned by the API. Those are different for PDF and XLSX containers.

Separate agent, world and verifier

The UID10001 agent has only public client/schema/evidence. The world holds seed state and an episode-random private collection token. Harbor stops the agent, collects a serialized world snapshot, destroys the agent environment, then starts a fresh networkless verifier. Agent-writable logs are not trusted; forged rewards are cleared before grading.

Bounded local runtime

Harbor 0.21.0 and Docker: main 1 CPU/1 GiB/pids256; world 0.5 CPU/512 MiB/pids128; separate verifier 1 CPU/1 GiB/pids256. Zero GPUs or paid model calls. Timeouts: build 600s, agent 1200s, collection 40s, verifier 120s. The declared 2 GiB storage budget is not a Docker-enforced quota. Default cleanup removes only trial resources, not downloaded releases or evidence.

Immutable publication and failed-version history

Package 0.1.2 verifies content-addressed build inputs; its authoring contract remains 0.1.0. The v0.1.0 registry run failed. Version 0.1.1 passed runtime oracles but truncated text in nine native PDFs. Both histories remain preserved, not relabeled as this release. Registry collection identifiers are opaque; the separately computed client hash and every object hash remain inspectable. HF files are pinned to one exact commit. All evidence and organizations are synthetic; code and fixtures are Apache-2.0.

Qualification controls — excluded from the leaderboard

Local Docker reference oracle: 100.0% across 6 tasks — Solvability and isolation check, not model performance.
Registry Docker reference oracle: 100.0% across 6 tasks — Solvability and isolation check, not model performance.