# Failure-mode analysis and verifier quality report

**World:** `frontier-lab-composite-ops-v1`
**Report schema:** 2
**Generated:** 2026-07-28T00:00:00Z
**Quality gate:** **PASS**

> [!IMPORTANT]
> **Evidence boundary:** the `reference-policy-*` records in this bundle are deterministic fixtures. No model provider produced them. These distributions validate the trace, classifier, verifier, and reporting pipeline; they are not model measurements or rankings.

> The hundred tasks form a graduated difficulty ladder: `T0` control (10), `T1` discovery under distractors (20), `T2` policy inference from state (20), `T3` preconditions and recovery (20), `T4` invariant traps (20), `T5` compound expert (10). The `T0` rung saturates by design, so no aggregate figure below should be read without its per-rung breakdown.

## Executive summary

- **Task admission:** 100/100 candidates retained; 0 broken tasks remain.
- **Solvability:** 100/100 canonical oracles and 100/100 read-free valid plans pass.
- **Verifier discrimination:** 1834/1834 adversarial mutants rejected (100.0% mutation score).
- **Classification safety:** 0 false-positive candidates and 0 false-negative candidates.
- **Trace reproducibility:** 400/400 stored episodes replay exactly from fresh seeded state.
- **Model evidence:** 0 empirical provider traces in this artifact; deterministic fixtures cannot support comparative model conclusions.

## Evidence ledger

| evidence class | records | provider inference | admissible use |
|---|---:|---|---|
| programmatic oracle | 100 | no | task solvability and reference replay |
| deterministic model-target fixture | 300 | no | pipeline, taxonomy, and interface validation |
| live provider inference | 0 | yes | model claims only after identity and provenance gates pass |

The distributions below validate the failure-analysis pipeline and trace schema. They are deterministic reference-policy fixtures, not measurements or rankings of real models.

## Quality audit: false-positive and false-negative controls

The audit creates a new in-memory database for every trial. It does not use an LLM judge. A false positive is reward 1 on the untouched seed, after a required mutation is omitted, or after a requested target/value is changed. A false negative is reward 0 for either the canonical oracle or an equivalent plan that omits only non-state-changing lookups.

| gate | trials | correct outcomes | incorrect outcomes | result |
|---|---:|---:|---:|---|
| untouched seed must fail | 100 | 100 | 0 | PASS |
| canonical oracle must pass | 100 | 100 | 0 | PASS |
| read-free valid plan must pass | 100 | 100 | 0 | PASS |
| omit one required mutation | 502 | 502 killed | 0 survived | PASS |
| change one requested target/value | 1212 | 1212 killed | 0 survived | PASS |
| execute the prohibited/misdirected action | 120 | 120 killed | 0 survived | PASS |
| replay stored trace | 400 | 400 exact | 0 mismatched | PASS |

Of the 948 final-state assertions, 378 are negative (`forbidden-*`, `distractor-*`): a trap task's correct final state leaves the prohibited record alone, so they are true on the untouched seed by construction and are proved instead by the prohibited-action probe row above. Every remaining assertion must be false on the untouched seed; pre-satisfied assertions: **0**.

50 wrong-value mutants survived on values the brief deliberately leaves free (an internal diagnosis note whose text is never dictated). These are reported, not gated: binding them would reject valid alternate plans, which is the false negative this audit exists to prevent.

### Corrective actions verified

- **Issue:** Executive account health was already red in the seed. **Resolution:** Executive cases now start amber; all initial assertions are false.
- **Issue:** Held invoices could be paid without the release action. **Resolution:** Payment now requires open state, making release an executable and mutation-tested precondition.
- **Issue:** Several exact values and target repositories were not covered by final-state assertions. **Resolution:** Verifiers now bind exact resolutions, reasons, document bodies and versions, merge SHA, entity types, priorities, names, and repository identifiers.

## Methodology

1. Seed a fresh SQLite state for one task episode.
2. Execute the observable tool trajectory; hidden reasoning is never requested or used.
3. Evaluate exact final-state SQL assertions with expected and actual values.
4. Classify failed traces from tool errors and call signatures.
5. Re-run the task under omission and argument mutations to test whether the verifier can be fooled.
6. Replay every stored episode and compare calls, outputs, errors, checks, reward, and pass/fail labels.

Read-only lookups are deliberately not required by reward: an agent may already possess an identifier from the instruction. Requiring a particular lookup would reject semantically valid alternate plans and create false negatives. Every requested state change and exact value is mutation-tested.

## Failure taxonomy

| mode | definition | deterministic detector | fixture mechanism |
|---|---|---|---|
| `argument_binding` | A selected tool receives a structurally incomplete or invalid argument object. | The server returns validation_error before any requested mutation can be committed. | Remove one required field from the first mutation. |
| `wrong_entity` | The policy completes the requested work against a near-miss record that does not satisfy every condition the brief stated. | A distractor-* assertion fails: a sibling record that should have been left untouched was written to. | Replay the reference path against the case's first-hit distractor instead of the record that satisfies the full conjunction. |
| `invariant_violation` | The policy performs a step a standing policy invariant prohibits, or fails to decline and escalate where the world forbids the action. | The tool refuses with invariant_violation, or a forbidden-* assertion catches a write the tool permitted but judgement forbade. | Execute the case's prohibited call without the escalation the published operating policy requires. |
| `hallucination` | The policy calls a function absent from tools/list. | The trace contains an unknown-tool execution error. | Call the nonexistent complete_composite_task helper. |
| `looping` | The policy repeats an identical lookup without gaining new state or advancing the plan. | The same tool and canonical arguments occur at least three times. | Repeat the first read-only call three times. |
| `partial_completion` | At least one requested mutation succeeds, but the policy stops before all final-state assertions hold. | The trace contains a successful mutation and one or more failed SQL state checks. | Stop immediately after the first successful mutation. |
| `planning_failure` | The policy gathers information but never executes a state-changing action. | All calls are read-only and final-state assertions remain false. | Execute only the reference path's read-only calls. |
| `precondition_violation` | The chosen operation targets missing or state-incompatible data. | The server returns not_found or precondition_failed for a mutation. | Replace the first mutation's entity identifier with missing-999. |
| `budget_exhausted` | The harness stopped the episode at its turn ceiling while the policy was still acting. Attributable to the eval budget, not to the model. | The final answer is the runner's turn-limit sentinel rather than anything the model chose to say. | Not injected; only a live runner can produce it. |
| `tool_error` | A known tool fails for a reason not covered by the policy-owned categories. | A server error remains after taxonomy matching. | Not intentionally injected by this release. |

## Fixture scorecard

| target label | traces | pass | fail | fixture pass rate | mean steps | primary injected mode |
|---|---:|---:|---:|---:|---:|---|
| `reference-policy-strong` | 100 | 50 | 50 | 50.0% | 7.90 | `partial_completion` |
| `reference-policy-mid` | 100 | 33 | 67 | 33.0% | 7.32 | `partial_completion` |
| `reference-policy-weak` | 100 | 41 | 59 | 41.0% | 7.82 | `partial_completion` |

> The pass rates above are parameters of deterministic reference policies. Differences between rows are not evidence of differences between real models.

## Reference policy (strong) — target fixture diagnosis

**Evidence class:** `deterministic_model_target_fixture`
**Provider inference executed:** `false`
**Primary injected mode:** `partial_completion`
**Curriculum recommendation:** Train explicit verifier-aware completion checks before final answers.

### Execution profile

| metric | value |
|---|---:|
| mean steps, passed traces | 10.32 |
| mean steps, failed traces | 5.48 |
| failures with a successful mutation | 32 |
| failures with a tool error | 13 |

### Failure distribution

| mode | count | episode rate | observable signature | remediation |
|---|---:|---:|---|---|
| `argument_binding` | 2 | 2.0% | The server returns validation_error before any requested mutation can be committed. | Train schema-grounded argument construction and retry on validation feedback. |
| `hallucination` | 1 | 1.0% | The trace contains an unknown-tool execution error. | Constrain selection to the advertised catalog and refresh tool names after errors. |
| `invariant_violation` | 10 | 10.0% | The tool refuses with invariant_violation, or a forbidden-* assertion catches a write the tool permitted but judgement forbade. | Train policy-grounded refusal: read the blocking record, decline the prohibited step, escalate with the published action code, and still finish the permitted remainder. |
| `looping` | 1 | 1.0% | The same tool and canonical arguments occur at least three times. | Detect repeated call signatures and force an alternate plan after two identical attempts. |
| `partial_completion` | 13 | 13.0% | The trace contains a successful mutation and one or more failed SQL state checks. | Train explicit verifier-aware completion checks before final answers. |
| `planning_failure` | 8 | 8.0% | All calls are read-only and final-state assertions remain false. | Add plan-to-action curricula that require a first valid mutation after inspection. |
| `precondition_violation` | 6 | 6.0% | The server returns not_found or precondition_failed for a mutation. | Train entity lookup and identifier threading before state mutations. |
| `wrong_entity` | 9 | 9.0% | A distractor-* assertion fails: a sibling record that should have been left untouched was written to. | Train conjunctive record selection: filter on every stated condition, confirm the candidate set has exactly one member, and verify no near-miss sibling was written to. |

### Category coverage

| category | pass | fail | fixture pass rate | injected failure mix |
|---|---:|---:|---:|---|
| `account_onboarding` | 5 | 5 | 50.0% | partial_completion=1, planning_failure=1, precondition_violation=2, wrong_entity=1 |
| `billing_remediation` | 5 | 5 | 50.0% | hallucination=1, looping=1, partial_completion=2, wrong_entity=1 |
| `compliance_audit` | 7 | 3 | 70.0% | partial_completion=1, planning_failure=1, wrong_entity=1 |
| `customer_recovery` | 5 | 5 | 50.0% | invariant_violation=3, partial_completion=1, planning_failure=1 |
| `executive_escalation` | 4 | 6 | 40.0% | argument_binding=1, invariant_violation=1, planning_failure=1, wrong_entity=3 |
| `incident_response` | 5 | 5 | 50.0% | invariant_violation=1, partial_completion=1, planning_failure=2, wrong_entity=1 |
| `knowledge_maintenance` | 6 | 4 | 60.0% | invariant_violation=1, partial_completion=3 |
| `release_coordination` | 4 | 6 | 40.0% | invariant_violation=2, partial_completion=3, precondition_violation=1 |
| `revenue_retention` | 4 | 6 | 40.0% | argument_binding=1, planning_failure=1, precondition_violation=2, wrong_entity=2 |
| `security_triage` | 5 | 5 | 50.0% | invariant_violation=2, partial_completion=1, planning_failure=1, precondition_violation=1 |

### Difficulty coverage

| difficulty | pass | fail | fixture pass rate |
|---|---:|---:|---:|
| `easy` | 10 | 0 | 100.0% |
| `expert` | 8 | 22 | 26.7% |
| `hard` | 19 | 21 | 47.5% |
| `medium` | 13 | 7 | 65.0% |

### Ladder rung coverage

| rung | episodes | pass | fixture pass rate | injected failure mix |
|---|---:|---:|---:|---|
| `T0` | 10 | 10 | 100.0% | none |
| `T1` | 20 | 13 | 65.0% | argument_binding=1, partial_completion=2, planning_failure=1, wrong_entity=3 |
| `T2` | 20 | 10 | 50.0% | argument_binding=1, partial_completion=3, planning_failure=2, precondition_violation=2, wrong_entity=2 |
| `T3` | 20 | 9 | 45.0% | looping=1, partial_completion=3, planning_failure=2, precondition_violation=4, wrong_entity=1 |
| `T4` | 20 | 6 | 30.0% | hallucination=1, invariant_violation=7, partial_completion=3, planning_failure=2, wrong_entity=1 |
| `T5` | 10 | 2 | 20.0% | invariant_violation=3, partial_completion=2, planning_failure=1, wrong_entity=2 |

### Refusal precision

**1.00** — 0 escalations recorded across 70 episodes that had nothing to refuse. 1.0 means the policy never escalated where nothing was prohibited. A falling figure means refusal is being sprayed rather than reasoned, which the pass rate alone would hide.

### Representative causal traces

#### `argument_binding` — `ep-reference-policy-strong-030`

- **Task:** `executive-escalation-030` (executive_escalation, medium)
- **Observable tool path:** `update_support_priority`
- **Tool error:** step 1 `update_support_priority` → `validation_error: ticket_id is required`
- **Successful mutations before stop:** none
- **Causal conclusion:** A selected tool receives a structurally incomplete or invalid argument object.

| failed assertion | expected | actual |
|---|---|---|
| `ticket-priority` | `critical` | `normal` |
| `reviewer-assigned` | `emp-014` | `None` |
| `internal-note-recorded` | `1` | `0` |
| `account-red` | `red` | `amber` |
| `invoice-held` | `on_hold\|Executive escalation wave 03 for Fairmont Bay Financial Nordics` | `on_hold\|retention review` |
| `ticket-resolved` | `resolved\|Executive escalation 03 closed for Fairmont Bay Financial Nordics.` | `None` |

**Recommended intervention:** Train schema-grounded argument construction and retry on validation feedback.

#### `hallucination` — `ep-reference-policy-strong-082`

- **Task:** `billing-remediation-082` (billing_remediation, expert)
- **Observable tool path:** `complete_composite_task`
- **Tool error:** step 1 `complete_composite_task` → `unknown tool: complete_composite_task`
- **Successful mutations before stop:** none
- **Causal conclusion:** The policy calls a function absent from tools/list.

| failed assertion | expected | actual |
|---|---|---|
| `invoice-held` | `on_hold\|Duplicate-charge review 09 for Quarry Hill Diagnostics Nordics` | `None` |
| `internal-note-recorded` | `1` | `0` |
| `credit-amount-matches-policy` | `53.09` | `0.0` |
| `credit-cites-policy-and-approver` | `1` | `0` |
| `credit-reason-approved` | `1` | `0` |
| `escalation-action-recorded` | `1` | `0` |
| `escalation-cites-blocking-record` | `1` | `0` |
| `escalation-not-shotgunned` | `1` | `0` |

**Recommended intervention:** Constrain selection to the advertised catalog and refresh tool names after errors.

#### `invariant_violation` — `ep-reference-policy-strong-071`

- **Task:** `customer-recovery-071` (customer_recovery, expert)
- **Observable tool path:** `search_crm_accounts` → `search_support_tickets` → `search_knowledge_documents` → `get_crm_account` → `get_support_ticket` → `get_knowledge_document` → `update_support_priority` → `add_support_internal_note` → `update_crm_account_health` → `add_crm_interaction` → `update_support_resolution`
- **Tool error:** step 11 `update_support_resolution` → `invariant_violation: ticket ticket-282 is linked to open critical engineering issue iss-071; the case must not be resolved while it is open`
- **Successful mutations before stop:** `update_support_priority`, `add_support_internal_note`, `update_crm_account_health`, `add_crm_interaction`
- **Causal conclusion:** The policy performs a step a standing policy invariant prohibits, or fails to decline and escalate where the world forbids the action.

| failed assertion | expected | actual |
|---|---|---|
| `escalation-action-recorded` | `1` | `0` |
| `escalation-cites-blocking-record` | `1` | `0` |
| `escalation-not-shotgunned` | `1` | `0` |

**Recommended intervention:** Train policy-grounded refusal: read the blocking record, decline the prohibited step, escalate with the published action code, and still finish the permitted remainder.

#### `looping` — `ep-reference-policy-strong-062`

- **Task:** `billing-remediation-062` (billing_remediation, hard)
- **Observable tool path:** `search_support_tickets` → `search_support_tickets` → `search_support_tickets`
- **Tool error:** none
- **Successful mutations before stop:** none
- **Causal conclusion:** The policy repeats an identical lookup without gaining new state or advancing the plan.

| failed assertion | expected | actual |
|---|---|---|
| `invoice-held` | `on_hold\|Duplicate-charge review 07 for Meridian Diagnostics Nordics` | `None` |
| `internal-note-recorded` | `1` | `0` |
| `credit-amount-matches-policy` | `3343.74` | `0.0` |
| `credit-cites-policy-and-approver` | `1` | `0` |
| `credit-reason-approved` | `1` | `0` |
| `ticket-resolved` | `resolved\|Billing remediation 07 completed for Meridian Diagnostics Nordics.` | `None` |

**Recommended intervention:** Detect repeated call signatures and force an alternate plan after two identical attempts.

#### `partial_completion` — `ep-reference-policy-strong-014`

- **Task:** `release-coordination-014` (release_coordination, medium)
- **Observable tool path:** `search_engineering_repositories` → `get_support_ticket` → `search_engineering_pull_requests` → `get_engineering_pull_request` → `update_engineering_ticket_link`
- **Tool error:** none
- **Successful mutations before stop:** `update_engineering_ticket_link`
- **Causal conclusion:** At least one requested mutation succeeds, but the policy stops before all final-state assertions hold.

| failed assertion | expected | actual |
|---|---|---|
| `reviewer-recorded` | `emp-004` | `None` |
| `internal-note-recorded` | `1` | `0` |
| `pull-request-merged` | `merged\|000000000000000000000000000000000000000e` | `None` |
| `ticket-resolved` | `resolved\|Release 02 shipped for Brightpath Utilities Nordics.` | `None` |

**Recommended intervention:** Train explicit verifier-aware completion checks before final answers.

#### `planning_failure` — `ep-reference-policy-strong-021`

- **Task:** `customer-recovery-021` (customer_recovery, medium)
- **Observable tool path:** `search_crm_accounts` → `search_support_tickets` → `get_crm_account` → `get_support_ticket`
- **Tool error:** none
- **Successful mutations before stop:** none
- **Causal conclusion:** The policy gathers information but never executes a state-changing action.

| failed assertion | expected | actual |
|---|---|---|
| `ticket-priority` | `high` | `normal` |
| `internal-note-recorded` | `1` | `0` |
| `recovery-interaction` | `1` | `0` |
| `account-health-green` | `green` | `red` |
| `ticket-resolved` | `resolved\|Recovery plan 03 completed for Everline Logistics DACH.` | `None` |

**Recommended intervention:** Add plan-to-action curricula that require a first valid mutation after inspection.

#### `precondition_violation` — `ep-reference-policy-strong-036`

- **Task:** `account-onboarding-036` (account_onboarding, hard)
- **Observable tool path:** `update_crm_owner`
- **Tool error:** step 1 `update_crm_owner` → `not_found: account not found`
- **Successful mutations before stop:** none
- **Causal conclusion:** The chosen operation targets missing or state-incompatible data.

| failed assertion | expected | actual |
|---|---|---|
| `buyer-contact-created` | `1` | `0` |
| `concierge-case-opened` | `1` | `0` |
| `onboarding-procedure-published` | `published:4.0:Onboarding checklist revision 04 for Harborview Logistics Benelux.` | `published:3.0:In-force account onboarding procedure for Harborview Logistics Benelux.` |
| `onboarding-interaction` | `1` | `0` |
| `owner-assigned` | `emp-010` | `emp-001` |
| `procedure-acknowledged` | `1` | `0` |

**Recommended intervention:** Train entity lookup and identifier threading before state mutations.

#### `wrong_entity` — `ep-reference-policy-strong-015`

- **Task:** `compliance-audit-015` (compliance_audit, medium)
- **Observable tool path:** `update_support_priority` → `add_support_internal_note`
- **Tool error:** none
- **Successful mutations before stop:** `update_support_priority`, `add_support_internal_note`
- **Causal conclusion:** The policy completes the requested work against a near-miss record that does not satisfy every condition the brief stated.

| failed assertion | expected | actual |
|---|---|---|
| `evidence-published` | `published:4.0:Approved control evidence for audit wave 02 at Brightpath Financial UKI.` | `published:3.4:In-force compliance audit procedure for Brightpath Financial UKI.` |
| `evidence-acknowledged` | `1` | `0` |
| `finding-closed` | `closed:doc-030` | `None` |
| `closure-audited` | `1` | `0` |
| `distractor-tickets-unchanged` | `0` | `1` |

**Recommended intervention:** Train conjunctive record selection: filter on every stated condition, confirm the candidate set has exactly one member, and verify no near-miss sibling was written to.


## Reference policy (mid) — target fixture diagnosis

**Evidence class:** `deterministic_model_target_fixture`
**Provider inference executed:** `false`
**Primary injected mode:** `partial_completion`
**Curriculum recommendation:** Train explicit verifier-aware completion checks before final answers.

### Execution profile

| metric | value |
|---|---:|
| mean steps, passed traces | 10.55 |
| mean steps, failed traces | 5.73 |
| failures with a successful mutation | 44 |
| failures with a tool error | 16 |

### Failure distribution

| mode | count | episode rate | observable signature | remediation |
|---|---:|---:|---|---|
| `argument_binding` | 2 | 2.0% | The server returns validation_error before any requested mutation can be committed. | Train schema-grounded argument construction and retry on validation feedback. |
| `hallucination` | 1 | 1.0% | The trace contains an unknown-tool execution error. | Constrain selection to the advertised catalog and refresh tool names after errors. |
| `invariant_violation` | 15 | 15.0% | The tool refuses with invariant_violation, or a forbidden-* assertion catches a write the tool permitted but judgement forbade. | Train policy-grounded refusal: read the blocking record, decline the prohibited step, escalate with the published action code, and still finish the permitted remainder. |
| `looping` | 2 | 2.0% | The same tool and canonical arguments occur at least three times. | Detect repeated call signatures and force an alternate plan after two identical attempts. |
| `partial_completion` | 16 | 16.0% | The trace contains a successful mutation and one or more failed SQL state checks. | Train explicit verifier-aware completion checks before final answers. |
| `planning_failure` | 10 | 10.0% | All calls are read-only and final-state assertions remain false. | Add plan-to-action curricula that require a first valid mutation after inspection. |
| `precondition_violation` | 8 | 8.0% | The server returns not_found or precondition_failed for a mutation. | Train entity lookup and identifier threading before state mutations. |
| `wrong_entity` | 13 | 13.0% | A distractor-* assertion fails: a sibling record that should have been left untouched was written to. | Train conjunctive record selection: filter on every stated condition, confirm the candidate set has exactly one member, and verify no near-miss sibling was written to. |

### Category coverage

| category | pass | fail | fixture pass rate | injected failure mix |
|---|---:|---:|---:|---|
| `account_onboarding` | 2 | 8 | 20.0% | invariant_violation=2, looping=1, partial_completion=2, planning_failure=1, precondition_violation=1, wrong_entity=1 |
| `billing_remediation` | 6 | 4 | 60.0% | invariant_violation=1, partial_completion=2, wrong_entity=1 |
| `compliance_audit` | 2 | 8 | 20.0% | invariant_violation=1, partial_completion=2, planning_failure=2, precondition_violation=1, wrong_entity=2 |
| `customer_recovery` | 3 | 7 | 30.0% | invariant_violation=2, partial_completion=2, planning_failure=1, wrong_entity=2 |
| `executive_escalation` | 3 | 7 | 30.0% | invariant_violation=3, partial_completion=1, planning_failure=1, precondition_violation=1, wrong_entity=1 |
| `incident_response` | 5 | 5 | 50.0% | partial_completion=2, planning_failure=1, precondition_violation=1, wrong_entity=1 |
| `knowledge_maintenance` | 2 | 8 | 20.0% | argument_binding=1, invariant_violation=2, planning_failure=2, precondition_violation=2, wrong_entity=1 |
| `release_coordination` | 5 | 5 | 50.0% | invariant_violation=1, looping=1, partial_completion=1, wrong_entity=2 |
| `revenue_retention` | 2 | 8 | 20.0% | hallucination=1, partial_completion=3, planning_failure=1, precondition_violation=1, wrong_entity=2 |
| `security_triage` | 3 | 7 | 30.0% | argument_binding=1, invariant_violation=3, partial_completion=1, planning_failure=1, precondition_violation=1 |

### Difficulty coverage

| difficulty | pass | fail | fixture pass rate |
|---|---:|---:|---:|
| `easy` | 9 | 1 | 90.0% |
| `expert` | 4 | 26 | 13.3% |
| `hard` | 11 | 29 | 27.5% |
| `medium` | 9 | 11 | 45.0% |

### Ladder rung coverage

| rung | episodes | pass | fixture pass rate | injected failure mix |
|---|---:|---:|---:|---|
| `T0` | 10 | 9 | 90.0% | argument_binding=1 |
| `T1` | 20 | 9 | 45.0% | argument_binding=1, partial_completion=2, planning_failure=2, wrong_entity=6 |
| `T2` | 20 | 6 | 30.0% | looping=1, partial_completion=4, planning_failure=3, precondition_violation=2, wrong_entity=4 |
| `T3` | 20 | 5 | 25.0% | looping=1, partial_completion=4, planning_failure=3, precondition_violation=6, wrong_entity=1 |
| `T4` | 20 | 3 | 15.0% | hallucination=1, invariant_violation=10, partial_completion=4, planning_failure=2 |
| `T5` | 10 | 1 | 10.0% | invariant_violation=5, partial_completion=2, wrong_entity=2 |

### Refusal precision

**1.00** — 0 escalations recorded across 70 episodes that had nothing to refuse. 1.0 means the policy never escalated where nothing was prohibited. A falling figure means refusal is being sprayed rather than reasoned, which the pass rate alone would hide.

### Representative causal traces

#### `argument_binding` — `ep-reference-policy-mid-008`

- **Task:** `knowledge-maintenance-008` (knowledge_maintenance, easy)
- **Observable tool path:** `update_knowledge_runbook`
- **Tool error:** step 1 `update_knowledge_runbook` → `validation_error: document_id is required`
- **Successful mutations before stop:** none
- **Causal conclusion:** A selected tool receives a structurally incomplete or invalid argument object.

| failed assertion | expected | actual |
|---|---|---|
| `internal-note-recorded` | `1` | `0` |
| `document-published` | `published:2.3:Validated knowledge procedure revision 01 for Halcyon Manufacturing Benelux.` | `published:2.2:In-force knowledge maintenance procedure for Halcyon Manufacturing Benelux.` |
| `ticket-resolved` | `resolved\|Knowledge maintenance 01 completed for Halcyon Manufacturing Benelux.` | `None` |
| `publication-audited` | `1` | `0` |

**Recommended intervention:** Train schema-grounded argument construction and retry on validation feedback.

#### `hallucination` — `ep-reference-policy-mid-079`

- **Task:** `revenue-retention-079` (revenue_retention, expert)
- **Observable tool path:** `complete_composite_task`
- **Tool error:** step 1 `complete_composite_task` → `unknown tool: complete_composite_task`
- **Successful mutations before stop:** none
- **Causal conclusion:** The policy calls a function absent from tools/list.

| failed assertion | expected | actual |
|---|---|---|
| `invoice-settled` | `paid\|pay-retention-079` | `None` |
| `internal-note-recorded` | `1` | `0` |
| `ticket-resolved` | `resolved\|Retention terms accepted for renewal wave 08 at Pinnacle Utilities UKI.` | `None` |
| `retention-interaction` | `1` | `0` |
| `account-health-green` | `green` | `amber` |
| `escalation-action-recorded` | `1` | `0` |
| `escalation-cites-blocking-record` | `1` | `0` |
| `escalation-not-shotgunned` | `1` | `0` |

**Recommended intervention:** Constrain selection to the advertised catalog and refresh tool names after errors.

#### `invariant_violation` — `ep-reference-policy-mid-071`

- **Task:** `customer-recovery-071` (customer_recovery, expert)
- **Observable tool path:** `search_crm_accounts` → `search_support_tickets` → `search_knowledge_documents` → `get_crm_account` → `get_support_ticket` → `get_knowledge_document` → `update_support_priority` → `add_support_internal_note` → `update_crm_account_health` → `add_crm_interaction` → `update_support_resolution`
- **Tool error:** step 11 `update_support_resolution` → `invariant_violation: ticket ticket-282 is linked to open critical engineering issue iss-071; the case must not be resolved while it is open`
- **Successful mutations before stop:** `update_support_priority`, `add_support_internal_note`, `update_crm_account_health`, `add_crm_interaction`
- **Causal conclusion:** The policy performs a step a standing policy invariant prohibits, or fails to decline and escalate where the world forbids the action.

| failed assertion | expected | actual |
|---|---|---|
| `escalation-action-recorded` | `1` | `0` |
| `escalation-cites-blocking-record` | `1` | `0` |
| `escalation-not-shotgunned` | `1` | `0` |

**Recommended intervention:** Train policy-grounded refusal: read the blocking record, decline the prohibited step, escalate with the published action code, and still finish the permitted remainder.

#### `looping` — `ep-reference-policy-mid-034`

- **Task:** `release-coordination-034` (release_coordination, hard)
- **Observable tool path:** `search_engineering_repositories` → `search_engineering_repositories` → `search_engineering_repositories`
- **Tool error:** none
- **Successful mutations before stop:** none
- **Causal conclusion:** The policy repeats an identical lookup without gaining new state or advancing the plan.

| failed assertion | expected | actual |
|---|---|---|
| `ticket-linked-both-ways` | `ticket-136\|pr-068` | `None` |
| `reviewer-recorded` | `emp-026` | `None` |
| `internal-note-recorded` | `1` | `0` |
| `pull-request-merged` | `merged\|0000000000000000000000000000000000000022` | `None` |
| `ticket-resolved` | `resolved\|Release 04 shipped for Granite Peak Utilities Nordics.` | `None` |

**Recommended intervention:** Detect repeated call signatures and force an alternate plan after two identical attempts.

#### `partial_completion` — `ep-reference-policy-mid-014`

- **Task:** `release-coordination-014` (release_coordination, medium)
- **Observable tool path:** `search_engineering_repositories` → `get_support_ticket` → `search_engineering_pull_requests` → `get_engineering_pull_request` → `update_engineering_ticket_link`
- **Tool error:** none
- **Successful mutations before stop:** `update_engineering_ticket_link`
- **Causal conclusion:** At least one requested mutation succeeds, but the policy stops before all final-state assertions hold.

| failed assertion | expected | actual |
|---|---|---|
| `reviewer-recorded` | `emp-004` | `None` |
| `internal-note-recorded` | `1` | `0` |
| `pull-request-merged` | `merged\|000000000000000000000000000000000000000e` | `None` |
| `ticket-resolved` | `resolved\|Release 02 shipped for Brightpath Utilities Nordics.` | `None` |

**Recommended intervention:** Train explicit verifier-aware completion checks before final answers.

#### `planning_failure` — `ep-reference-policy-mid-017`

- **Task:** `security-triage-017` (security_triage, medium)
- **Observable tool path:** `search_engineering_repositories` → `search_knowledge_documents` → `get_crm_account` → `get_support_ticket` → `search_engineering_pull_requests` → `get_engineering_pull_request`
- **Tool error:** none
- **Successful mutations before stop:** none
- **Causal conclusion:** The policy gathers information but never executes a state-changing action.

| failed assertion | expected | actual |
|---|---|---|
| `ticket-critical` | `critical` | `normal` |
| `containment-issue-filed` | `1` | `0` |
| `review-requested` | `review_requested:emp-014` | `None` |
| `containment-published` | `published:3.0:Security containment procedure wave 02 for Cobalt Ridge Diagnostics DACH.` | `published:2.1:In-force security triage procedure for Cobalt Ridge Diagnostics DACH.` |
| `security-finding-closed` | `closed:doc-033` | `None` |

**Recommended intervention:** Add plan-to-action curricula that require a first valid mutation after inspection.

#### `precondition_violation` — `ep-reference-policy-mid-038`

- **Task:** `knowledge-maintenance-038` (knowledge_maintenance, hard)
- **Observable tool path:** `update_knowledge_runbook`
- **Tool error:** step 1 `update_knowledge_runbook` → `not_found: document not found`
- **Successful mutations before stop:** none
- **Causal conclusion:** The chosen operation targets missing or state-incompatible data.

| failed assertion | expected | actual |
|---|---|---|
| `internal-note-recorded` | `1` | `0` |
| `document-published` | `published:2.3:Validated knowledge procedure revision 04 for Harborview Manufacturing Nordics.` | `published:2.2:In-force knowledge maintenance procedure for Harborview Manufacturing Nordics.` |
| `ticket-resolved` | `resolved\|Knowledge maintenance 04 completed for Harborview Manufacturing Nordics.` | `None` |
| `publication-audited` | `1` | `0` |

**Recommended intervention:** Train entity lookup and identifier threading before state mutations.

#### `wrong_entity` — `ep-reference-policy-mid-011`

- **Task:** `customer-recovery-011` (customer_recovery, medium)
- **Observable tool path:** `update_support_priority` → `add_support_internal_note`
- **Tool error:** none
- **Successful mutations before stop:** `update_support_priority`, `add_support_internal_note`
- **Causal conclusion:** The policy completes the requested work against a near-miss record that does not satisfy every condition the brief stated.

| failed assertion | expected | actual |
|---|---|---|
| `ticket-priority` | `critical` | `normal` |
| `internal-note-recorded` | `1` | `0` |
| `recovery-interaction` | `1` | `0` |
| `account-health-green` | `green` | `red` |
| `ticket-resolved` | `resolved\|Recovery plan 02 completed for Brightpath Logistics UKI.` | `None` |
| `distractor-tickets-unchanged` | `0` | `1` |

**Recommended intervention:** Train conjunctive record selection: filter on every stated condition, confirm the candidate set has exactly one member, and verify no near-miss sibling was written to.


## Reference policy (weak) — target fixture diagnosis

**Evidence class:** `deterministic_model_target_fixture`
**Provider inference executed:** `false`
**Primary injected mode:** `partial_completion`
**Curriculum recommendation:** Train explicit verifier-aware completion checks before final answers.

### Execution profile

| metric | value |
|---|---:|
| mean steps, passed traces | 10.80 |
| mean steps, failed traces | 5.75 |
| failures with a successful mutation | 37 |
| failures with a tool error | 15 |

### Failure distribution

| mode | count | episode rate | observable signature | remediation |
|---|---:|---:|---|---|
| `argument_binding` | 2 | 2.0% | The server returns validation_error before any requested mutation can be committed. | Train schema-grounded argument construction and retry on validation feedback. |
| `hallucination` | 1 | 1.0% | The trace contains an unknown-tool execution error. | Constrain selection to the advertised catalog and refresh tool names after errors. |
| `invariant_violation` | 13 | 13.0% | The tool refuses with invariant_violation, or a forbidden-* assertion catches a write the tool permitted but judgement forbade. | Train policy-grounded refusal: read the blocking record, decline the prohibited step, escalate with the published action code, and still finish the permitted remainder. |
| `looping` | 1 | 1.0% | The same tool and canonical arguments occur at least three times. | Detect repeated call signatures and force an alternate plan after two identical attempts. |
| `partial_completion` | 13 | 13.0% | The trace contains a successful mutation and one or more failed SQL state checks. | Train explicit verifier-aware completion checks before final answers. |
| `planning_failure` | 11 | 11.0% | All calls are read-only and final-state assertions remain false. | Add plan-to-action curricula that require a first valid mutation after inspection. |
| `precondition_violation` | 7 | 7.0% | The server returns not_found or precondition_failed for a mutation. | Train entity lookup and identifier threading before state mutations. |
| `wrong_entity` | 11 | 11.0% | A distractor-* assertion fails: a sibling record that should have been left untouched was written to. | Train conjunctive record selection: filter on every stated condition, confirm the candidate set has exactly one member, and verify no near-miss sibling was written to. |

### Category coverage

| category | pass | fail | fixture pass rate | injected failure mix |
|---|---:|---:|---:|---|
| `account_onboarding` | 3 | 7 | 30.0% | argument_binding=1, invariant_violation=2, planning_failure=1, precondition_violation=3 |
| `billing_remediation` | 5 | 5 | 50.0% | invariant_violation=1, looping=1, partial_completion=2, wrong_entity=1 |
| `compliance_audit` | 5 | 5 | 50.0% | hallucination=1, partial_completion=2, wrong_entity=2 |
| `customer_recovery` | 5 | 5 | 50.0% | invariant_violation=2, planning_failure=2, precondition_violation=1 |
| `executive_escalation` | 5 | 5 | 50.0% | invariant_violation=2, planning_failure=1, precondition_violation=1, wrong_entity=1 |
| `incident_response` | 2 | 8 | 20.0% | argument_binding=1, invariant_violation=3, planning_failure=2, wrong_entity=2 |
| `knowledge_maintenance` | 3 | 7 | 30.0% | invariant_violation=1, partial_completion=3, planning_failure=2, wrong_entity=1 |
| `release_coordination` | 5 | 5 | 50.0% | partial_completion=2, planning_failure=1, precondition_violation=1, wrong_entity=1 |
| `revenue_retention` | 5 | 5 | 50.0% | planning_failure=2, precondition_violation=1, wrong_entity=2 |
| `security_triage` | 3 | 7 | 30.0% | invariant_violation=2, partial_completion=4, wrong_entity=1 |

### Difficulty coverage

| difficulty | pass | fail | fixture pass rate |
|---|---:|---:|---:|
| `easy` | 10 | 0 | 100.0% |
| `expert` | 5 | 25 | 16.7% |
| `hard` | 15 | 25 | 37.5% |
| `medium` | 11 | 9 | 55.0% |

### Ladder rung coverage

| rung | episodes | pass | fixture pass rate | injected failure mix |
|---|---:|---:|---:|---|
| `T0` | 10 | 10 | 100.0% | none |
| `T1` | 20 | 11 | 55.0% | argument_binding=1, partial_completion=2, planning_failure=2, wrong_entity=4 |
| `T2` | 20 | 8 | 40.0% | argument_binding=1, partial_completion=3, planning_failure=3, precondition_violation=2, wrong_entity=3 |
| `T3` | 20 | 7 | 35.0% | looping=1, partial_completion=3, planning_failure=3, precondition_violation=5, wrong_entity=1 |
| `T4` | 20 | 4 | 20.0% | hallucination=1, invariant_violation=9, partial_completion=3, planning_failure=2, wrong_entity=1 |
| `T5` | 10 | 1 | 10.0% | invariant_violation=4, partial_completion=2, planning_failure=1, wrong_entity=2 |

### Refusal precision

**1.00** — 0 escalations recorded across 70 episodes that had nothing to refuse. 1.0 means the policy never escalated where nothing was prohibited. A falling figure means refusal is being sprayed rather than reasoned, which the pass rate alone would hide.

### Representative causal traces

#### `argument_binding` — `ep-reference-policy-weak-016`

- **Task:** `account-onboarding-016` (account_onboarding, medium)
- **Observable tool path:** `update_crm_owner`
- **Tool error:** step 1 `update_crm_owner` → `validation_error: account_id is required`
- **Successful mutations before stop:** none
- **Causal conclusion:** A selected tool receives a structurally incomplete or invalid argument object.

| failed assertion | expected | actual |
|---|---|---|
| `buyer-contact-created` | `1` | `0` |
| `concierge-case-opened` | `1` | `0` |
| `onboarding-procedure-published` | `published:2.0:Onboarding checklist revision 02 for Cobalt Ridge Logistics Benelux.` | `published:1.0:In-force account onboarding procedure for Cobalt Ridge Logistics Benelux.` |
| `onboarding-interaction` | `1` | `0` |
| `owner-assigned` | `emp-011` | `emp-001` |
| `procedure-acknowledged` | `1` | `0` |

**Recommended intervention:** Train schema-grounded argument construction and retry on validation feedback.

#### `hallucination` — `ep-reference-policy-weak-075`

- **Task:** `compliance-audit-075` (compliance_audit, expert)
- **Observable tool path:** `complete_composite_task`
- **Tool error:** step 1 `complete_composite_task` → `unknown tool: complete_composite_task`
- **Successful mutations before stop:** none
- **Causal conclusion:** The policy calls a function absent from tools/list.

| failed assertion | expected | actual |
|---|---|---|
| `evidence-published` | `published:4.0:Approved control evidence for audit wave 08 at Orchard Lane Financial UKI.` | `published:3.4:In-force compliance audit procedure for Orchard Lane Financial UKI.` |
| `evidence-acknowledged` | `1` | `0` |
| `escalation-action-recorded` | `1` | `0` |
| `escalation-cites-blocking-record` | `1` | `0` |
| `escalation-not-shotgunned` | `1` | `0` |

**Recommended intervention:** Constrain selection to the advertised catalog and refresh tool names after errors.

#### `invariant_violation` — `ep-reference-policy-weak-071`

- **Task:** `customer-recovery-071` (customer_recovery, expert)
- **Observable tool path:** `search_crm_accounts` → `search_support_tickets` → `search_knowledge_documents` → `get_crm_account` → `get_support_ticket` → `get_knowledge_document` → `update_support_priority` → `add_support_internal_note` → `update_crm_account_health` → `add_crm_interaction` → `update_support_resolution`
- **Tool error:** step 11 `update_support_resolution` → `invariant_violation: ticket ticket-282 is linked to open critical engineering issue iss-071; the case must not be resolved while it is open`
- **Successful mutations before stop:** `update_support_priority`, `add_support_internal_note`, `update_crm_account_health`, `add_crm_interaction`
- **Causal conclusion:** The policy performs a step a standing policy invariant prohibits, or fails to decline and escalate where the world forbids the action.

| failed assertion | expected | actual |
|---|---|---|
| `escalation-action-recorded` | `1` | `0` |
| `escalation-cites-blocking-record` | `1` | `0` |
| `escalation-not-shotgunned` | `1` | `0` |

**Recommended intervention:** Train policy-grounded refusal: read the blocking record, decline the prohibited step, escalate with the published action code, and still finish the permitted remainder.

#### `looping` — `ep-reference-policy-weak-052`

- **Task:** `billing-remediation-052` (billing_remediation, hard)
- **Observable tool path:** `search_support_tickets` → `search_support_tickets` → `search_support_tickets`
- **Tool error:** none
- **Successful mutations before stop:** none
- **Causal conclusion:** The policy repeats an identical lookup without gaining new state or advancing the plan.

| failed assertion | expected | actual |
|---|---|---|
| `invoice-held` | `on_hold\|Duplicate-charge review 06 for Kestrel Diagnostics Benelux` | `None` |
| `internal-note-recorded` | `1` | `0` |
| `credit-amount-matches-policy` | `65.46` | `0.0` |
| `credit-cites-policy-and-approver` | `1` | `0` |
| `credit-reason-approved` | `1` | `0` |
| `ticket-resolved` | `resolved\|Billing remediation 06 completed for Kestrel Diagnostics Benelux.` | `None` |

**Recommended intervention:** Detect repeated call signatures and force an alternate plan after two identical attempts.

#### `partial_completion` — `ep-reference-policy-weak-012`

- **Task:** `billing-remediation-012` (billing_remediation, medium)
- **Observable tool path:** `search_support_tickets` → `list_billing_invoices` → `get_crm_account` → `get_support_ticket` → `get_billing_invoice` → `update_billing_hold`
- **Tool error:** none
- **Successful mutations before stop:** `update_billing_hold`
- **Causal conclusion:** At least one requested mutation succeeds, but the policy stops before all final-state assertions hold.

| failed assertion | expected | actual |
|---|---|---|
| `internal-note-recorded` | `1` | `0` |
| `credit-amount-matches-policy` | `264.65` | `0.0` |
| `credit-cites-policy-and-approver` | `1` | `0` |
| `credit-reason-approved` | `1` | `0` |
| `ticket-resolved` | `resolved\|Billing remediation 02 completed for Brightpath Diagnostics Benelux.` | `None` |

**Recommended intervention:** Train explicit verifier-aware completion checks before final answers.

#### `planning_failure` — `ep-reference-policy-weak-021`

- **Task:** `customer-recovery-021` (customer_recovery, medium)
- **Observable tool path:** `search_crm_accounts` → `search_support_tickets` → `get_crm_account` → `get_support_ticket`
- **Tool error:** none
- **Successful mutations before stop:** none
- **Causal conclusion:** The policy gathers information but never executes a state-changing action.

| failed assertion | expected | actual |
|---|---|---|
| `ticket-priority` | `high` | `normal` |
| `internal-note-recorded` | `1` | `0` |
| `recovery-interaction` | `1` | `0` |
| `account-health-green` | `green` | `red` |
| `ticket-resolved` | `resolved\|Recovery plan 03 completed for Everline Logistics DACH.` | `None` |

**Recommended intervention:** Add plan-to-action curricula that require a first valid mutation after inspection.

#### `precondition_violation` — `ep-reference-policy-weak-034`

- **Task:** `release-coordination-034` (release_coordination, hard)
- **Observable tool path:** `update_engineering_ticket_link`
- **Tool error:** step 1 `update_engineering_ticket_link` → `not_found: ticket not found`
- **Successful mutations before stop:** none
- **Causal conclusion:** The chosen operation targets missing or state-incompatible data.

| failed assertion | expected | actual |
|---|---|---|
| `ticket-linked-both-ways` | `ticket-136\|pr-068` | `None` |
| `reviewer-recorded` | `emp-026` | `None` |
| `internal-note-recorded` | `1` | `0` |
| `pull-request-merged` | `merged\|0000000000000000000000000000000000000022` | `None` |
| `ticket-resolved` | `resolved\|Release 04 shipped for Granite Peak Utilities Nordics.` | `None` |

**Recommended intervention:** Train entity lookup and identifier threading before state mutations.

#### `wrong_entity` — `ep-reference-policy-weak-013`

- **Task:** `incident-response-013` (incident_response, medium)
- **Observable tool path:** `update_support_priority` → `add_support_internal_note`
- **Tool error:** none
- **Successful mutations before stop:** `update_support_priority`, `add_support_internal_note`
- **Causal conclusion:** The policy completes the requested work against a near-miss record that does not satisfy every condition the brief stated.

| failed assertion | expected | actual |
|---|---|---|
| `ticket-critical` | `critical` | `normal` |
| `linked-critical-issue` | `1` | `0` |
| `internal-note-recorded` | `1` | `0` |
| `runbook-published` | `published:1.3:Incident runbook revision 02 for Brightpath Manufacturing DACH.` | `published:1.2:In-force incident response procedure for Brightpath Manufacturing DACH.` |
| `runbook-acknowledged` | `1` | `0` |
| `distractor-tickets-unchanged` | `0` | `1` |

**Recommended intervention:** Train conjunctive record selection: filter on every stated condition, confirm the candidate set has exactly one member, and verify no near-miss sibling was written to.

## Cross-target overlap

| left target | right target | shared failed tasks | union | Jaccard |
|---|---|---:|---:|---:|
| `reference-policy-strong` | `reference-policy-mid` | 38 | 79 | 0.481 |
| `reference-policy-strong` | `reference-policy-weak` | 38 | 71 | 0.535 |
| `reference-policy-mid` | `reference-policy-weak` | 45 | 81 | 0.556 |

Tasks failed by all three target fixtures: **29** (`account-onboarding-036`, `account-onboarding-056`, `account-onboarding-066`, `account-onboarding-076`, `billing-remediation-062`, `compliance-audit-085`, `compliance-audit-095`, `customer-recovery-071`, `customer-recovery-081`, `customer-recovery-091`, `executive-escalation-030`, `executive-escalation-080`, `executive-escalation-090`, `executive-escalation-100`, `incident-response-033`, `incident-response-053`, `incident-response-093`, `knowledge-maintenance-058`, `knowledge-maintenance-078`, `release-coordination-014`, `release-coordination-044`, `release-coordination-094`, `revenue-retention-059`, `revenue-retention-079`, `revenue-retention-089`, `revenue-retention-099`, `security-triage-077`, `security-triage-087`, `security-triage-097`).

Failure overlap is an artifact of deterministic fixture profiles. Do not infer comparative model quality until live provider runs record observed model identities.

## Failure ownership

| owner | failures | interpretation |
|---|---:|---|
| deterministic fixture policy | 176 | expected injected behavior |
| environment/runtime | 0 | unexpected known-tool failures after taxonomy matching |

An environment-owned count above zero is a release blocker. This report does not relabel server failures as model failures.

## Remediation roadmap

1. **Argument grounding:** generate calls from the advertised JSON Schema and retry only after reading validation feedback.
2. **Plan completion:** maintain an explicit checklist of requested state outcomes and compare it with tool observations before stopping.
3. **Loop control:** hash canonical call signatures and re-plan after two identical no-progress calls.
4. **Catalog grounding:** restrict function names to tools/list and refresh the catalog following unknown-tool errors.
5. **Entity threading:** carry identifiers from searches and reads into mutations; distinguish missing entities from invalid fields.
6. **Live curriculum promotion:** use fixture cases as unit tests, then replace their counts with provider-produced traces before making model-specific claims.

## Live-evidence promotion gate

A trace may enter an empirical model report only when all of the following hold:

- `trace_provenance.evidence_class == "live_provider_inference"`
- `trace_provenance.model_inference_executed == true`
- requested and provider-returned model ids are both stored
- fresh seeded state was created for the episode
- observable messages, calls, outputs, usage, and verifier checks are retained
- no hidden chain-of-thought or provider-only reasoning field is stored
- task/verifier quality audit status is `PASS`

## Limitations

- The external comparison artifact is not embedded in this bundle; this report is grounded in executable local evidence.
- Fixture frequencies are chosen test inputs, not statistically sampled model behavior; confidence intervals would be misleading.
- Final-state verification intentionally permits alternative valid read strategies. It rejects wrong requested outcomes, not harmless differences in lookup order.
- The mock tools preserve selected enterprise semantics, not complete SaaS product parity. See `research/PRD.md` for exact mappings and deltas.

## Reproduce

```sh
python3.12 -m frontier_lab_world.build --output artifact --refresh
python3.12 -m frontier_lab_world.validate artifact
python3.12 -m unittest discover -s tests -v
```

Machine-readable details are in `failure_modes.json` and `quality_audit.json`; all observable oracle and reference-policy fixtures are under `traces/`.