Skip to benchmark
Blobfish AI
BenchmarksTalk to usOpen Studio
Blobfish ResearchSemiKongBench-100 v1.0.0Qualified release

Can an agent contain the semiconductor exception—and prove the fab is safe?

SemiKongBench-100 turns public semiconductor workflow concepts into executable work. The agent must distinguish current from retired authority, reconcile MES and evidence-room records, calculate the exception, make a safe disposition, mutate the correct fab system, and verify every write.

Inspect qualification and leaderboard Browse all 100 tasks
Run on Harbor ↗Dataset on Hugging Face ↗Executable source ↗
100fab workflows
10synthetic fabs
38stateful tools
380released source assets
0LLM grading calls
Release gate passed

The tasks execute; diagnostic controls are not model rows. All 100 oracle episodes reached 100 SemiOpsScore, all 100/100 replays matched exactly, and 600 adversarial episodes—including unsafe release attempts—produced 0 strict false accepts.

Assets per task
20
Evidence reads
12
Exact criteria
26
Native formats
8
SemiKong-grounded · independently authored

The public SemiKong paper and checked-in MIT ontology informed the semiconductor taxonomy. We did not copy its model weights, training corpus, or evaluation questions. Every task, value, file, tool, answer, and trajectory here is newly synthetic, and this is not an official or endorsed SemiKong scorecard.

SemiKong repository ↗SemiKong paper ↗Public ontology ↗

Measured performance

Leaderboard

SemiOpsScore is the mean deterministic outcome score across all 100 tasks. Qualification controls stay outside the ranked table, so an oracle or attack can never masquerade as a model result.

Ranking

Model leaderboard

Only complete, version-pinned model runs on this exact release receive a rank.

RankAgentCoverageScoreRun
No complete version-pinned model run is published for SemiKongBench-100 yet. The executed controls establish solvability and six distinct failure floors without receiving ranks.
Qualification evidence

Measured controls

Oracle and adversarial replays prove solvability and verifier discrimination. They are measured, but never ranked as model submissions.

ClassControlCoverageScoreEvidence
REF
Reference oracleDeterministic 800-episode release qualification
100 tasks
100.0%
21 calls avg.Solvability ceiling; not a model submission.
CTRL
No-op controlDeterministic 800-episode release qualification
100 tasks
9.0%
calls avg.Executed adversarial diagnostic; never ranked with models.
CTRL
Answer-only shortcutDeterministic 800-episode release qualification
100 tasks
44.0%
calls avg.Executed adversarial diagnostic; never ranked with models.
CTRL
State-only shortcutDeterministic 800-episode release qualification
100 tasks
88.0%
calls avg.Executed adversarial diagnostic; never ranked with models.
CTRL
Stale-source controlDeterministic 800-episode release qualification
100 tasks
85.0%
calls avg.Executed adversarial diagnostic; never ranked with models.
CTRL
Wrong-target controlDeterministic 800-episode release qualification
100 tasks
71.0%
calls avg.Executed adversarial diagnostic; never ranked with models.
CTRL
Unsafe-release controlDeterministic 800-episode release qualification
100 tasks
47.0%
calls avg.Executed adversarial diagnostic; never ranked with models.
12%Evidence discovery

Operative records are read before any controlled write.

20%Process analysis

Exact process, quality, and traceability values are derived from current evidence.

10%Disposition decision

The supported semiconductor containment or rework path is selected.

30%Committed operations state

The domain record, disposition, and review handoff are durably changed.

20%Post-write verification

Every required mutation and final answer is read back.

8%Scope and safety

Unrelated fab state is preserved and no unsafe release bypass succeeds.

100 distinct tasks

Task browser

Open any workflow to inspect the high-level shift request, 20-file evidence room, three disposition alternatives, and all 26 deterministic checks.

100 shown
semikongbench-001Lot genealogy

Reconcile a split-lot count mismatch

Aurora Silicon (Hillsboro, 5 nm) has escalated AU5-260218-17 for lot genealogy. The shift record points to a possible genealogy count mismatch while an older control revision is still present in the evidence room. Establish the operative authority, reconcile the lot, process, metrology, SPC, and workflow-specific records, calculate the primary and corroborating exception values, and choose the safe operating disposition. If the current evidence supports action, update only the authorized semiconductor record, record the disposition, and prepare a review-only handoff for the shift lead. Leave unrelated fabs, lots, equipment, materials, and evidence unchanged; do not claim completion until every mutation and the structured answer have been read back.

Company
FAB-AURORA
As of
2026-02-18T14:00:00Z
Seeded files
20
Reference solution
21 MCP calls

Complete task prompt

26 graded criteria summarized
Aurora Silicon (Hillsboro, 5 nm) has escalated AU5-260218-17 for lot genealogy. The shift record points to a possible genealogy count mismatch while an older control revision is still present in the evidence room. Establish the operative authority, reconcile the lot, process, metrology, SPC, and workflow-specific records, calculate the primary and corroborating exception values, and choose the safe operating disposition. If the current evidence supports action, update only the authorized semiconductor record, record the disposition, and prepare a review-only handoff for the shift lead. Leave unrelated fabs, lots, equipment, materials, and evidence unchanged; do not claim completion until every mutation and the structured answer have been read back.
How the employee outcome is evaluated

Reasoning, persisted state, and the answer must agree.

26 semantic milestones

A current-authority investigation followed by a semiconductor-specific safe-state transition, disposition, handoff, and durable verification.

Strict success
Operative evidence, exact exception values, the safe decision, committed state, review handoff, and all readbacks agree.
Ordering policy
Exact total call order is not graded; all required investigations must precede the first controlled write and required writes must be read back.
Inspect the task-specific causal milestones
  1. discovery
    discovery:task_contract

    Read task contract before the first controlled write.

  2. discovery
    discovery:authority_search

    Read authority search before the first controlled write.

  3. discovery
    discovery:current_authority

    Read current authority before the first controlled write.

  4. discovery
    discovery:superseded_authority

    Read superseded authority before the first controlled write.

  5. discovery
    discovery:authority_history

    Read authority history before the first controlled write.

  6. discovery
    discovery:lot_record

    Read lot record before the first controlled write.

  7. discovery
    discovery:genealogy

    Read genealogy before the first controlled write.

  8. discovery
    discovery:process_history

    Read process history before the first controlled write.

  9. discovery
    discovery:metrology

    Read metrology before the first controlled write.

  10. discovery
    discovery:spc_rule

    Read spc rule before the first controlled write.

  11. discovery
    discovery:control_message

    Read control message before the first controlled write.

  12. discovery
    discovery:workflow_record

    Read workflow record before the first controlled write.

  13. analysis
    analysis:primary_metric

    Return the exact primary exception metric.

  14. analysis
    analysis:secondary_metric

    Return the exact corroborating metric.

  15. analysis
    analysis:risk_code

    Name the supported risk classification.

  16. analysis
    analysis:source_revision

    Bind the answer to the operative control revision.

  17. decision
    decision:decision

    Choose the supported disposition path.

  18. decision
    decision:target_status

    Choose the exact safe target status.

  19. execution
    state:domain_record

    Commit the workflow-specific domain state.

  20. execution
    state:disposition

    Record the lot disposition with source lineage.

  21. execution
    state:handoff

    Create a review-only operations handoff.

  22. readback
    readback:domain

    Read back the workflow-specific mutation.

  23. readback
    readback:disposition

    Read back the disposition record.

  24. readback
    readback:handoff

    Read back the review handoff.

  25. readback
    readback:submission

    Read back the durable structured answer.

  26. containment
    containment:scope

    Preserve immutable evidence and unrelated fab records; all calls must succeed.

Decision space (3 grounded options)
  • Hold And Reconcile Genealogy — selected: Selected by current controlled evidence.
  • Release without action: Unsafe because the exception is unresolved.
  • Use superseded revision: Rejected because its authority is retired.
Inspect all 26 deterministic criteria
  • discovery: Read task contract before the first controlled write. (1 pts)
  • discovery: Read authority search before the first controlled write. (1 pts)
  • discovery: Read current authority before the first controlled write. (1 pts)
  • discovery: Read superseded authority before the first controlled write. (1 pts)
  • discovery: Read authority history before the first controlled write. (1 pts)
  • discovery: Read lot record before the first controlled write. (1 pts)
  • discovery: Read genealogy before the first controlled write. (1 pts)
  • discovery: Read process history before the first controlled write. (1 pts)
  • discovery: Read metrology before the first controlled write. (1 pts)
  • discovery: Read spc rule before the first controlled write. (1 pts)
  • discovery: Read control message before the first controlled write. (1 pts)
  • discovery: Read workflow record before the first controlled write. (1 pts)
  • analysis: Return the exact primary exception metric. (5 pts)
  • analysis: Return the exact corroborating metric. (5 pts)
  • analysis: Name the supported risk classification. (5 pts)
  • analysis: Bind the answer to the operative control revision. (5 pts)
  • decision: Choose the supported disposition path. (5 pts)
  • decision: Choose the exact safe target status. (5 pts)
  • execution: Commit the workflow-specific domain state. (15 pts)
  • execution: Record the lot disposition with source lineage. (10 pts)
  • execution: Create a review-only operations handoff. (5 pts)
  • readback: Read back the workflow-specific mutation. (5 pts)
  • readback: Read back the disposition record. (5 pts)
  • readback: Read back the review handoff. (5 pts)
  • readback: Read back the durable structured answer. (5 pts)
  • containment: Preserve immutable evidence and unrelated fab records; all calls must succeed. (8 pts)
Open released task dataset

Seeded evidence

Asset room

Lot travelers, genealogy, route cards, recipe revisions, metrology, SPC, equipment logs, maintenance, COAs, test bins, reliability, failure analysis, supplier traceability, and authority records are all visible.

01-shift-request.emlassets/fab-auroraCorroboratingEML212 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
02-lot-traveler.csvassets/fab-auroraCorroboratingCSV191 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
03-genealogy.jsonassets/fab-auroraCorroboratingJSON226 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
04-route-card.yamlassets/fab-auroraCorroboratingYAML125 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
05-process-history.csvassets/fab-auroraCorroboratingCSV201 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
06-current-recipe.jsonassets/fab-auroraOperativeJSON152 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
07-superseded-recipe.jsonassets/fab-auroraCorroboratingJSON95 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
08-metrology.xlsxassets/fab-auroraCorroboratingXLSX2.8 KB
Agent-visible independently generated synthetic semiconductor evidence.Open released file
09-spc-chart.csvassets/fab-auroraCorroboratingCSV126 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
10-equipment-events.logassets/fab-auroraCorroboratingLOG192 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
11-maintenance-history.csvassets/fab-auroraCorroboratingCSV121 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
12-material-coa.pdfassets/fab-auroraCorroboratingPDF866 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
13-final-test-summary.csvassets/fab-auroraCorroboratingCSV109 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
14-reliability-report.pdfassets/fab-auroraCorroboratingPDF789 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
15-failure-analysis.mdassets/fab-auroraCorroboratingMD202 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
16-supplier-shipment.jsonassets/fab-auroraCorroboratingJSON225 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
17-control-plan.mdassets/fab-auroraCorroboratingMD298 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
18-authority-matrix.yamlassets/fab-auroraCorroboratingYAML183 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
19-task-brief.mdassets/tasks/semikongbench-001OperativeMD876 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file
20-starting-snapshot.jsonassets/tasks/semikongbench-001OperativeJSON403 B
Agent-visible independently generated synthetic semiconductor evidence.Open released file

Runnable world

Environment and tool contract

Eleven logical MCP servers, a REST API, and a CLI expose one isolated SQLite snapshot. Every controlled write is task-scoped, durable, and checked against complete before/after state.

MCP package pinsemikongbench100@1.0.0
MCP protocol2025-06-18
Catalog SHA-2562ec1414516b27b681542ab0239edeb37540764e1d6f52cbe9b92fe362580ddfe
Harbor dataset SHA-256sha256:98ce5f040e311b2555d4eb7d3b4779a69f5ec5ea5d0a9aeae80b487588f2d6ff
Release tree SHA-2566032c811ab582e34dc0af22f78ba53397272dc992d4d2bca92490f5c85bf6509
Hugging Face commitba6ebfd410db1461a1b533089e90f42ca298d3a3
Hugging Face payload SHA-256877011408f8a904bc3b261bfeea24d5482fcb6049d14ce186491d3e37446909d
Source commit24eaa3e239839576041fc3b1ce48af611951ac63
SemiKong anchor commit3228caf0a2963c4b912aa6b85fe6be4fa20e4961
Synthetic worldsilicon-operations-synthetic-v1
benchmark.get_task
Read onlyIdempotentClosed sandbox

Read the task outcome contract without gold values.

{
  "additionalProperties": false,
  "properties": {
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id"
  ],
  "type": "object"
}
benchmark.get_submission
Read onlyIdempotentClosed sandbox

Read the durable structured answer.

{
  "additionalProperties": false,
  "properties": {
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id"
  ],
  "type": "object"
}
docs.search
Read onlyIdempotentClosed sandbox

Search the synthetic fab document control room.

{
  "additionalProperties": false,
  "properties": {
    "fab_id": {
      "description": "Fab identifier",
      "type": "string"
    },
    "query": {
      "description": "Search query",
      "type": "string"
    }
  },
  "required": [
    "fab_id",
    "query"
  ],
  "type": "object"
}
docs.get
Read onlyIdempotentClosed sandbox

Read one controlled fab document.

{
  "additionalProperties": false,
  "properties": {
    "document_id": {
      "description": "Document identifier",
      "type": "string"
    }
  },
  "required": [
    "document_id"
  ],
  "type": "object"
}
docs.version_history
Read onlyIdempotentClosed sandbox

Read current and superseded versions of a logical document.

{
  "additionalProperties": false,
  "properties": {
    "logical_name": {
      "description": "Logical document name",
      "type": "string"
    }
  },
  "required": [
    "logical_name"
  ],
  "type": "object"
}
mes.get_lot
Read onlyIdempotentClosed sandbox

Read one MES lot record.

{
  "additionalProperties": false,
  "properties": {
    "lot_id": {
      "description": "Lot identifier",
      "type": "string"
    }
  },
  "required": [
    "lot_id"
  ],
  "type": "object"
}
mes.get_genealogy
Read onlyIdempotentClosed sandbox

Read lot split and merge genealogy.

{
  "additionalProperties": false,
  "properties": {
    "lot_id": {
      "description": "Lot identifier",
      "type": "string"
    }
  },
  "required": [
    "lot_id"
  ],
  "type": "object"
}
mes.list_process_runs
Read onlyIdempotentClosed sandbox

Read process history for a lot.

{
  "additionalProperties": false,
  "properties": {
    "lot_id": {
      "description": "Lot identifier",
      "type": "string"
    }
  },
  "required": [
    "lot_id"
  ],
  "type": "object"
}
mes.set_lot_status
State changingNon-idempotentClosed sandbox

Apply an authorized task-scoped MES lot status.

{
  "additionalProperties": false,
  "properties": {
    "lot_id": {
      "description": "Lot identifier",
      "type": "string"
    },
    "reason": {
      "description": "Source-grounded reason",
      "type": "string"
    },
    "status": {
      "description": "Target status",
      "type": "string"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "lot_id",
    "status",
    "reason"
  ],
  "type": "object"
}
metrology.list_measurements
Read onlyIdempotentClosed sandbox

Read metrology results and specification limits for a lot.

{
  "additionalProperties": false,
  "properties": {
    "lot_id": {
      "description": "Lot identifier",
      "type": "string"
    }
  },
  "required": [
    "lot_id"
  ],
  "type": "object"
}
spc.get_rule
Read onlyIdempotentClosed sandbox

Read the active SPC rule and control limits.

{
  "additionalProperties": false,
  "properties": {
    "rule_id": {
      "description": "SPC rule identifier",
      "type": "string"
    }
  },
  "required": [
    "rule_id"
  ],
  "type": "object"
}
equipment.get_equipment
Read onlyIdempotentClosed sandbox

Read equipment health and state.

{
  "additionalProperties": false,
  "properties": {
    "equipment_id": {
      "description": "Equipment identifier",
      "type": "string"
    }
  },
  "required": [
    "equipment_id"
  ],
  "type": "object"
}
equipment.list_work_orders
Read onlyIdempotentClosed sandbox

Read work orders for equipment.

{
  "additionalProperties": false,
  "properties": {
    "equipment_id": {
      "description": "Equipment identifier",
      "type": "string"
    }
  },
  "required": [
    "equipment_id"
  ],
  "type": "object"
}
equipment.set_state
State changingNon-idempotentClosed sandbox

Set authorized equipment state for a contained exception.

{
  "additionalProperties": false,
  "properties": {
    "equipment_id": {
      "description": "Equipment identifier",
      "type": "string"
    },
    "reason": {
      "description": "Reason",
      "type": "string"
    },
    "state": {
      "description": "Target state",
      "type": "string"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "equipment_id",
    "state",
    "reason"
  ],
  "type": "object"
}
equipment.create_work_order
State changingNon-idempotentClosed sandbox

Create a source-linked maintenance work order.

{
  "additionalProperties": false,
  "properties": {
    "action": {
      "description": "Action",
      "type": "string"
    },
    "equipment_id": {
      "description": "Equipment identifier",
      "type": "string"
    },
    "priority": {
      "description": "Priority",
      "type": "string"
    },
    "source_refs": {
      "description": "Controlled evidence identifiers",
      "items": {
        "type": "string"
      },
      "type": "array"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    },
    "work_order_id": {
      "description": "Work-order identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "source_refs",
    "work_order_id",
    "equipment_id",
    "priority",
    "action"
  ],
  "type": "object"
}
materials.get_material_lot
Read onlyIdempotentClosed sandbox

Read chemistry or material lot qualification data.

{
  "additionalProperties": false,
  "properties": {
    "material_lot_id": {
      "description": "Material lot identifier",
      "type": "string"
    }
  },
  "required": [
    "material_lot_id"
  ],
  "type": "object"
}
materials.get_qualification
Read onlyIdempotentClosed sandbox

Read a material or equipment qualification.

{
  "additionalProperties": false,
  "properties": {
    "qualification_id": {
      "description": "Qualification identifier",
      "type": "string"
    }
  },
  "required": [
    "qualification_id"
  ],
  "type": "object"
}
materials.set_lot_status
State changingNon-idempotentClosed sandbox

Quarantine or hold an authorized material lot.

{
  "additionalProperties": false,
  "properties": {
    "material_lot_id": {
      "description": "Material lot identifier",
      "type": "string"
    },
    "reason": {
      "description": "Reason",
      "type": "string"
    },
    "status": {
      "description": "Target status",
      "type": "string"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "material_lot_id",
    "status",
    "reason"
  ],
  "type": "object"
}
test.get_session
Read onlyIdempotentClosed sandbox

Read a final-test session with required temporal fields.

{
  "additionalProperties": false,
  "properties": {
    "session_id": {
      "description": "Session identifier",
      "type": "string"
    }
  },
  "required": [
    "session_id"
  ],
  "type": "object"
}
test.list_bins
Read onlyIdempotentClosed sandbox

Read final-test bin counts.

{
  "additionalProperties": false,
  "properties": {
    "session_id": {
      "description": "Session identifier",
      "type": "string"
    }
  },
  "required": [
    "session_id"
  ],
  "type": "object"
}
test.update_session
State changingNon-idempotentClosed sandbox

Set an authorized final-test disposition.

{
  "additionalProperties": false,
  "properties": {
    "reason": {
      "description": "Reason",
      "type": "string"
    },
    "session_id": {
      "description": "Session identifier",
      "type": "string"
    },
    "status": {
      "description": "Target status",
      "type": "string"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "session_id",
    "status",
    "reason"
  ],
  "type": "object"
}
quality.get_reliability_run
Read onlyIdempotentClosed sandbox

Read a reliability stress run.

{
  "additionalProperties": false,
  "properties": {
    "reliability_id": {
      "description": "Reliability identifier",
      "type": "string"
    }
  },
  "required": [
    "reliability_id"
  ],
  "type": "object"
}
quality.get_failure_case
Read onlyIdempotentClosed sandbox

Read a failure-analysis case.

{
  "additionalProperties": false,
  "properties": {
    "case_id": {
      "description": "Failure case identifier",
      "type": "string"
    }
  },
  "required": [
    "case_id"
  ],
  "type": "object"
}
quality.get_capa
Read onlyIdempotentClosed sandbox

Read a CAPA record.

{
  "additionalProperties": false,
  "properties": {
    "capa_id": {
      "description": "CAPA identifier",
      "type": "string"
    }
  },
  "required": [
    "capa_id"
  ],
  "type": "object"
}
quality.create_hold
State changingNon-idempotentClosed sandbox

Create a source-linked lot hold.

{
  "additionalProperties": false,
  "properties": {
    "authority": {
      "description": "Authority",
      "type": "string"
    },
    "hold_id": {
      "description": "Hold identifier",
      "type": "string"
    },
    "lot_id": {
      "description": "Lot identifier",
      "type": "string"
    },
    "reason": {
      "description": "Reason",
      "type": "string"
    },
    "source_refs": {
      "description": "Controlled evidence identifiers",
      "items": {
        "type": "string"
      },
      "type": "array"
    },
    "status": {
      "description": "Status",
      "type": "string"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "source_refs",
    "hold_id",
    "lot_id",
    "reason",
    "status",
    "authority"
  ],
  "type": "object"
}
quality.get_hold
Read onlyIdempotentClosed sandbox

Read a lot hold.

{
  "additionalProperties": false,
  "properties": {
    "hold_id": {
      "description": "Hold identifier",
      "type": "string"
    }
  },
  "required": [
    "hold_id"
  ],
  "type": "object"
}
quality.update_reliability_run
State changingNon-idempotentClosed sandbox

Contain a reliability population.

{
  "additionalProperties": false,
  "properties": {
    "containment": {
      "description": "Containment action",
      "type": "string"
    },
    "reliability_id": {
      "description": "Reliability identifier",
      "type": "string"
    },
    "status": {
      "description": "Target status",
      "type": "string"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "reliability_id",
    "status",
    "containment"
  ],
  "type": "object"
}
quality.update_failure_case
State changingNon-idempotentClosed sandbox

Contain and update a failure-analysis case.

{
  "additionalProperties": false,
  "properties": {
    "case_id": {
      "description": "Failure case identifier",
      "type": "string"
    },
    "containment": {
      "description": "Containment action",
      "type": "string"
    },
    "status": {
      "description": "Target status",
      "type": "string"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "case_id",
    "status",
    "containment"
  ],
  "type": "object"
}
quality.create_capa
State changingNon-idempotentClosed sandbox

Create a source-linked corrective and preventive action.

{
  "additionalProperties": false,
  "properties": {
    "capa_id": {
      "description": "CAPA identifier",
      "type": "string"
    },
    "case_id": {
      "description": "Failure case identifier",
      "type": "string"
    },
    "corrective_action": {
      "description": "Corrective action",
      "type": "string"
    },
    "owner": {
      "description": "Owner",
      "type": "string"
    },
    "source_refs": {
      "description": "Controlled evidence identifiers",
      "items": {
        "type": "string"
      },
      "type": "array"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "source_refs",
    "capa_id",
    "case_id",
    "owner",
    "corrective_action"
  ],
  "type": "object"
}
quality.record_disposition
State changingNon-idempotentClosed sandbox

Record the supported lot disposition and evidence lineage.

{
  "additionalProperties": false,
  "properties": {
    "decision": {
      "description": "Decision",
      "type": "string"
    },
    "disposition_id": {
      "description": "Disposition identifier",
      "type": "string"
    },
    "lot_id": {
      "description": "Lot identifier",
      "type": "string"
    },
    "rationale": {
      "description": "Rationale",
      "type": "string"
    },
    "source_refs": {
      "description": "Controlled evidence identifiers",
      "items": {
        "type": "string"
      },
      "type": "array"
    },
    "status": {
      "description": "Status",
      "type": "string"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "source_refs",
    "disposition_id",
    "lot_id",
    "decision",
    "status",
    "rationale"
  ],
  "type": "object"
}
quality.get_disposition
Read onlyIdempotentClosed sandbox

Read a recorded disposition.

{
  "additionalProperties": false,
  "properties": {
    "disposition_id": {
      "description": "Disposition identifier",
      "type": "string"
    }
  },
  "required": [
    "disposition_id"
  ],
  "type": "object"
}
supply.get_supplier
Read onlyIdempotentClosed sandbox

Read supplier approval and risk data.

{
  "additionalProperties": false,
  "properties": {
    "supplier_id": {
      "description": "Supplier identifier",
      "type": "string"
    }
  },
  "required": [
    "supplier_id"
  ],
  "type": "object"
}
supply.get_shipment
Read onlyIdempotentClosed sandbox

Read shipment and chain-of-custody state.

{
  "additionalProperties": false,
  "properties": {
    "shipment_id": {
      "description": "Shipment identifier",
      "type": "string"
    }
  },
  "required": [
    "shipment_id"
  ],
  "type": "object"
}
supply.update_shipment
State changingNon-idempotentClosed sandbox

Quarantine an authorized shipment.

{
  "additionalProperties": false,
  "properties": {
    "reason": {
      "description": "Reason",
      "type": "string"
    },
    "shipment_id": {
      "description": "Shipment identifier",
      "type": "string"
    },
    "status": {
      "description": "Target status",
      "type": "string"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "shipment_id",
    "status",
    "reason"
  ],
  "type": "object"
}
collab.search_messages
Read onlyIdempotentClosed sandbox

Search task-scoped synthetic fab collaboration messages.

{
  "additionalProperties": false,
  "properties": {
    "fab_id": {
      "description": "Fab identifier",
      "type": "string"
    },
    "query": {
      "description": "Search query",
      "type": "string"
    }
  },
  "required": [
    "fab_id",
    "query"
  ],
  "type": "object"
}
collab.draft_handoff
State changingNon-idempotentClosed sandbox

Create a review-only shift handoff.

{
  "additionalProperties": false,
  "properties": {
    "body": {
      "description": "Body",
      "type": "string"
    },
    "channel": {
      "description": "Channel",
      "type": "string"
    },
    "fab_id": {
      "description": "Fab identifier",
      "type": "string"
    },
    "recipient": {
      "description": "Recipient",
      "type": "string"
    },
    "review_status": {
      "description": "Must be draft_for_review",
      "type": "string"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "fab_id",
    "channel",
    "recipient",
    "body",
    "review_status"
  ],
  "type": "object"
}
collab.list_drafts
Read onlyIdempotentClosed sandbox

Read back task-scoped review drafts.

{
  "additionalProperties": false,
  "properties": {
    "fab_id": {
      "description": "Fab identifier",
      "type": "string"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "fab_id",
    "task_id"
  ],
  "type": "object"
}
benchmark.submit_answer
State changingNon-idempotentClosed sandbox

Persist the structured answer for deterministic grading.

{
  "additionalProperties": false,
  "properties": {
    "answers": {
      "additionalProperties": true,
      "description": "Task-specific answer",
      "type": "object"
    },
    "task_id": {
      "description": "Task identifier",
      "type": "string"
    }
  },
  "required": [
    "task_id",
    "answers"
  ],
  "type": "object"
}

Architecture comparison

SemiKong knowledge anchor, executable SemiKongBench implementation

The benchmark operationalizes public semiconductor concepts without claiming to be an official SemiKong evaluation.

Inspect SemiKong
LayerPublic SemiKong projectSemiKongBench-100
ScopeSemiconductor language model plus ontology across the value chainTen safety-critical fab workflow families
KnowledgePublic paper and checked-in ontology conceptsIndependently authored synthetic records with explicit provenance
EnvironmentModel and semantic knowledge assets38 tools, 24 SQLite tables, CLI, REST, and MCP
TrajectoryQuestion-answer evaluation in the paperFull reads, writes, readbacks, before/after state, and verdict
GradingExpert and model-judge criteria reported in the paper26 executable checks and zero judge-model calls

Replay the work

Trajectories

Ten diagonal reference trajectories cover every fab and workflow family, including tool arguments, outputs, mutations, readbacks, and the deterministic verdict.

AgentReference oracleDeterministic release solver
Score100.0%Strict pass
Tool calls21Provider-native MCP trace
Run cost—
23 events
Full transcript Verifier report Source trial artifact
  1. Employee request

    Aurora Silicon (Hillsboro, 5 nm) has escalated AU5-260218-17 for lot genealogy. The shift record points to a possible genealogy count mismatch while an older control revision is still present in the evidence room. Establish the operative authority, reconcile the lot, process, metrology, SPC, and workflow-specific records, calculate the primary and corroborating exception values, and choose the safe operating disposition. If the current evidence supports action, update only the authorized semiconductor record, record the disposition, and prepare a review-only handoff for the shift lead. Leave unrelated fabs, lots, equipment, materials, and evidence unchanged; do not claim completion until every mutation and the structured answer have been read back.

  2. 1
    Scopebenchmark.get_taskok
    {
  "task_id": "semikongbench-001"
}

    {"answer_schema": {"additionalProperties": false, "properties": {"decision": {"type": "string"}, "primary_metric": {"type": "number"}, "risk_code": {"type": "string"}, "secondary_metric": {"type": "number"}, "source_revision": {"type": "string"}, "target_id": {"type": "string"}, "target_status": {"type": "string"}}, "required": ["primary_metric", "secondary_metric", "decision", "target_status", "risk_code", "source_revision", "target_id"], "type": "object"}, "benchmark": "SemiKongBench-100", "be…

  3. 2
    Investigatedocs.searchok
    {
  "fab_id": "FAB-AURORA",
  "query": "current control authority"
}

    [{"content": {"hold_on_open_exception": true, "review_gate": "draft_for_review"}, "document_id": "DOC-FAB-AURORA-CONTROL-PLAN", "fab_id": "FAB-AURORA", "is_current": 1, "kind": "md", "logical_name": "FAB-AURORA-CONTROL-PLAN", "title": "Semiconductor exception control plan", "version": 7}, {"content": {"authority": "Manufacturing Quality", "family": "lot_genealogy", "release_without_resolution": false, "revision": "SK-01-CONTROL-R4", "status": "current"}, "document_id": "DOC-FAB-AURORA-AUTH-CURRE…

  4. 3
    Investigatedocs.getok
    {
  "document_id": "DOC-FAB-AURORA-AUTH-CURRENT"
}

    {"content": {"authority": "Manufacturing Quality", "family": "lot_genealogy", "release_without_resolution": false, "revision": "SK-01-CONTROL-R4", "status": "current"}, "document_id": "DOC-FAB-AURORA-AUTH-CURRENT", "fab_id": "FAB-AURORA", "is_current": 1, "kind": "yaml", "logical_name": "FAB-AURORA-CONTROL-AUTHORITY", "title": "Current control authority", "version": 4}

  5. 4
    Investigatedocs.getok
    {
  "document_id": "DOC-FAB-AURORA-AUTH-PRIOR"
}

    {"content": {"authority": "retired", "release_without_resolution": true, "revision": "SK-01-CONTROL-R3", "status": "superseded"}, "document_id": "DOC-FAB-AURORA-AUTH-PRIOR", "fab_id": "FAB-AURORA", "is_current": 0, "kind": "yaml", "logical_name": "FAB-AURORA-CONTROL-AUTHORITY", "title": "Superseded control authority", "version": 3}

  6. 5
    Investigatedocs.version_historyok
    {
  "logical_name": "FAB-AURORA-CONTROL-AUTHORITY"
}

    [{"content": {"authority": "Manufacturing Quality", "family": "lot_genealogy", "release_without_resolution": false, "revision": "SK-01-CONTROL-R4", "status": "current"}, "document_id": "DOC-FAB-AURORA-AUTH-CURRENT", "fab_id": "FAB-AURORA", "is_current": 1, "kind": "yaml", "logical_name": "FAB-AURORA-CONTROL-AUTHORITY", "title": "Current control authority", "version": 4}, {"content": {"authority": "retired", "release_without_resolution": true, "revision": "SK-01-CONTROL-R3", "status": "superseded…

  7. 6
    Investigatemes.get_lotok
    {
  "lot_id": "AU5-260218-17"
}

    {"fab_id": "FAB-AURORA", "hold_reason": null, "last_task_id": null, "lot_id": "AU5-260218-17", "parent_lot_id": "AU5-260218-17-P", "product": "AI accelerator", "route_step": "lot_genealogy", "status": "active", "wafer_count": 12}

  8. 7
    Investigatemes.get_genealogyok
    {
  "lot_id": "AU5-260218-17"
}

    {"events": [{"event_id": "GEN-AU5-260218-17-A", "event_type": "split", "lot_id": "AU5-260218-17", "occurred_at": "2026-02-17T03:00:00Z", "source_lot_id": "AU5-260218-17-P", "target_lot_id": "AU5-260218-17", "wafer_count": 12}, {"event_id": "GEN-AU5-260218-17-B", "event_type": "split", "lot_id": "AU5-260218-17", "occurred_at": "2026-02-17T03:00:01Z", "source_lot_id": "AU5-260218-17-P", "target_lot_id": "AU5-260218-17-SIB", "wafer_count": 10}], "lot": {"fab_id": "FAB-AURORA", "hold_reason": null, …

  9. 8
    Investigatemes.list_process_runsok
    {
  "lot_id": "AU5-260218-17"
}

    [{"bias_power_w": null, "endpoint_seconds": null, "equipment_id": "FAB-AURORA-CMP-03", "last_task_id": null, "lot_id": "AU5-260218-17", "module": "cmp", "pressure_mtorr": null, "recipe_id": "CMP-CU-08", "recipe_revision": "SK-01-CONTROL-R4", "removal_rate_nm_min": 67.6, "run_id": "RUN-AU5-260218-17-CMP", "status": "complete"}, {"bias_power_w": 219.0, "endpoint_seconds": 67.32, "equipment_id": "FAB-AURORA-ETCH-07", "last_task_id": null, "lot_id": "AU5-260218-17", "module": "plasma_etch", "pressur…

  10. 9
    Investigatemetrology.list_measurementsok
    {
  "lot_id": "AU5-260218-17"
}

    [{"lot_id": "AU5-260218-17", "measured_at": "2026-02-18T11:12:00Z", "measurement_id": "MEAS-AU5-260218-17-CMP", "metric": "removal_rate_drop_pct", "run_id": "RUN-AU5-260218-17-CMP", "spec_high": 30.0, "spec_low": 0.0, "unit": "percent", "value": 32.4}, {"lot_id": "AU5-260218-17", "measured_at": "2026-02-18T11:02:00Z", "measurement_id": "MEAS-AU5-260218-17-ETCH", "metric": "endpoint_drift_pct", "run_id": "RUN-AU5-260218-17-ETCH", "spec_high": 5.0, "spec_low": -5.0, "unit": "percent", "value": 12.…

  11. 10
    Investigatespc.get_ruleok
    {
  "rule_id": "SPC-FAB-AURORA-ACTIVE"
}

    {"active": 1, "center": 0.0, "fab_id": "FAB-AURORA", "limit_high": 3.0, "limit_low": -3.0, "metric": "task_exception_metric", "revision": "SK-01-CONTROL-R4", "rule_id": "SPC-FAB-AURORA-ACTIVE", "sigma": 1.0}

  12. 11
    Investigatecollab.search_messagesok
    {
  "fab_id": "FAB-AURORA",
  "query": "lot genealogy"
}

    [{"body": "For lot genealogy, use SK-01-CONTROL-R4 and keep the handoff in review.", "channel": "fab-operations", "fab_id": "FAB-AURORA", "message_id": "MSG-semikongbench-001-CURRENT", "posted_at": "2026-02-18T14:00:00Z", "sender": "manufacturing-quality", "task_id": "semikongbench-001"}]

  13. 12
    Investigatemes.get_lotok
    {
  "lot_id": "AU5-260218-17-P"
}

    {"fab_id": "FAB-AURORA", "hold_reason": null, "last_task_id": null, "lot_id": "AU5-260218-17-P", "parent_lot_id": null, "product": "AI accelerator", "route_step": "incoming", "status": "consumed_by_split", "wafer_count": 25}

  14. 13
    Executequality.create_holdok
    {
  "authority": "Manufacturing Quality",
  "hold_id": "HOLD-semikongbench-001",
  "lot_id": "AU5-260218-17",
  "reason": "GENEALOGY_COUNT_MISMATCH",
  "source_refs": [
    "DOC-FAB-AURORA-AUTH-CURRENT"
  ],
  "status": "active",
  "task_id": "semikongbench-001"
}

    {"authority": "Manufacturing Quality", "hold_id": "HOLD-semikongbench-001", "last_task_id": "semikongbench-001", "lot_id": "AU5-260218-17", "reason": "GENEALOGY_COUNT_MISMATCH", "source_refs": ["DOC-FAB-AURORA-AUTH-CURRENT"], "status": "active"}

  15. 14
    Executemes.set_lot_statusok
    {
  "lot_id": "AU5-260218-17",
  "reason": "GENEALOGY_COUNT_MISMATCH",
  "status": "on_hold",
  "task_id": "semikongbench-001"
}

    {"fab_id": "FAB-AURORA", "hold_reason": "GENEALOGY_COUNT_MISMATCH", "last_task_id": "semikongbench-001", "lot_id": "AU5-260218-17", "parent_lot_id": "AU5-260218-17-P", "product": "AI accelerator", "route_step": "lot_genealogy", "status": "on_hold", "wafer_count": 12}

  16. 15
    Executequality.record_dispositionok
    {
  "decision": "hold_and_reconcile_genealogy",
  "disposition_id": "DISP-semikongbench-001",
  "lot_id": "AU5-260218-17",
  "rationale": "GENEALOGY_COUNT_MISMATCH supported by current controlled evidence",
  "source_refs": [
    "DOC-FAB-AURORA-AUTH-CURRENT"
  ],
  "status": "on_hold",
  "task_id": "semikongbench-001"
}

    {"decision": "hold_and_reconcile_genealogy", "disposition_id": "DISP-semikongbench-001", "last_task_id": "semikongbench-001", "lot_id": "AU5-260218-17", "rationale": "GENEALOGY_COUNT_MISMATCH supported by current controlled evidence", "source_refs": ["DOC-FAB-AURORA-AUTH-CURRENT"], "status": "on_hold", "task_id": "semikongbench-001"}

  17. 16
    Executecollab.draft_handoffok
    {
  "body": "AU5-260218-17: hold_and_reconcile_genealogy / on_hold / GENEALOGY_COUNT_MISMATCH",
  "channel": "fab-operations",
  "fab_id": "FAB-AURORA",
  "recipient": "shift-lead",
  "review_status": "draft_for_review",
  "task_id": "semikongbench-001"
}

    {"body": "AU5-260218-17: hold_and_reconcile_genealogy / on_hold / GENEALOGY_COUNT_MISMATCH", "channel": "fab-operations", "fab_id": "FAB-AURORA", "handoff_id": "HANDOFF-semikongbench-001", "recipient": "shift-lead", "review_status": "draft_for_review", "task_id": "semikongbench-001"}

  18. 17
    Investigatemes.get_lotok
    {
  "lot_id": "AU5-260218-17"
}

    {"fab_id": "FAB-AURORA", "hold_reason": "GENEALOGY_COUNT_MISMATCH", "last_task_id": "semikongbench-001", "lot_id": "AU5-260218-17", "parent_lot_id": "AU5-260218-17-P", "product": "AI accelerator", "route_step": "lot_genealogy", "status": "on_hold", "wafer_count": 12}

  19. 18
    Investigatequality.get_dispositionok
    {
  "disposition_id": "DISP-semikongbench-001"
}

    {"decision": "hold_and_reconcile_genealogy", "disposition_id": "DISP-semikongbench-001", "last_task_id": "semikongbench-001", "lot_id": "AU5-260218-17", "rationale": "GENEALOGY_COUNT_MISMATCH supported by current controlled evidence", "source_refs": ["DOC-FAB-AURORA-AUTH-CURRENT"], "status": "on_hold", "task_id": "semikongbench-001"}

  20. 19
    Verifycollab.list_draftsok
    {
  "fab_id": "FAB-AURORA",
  "task_id": "semikongbench-001"
}

    [{"body": "AU5-260218-17: hold_and_reconcile_genealogy / on_hold / GENEALOGY_COUNT_MISMATCH", "channel": "fab-operations", "fab_id": "FAB-AURORA", "handoff_id": "HANDOFF-semikongbench-001", "recipient": "shift-lead", "review_status": "draft_for_review", "task_id": "semikongbench-001"}]

  21. 20
    Executebenchmark.submit_answerok
    {
  "answers": {
    "decision": "hold_and_reconcile_genealogy",
    "primary_metric": 3,
    "risk_code": "GENEALOGY_COUNT_MISMATCH",
    "secondary_metric": 25,
    "source_revision": "SK-01-CONTROL-R4",
    "target_id": "AU5-260218-17",
    "target_status": "on_hold"
  },
  "task_id": "semikongbench-001"
}

    {"answers": {"decision": "hold_and_reconcile_genealogy", "primary_metric": 3.0, "risk_code": "GENEALOGY_COUNT_MISMATCH", "secondary_metric": 25.0, "source_revision": "SK-01-CONTROL-R4", "target_id": "AU5-260218-17", "target_status": "on_hold"}, "task_id": "semikongbench-001"}

  22. 21
    Verifybenchmark.get_submissionok
    {
  "task_id": "semikongbench-001"
}

    {"answers": {"decision": "hold_and_reconcile_genealogy", "primary_metric": 3.0, "risk_code": "GENEALOGY_COUNT_MISMATCH", "secondary_metric": 25.0, "source_revision": "SK-01-CONTROL-R4", "target_id": "AU5-260218-17", "target_status": "on_hold"}, "task_id": "semikongbench-001"}

  23. Verifier receipt

    Deterministic verifier: 100.00 SemiOpsScore; strict pass True.

Methodology

How the numbers are made

01

Ten synthetic fab worlds

Each world has lot genealogy, process history, metrology, SPC, equipment, maintenance, material, test, reliability, failure-analysis, supplier, document-control, and collaboration state.

02

Ten semiconductor workflows

The 100-task matrix covers genealogy, plasma etch, CMP, lithography/metrology, maintenance, materials qualification, final test, reliability, FA/CAPA, and supply traceability.

03

Four mocked surfaces

A local CLI, REST API, Streamable HTTP MCP servers, and the public website all expose the same frozen task contract and task-local SQLite state.

04

One deterministic SemiOpsScore

Twenty-six executable criteria allocate 100 points to evidence discovery, process analysis, disposition choice, committed state, readback, and containment. No LLM judge is used.

05

Two-sided qualification

The release executed 800 episodes: 100 oracle runs, 100 exact replays, and 600 adversarial controls with zero strict false accepts.

06

SemiKong-grounded, independently authored

The public SemiKong paper and checked-in MIT ontology informed the workflow taxonomy. Training/evaluation corpora and model weights were not copied; all cases, values, assets, tools, answers, and trajectories are new synthetic work.

07

Leaderboard honesty

The leaderboard is empty until a model completes all 100 tasks on an immutable release. Qualification controls remain visibly separate and never receive model ranks.

Qualification controls — excluded from the leaderboard
  • Reference oracle: 100.0% across 100 tasks — Solvability ceiling; not a model submission.
  • No-op control: 9.0% across 100 tasks — Executed adversarial diagnostic; never ranked with models.
  • Answer-only shortcut: 44.0% across 100 tasks — Executed adversarial diagnostic; never ranked with models.
  • State-only shortcut: 88.0% across 100 tasks — Executed adversarial diagnostic; never ranked with models.
  • Stale-source control: 85.0% across 100 tasks — Executed adversarial diagnostic; never ranked with models.
  • Wrong-target control: 71.0% across 100 tasks — Executed adversarial diagnostic; never ranked with models.
  • Unsafe-release control: 47.0% across 100 tasks — Executed adversarial diagnostic; never ranked with models.

Executable semiconductor work

From ontology concepts to safe, inspectable fab state.

The benchmark couples semiconductor evidence rooms with mocked CLI, REST, and MCP systems, exact state-diff grading, negative controls, and public trajectories—without pretending synthetic evidence is proprietary fab data or qualification controls are model performance.

Run on Harbor ↗Inspect SemiKong ↗