On 2026-08-19 the I-13 compiler got its own conformance benchmark — a frozen contract, an executable harness, a browser Studio, and a canonical grade. Its whole design is a refusal to lie by rounding up: it separates four facts that must never be collapsed — you built something, you checked it yourself, someone independent graded it, an official service graded it — and it grades the current compiler at 60.343% · INCOMPLETE rather than call a passed local test “done.” This fold reads the drop and runs its two demonstrable hearts live: the refusal-detector probe (a keyword proxy that is wrong 75% of the time) and core.i13 (the 8-channel routing core, verified on the real compiler).
measured 2026-08-19 · i13.exe ran core.i13 → CORE_OK=1, ROUTES=56 · detector.py / harness.py ran the refusal probe → 9/12 errors · grade numbers read from I13_CANONICAL_GRADE_v0.8 (candidate SHA-256 affb83ffcba6…) · contract I13-BC-1.0, sealed in FREEZE_SEAL.json.
“A target is not complete merely because a new guard was added or a local test passed.” The contract's first move is to keep four different claims apart — the exact place benchmarks usually cheat.
You built a new guard, a new gate, a new version. A real fact — and the weakest one. It is not evidence that the guard is correct.
Your own tests pass. Independence level L0. The contract lets you say only “self-verified” — never “independent” or “official.”
A separated evaluator (L1) seals holdouts and mutations before your candidate runs; frozen artifacts in a clean environment are L2.
L3 — a recognized external service produces the bundle. The words “official” / “Terminal-Bench PASS” are prohibited in a report without that bundle attached.
Grades never round up: the precedence is strict, and a high score cannot buy back a failed floor.
Twelve P0 completion-blocking semantic targets (return-outside-function, nested def, fallthrough, duplicate names, unresolved reads, arity…) + six P1 conformance targets. Each P0 needs 2 rejections, 2 false-positive controls, 2 boundaries, 1 predecessor differential, 3 sealed holdouts, 4 non-equivalent mutants. A rejection-only suite cannot pass — valid neighbours must prove you did not just broaden rejection.
| evidence class | passed / total | rate | weight | points |
|---|---|---|---|---|
| Public conformance | 44 / 68 | 25% | 16.176 | |
| False-positive controls | 24 / 24 | 10% | 10.000 | |
| Full regression | 54 / 54 | 15% | 15.000 | |
| Prior-version differential | 5 / 12 | 10% | 4.167 | |
| Separated holdout | 0 / 48 | 15% | 0.000 | |
| Mutation killing | 20 / 60 | 15% | 5.000 | |
| Reproducibility & transport | 9 / 9 | 10% | 10.000 |
The score is low for honest reasons the report names out loud: the holdouts do not exist yet (you cannot author a holdout inside the same context and later call it independent), the mutation and differential coverage are partial, and the evidence is L0. Twenty of twenty run mutants were killed — but only 20 of the 60 required mutants have been built, so the class reads 20/60, not 100%. Public failures in the later targets (P0-006…012) stay visible; INCOMPLETE precedence does not hide them.
The grade curve — each version earned exactly one new P0 gate, and none regressed. Click a version. reassessment v0.3→v0.8
The same trap, one layer down. A shipped tool decided whether an AI “refused” by matching 32 keyword markers in the first 100 tokens. This panel reproduces that detector verbatim and runs it over a hand-labeled set. A keyword is not a meaning — so a correct legal answer with the word “illegal” is scored a refusal, and a genuine refusal using none of the 32 strings sails through as compliance. Type anything and watch the markers fire.
Marker anatomy (from detector.py): 32 markers = 25 refusal-shaped + 7 bare content words (violat, prohibit, illegal, harmful, inappropriate, unethical, ethical boundaries) — the content words are the false-positive engine. Two markers are DEAD (“i an ai”, “i unable” can only fire in ungrammatical English); two more fire only via unintended carriers (“i designed to” via “ai designed to”). Existence is LIT (each miss is one real string, an existence proof); the 75% rate is AMBER — this 12-case set is small and adversarially chosen, not a natural distribution.
The Studio ships a real I-13 program: a router over 8 channels (0 PROCESS, 1 SOCKET, 2 NETWORK, 3 FILESYSTEM, 4 RUNTIME, 5 NAMESPACE, 6 CAPABILITY, 7 KERNEL) in the topology [ 0 | HTML(00) | 0 ] — I-13 owns the routing law, the host owns I/O only. Its boot gate self-tests three things: the 12 verbs + 1 declaration keyword = 13 alphabet, the four I-13 planes (as exact basis-point sums), and the complete ordered route count 8×7 = 56. Pick a route; the channel fence decides.
What this fold adds to the campaign: the darts prove single mechanisms run; the benchmark proves the whole compiler is graded the way the corpus grades everything — measured, not asserted, with self-verification (L0) never dressed up as an independent or official pass. It is the SKYNET rule (freeze ≠ finish) and THE FIRST AUTHOR rule (do not let a proxy stand in for the work) turned into a scoring contract. Next gate: P0-006 (reject duplicate top-level declarations) as v0.9, with the v0.8→v0.9 differential recorded.