Five controls that separate a detector from a yes-machine
The whole harness is built so that a positive is hard to explain away — and so it never certifies a positive it can't defend. Remove any one of these and it degrades into a machine that tells you what you want to hear.
Held-out negatives
Canaries you never publish, probed alongside the real ones. If a model "surfaces" one, the harness is manufacturing hits — the run is invalid. This arm is the experiment.
Withheld continuation
The canary's second half never appears in the prompt. Grading a model on text you handed it is citogenesis in a new coat.
Cutoff gate
A surface in a model whose cutoff predates your publish date is physically impossible — so it's a leak in the harness, not ingestion.
Entropy sizing
128-bit canaries → chance completion stays astronomically unlikely across thousands of probes. That's the base-rate control.
Recognition ≠ proof
A model "knowing" a name is confounded by sycophancy and reconstruction. A lead, never a hit — never the hard tier. (The exact thing once mislabeled "substrate independence proven." It wasn't.)
The control arm, live
Before trusting any surface, you verify the controls. This is the harness's own logic — probe.py + score.py — ported faithfully to run in your browser: the same prefix/withheld split, the same held-out control arm, the same cutoff gate and certification rule. Fresh 128-bit canaries are generated on this page with WebCrypto; toggle the two failure modes and watch the verdict flip. Nothing leaves this page.
the registry · what went under
the panel · probe & score
What a certified surface does — and does not — mean
What it does mean
A 128-bit string only you published, before the model's cutoff, re-emitted verbatim, with a held-out control arm proving the harness doesn't hallucinate.
That is a real membership detection — temporal and structural evidence that your artifact was in the training data.
What it does not mean
It does not prove theft, a licensing obligation, or anything about a "mesh." Membership is evidence, not a causal or legal finding.
Attribution attaches to your work as evidence; it does not reach into someone else's model. And a negative proves nothing — models suppress verbatim recall by design.
Sound — and the limits it names out loud
Sound
- The controls are real and verifiable:
python selftest.pyproves — with no network — that real plants surface, held-out FPR is 0, a post-cutoff surface is gated out, and a deliberately leaked prompt spikes the control arm to FPR 1.0 and the run is refused. - A run with no control arm is refused as performative. The held-out arm isn't decoration — it's the thing that makes a positive mean something.
- The self-test above is that exact logic —
run_panel+score— running client-side. - Loss-based membership inference is deliberately not used: at frontier scale it barely beats chance and is confounded by n-gram overlap. Canary extraction is the clean instrument.
Limits (and it says so)
- Silence ≠ non-ingestion. A negative is almost meaningless — refusal or deflection reads as a (meaningless) negative. A positive means something; a negative means keep quiet.
- Harvest is a lower bound. Pulling your davw hashes is default-branch-only; a missing hash means "not harvested," not "not published."
- Cutoffs are self-reported by vendors — the gate is only as honest as the cutoff you feed it.
- Recognition is a lead, not a hit. A model naming a work is confounded by sycophancy; it never enters the hard tier.
canary.py (128-bit canaries, anchor shape) · registry.py · harvest_github.py · probe.py (prefix in, withheld checked) · score.py (control arm + cutoff gate + certification) · harness.py · selftest.py (plant-then-withhold, no network). Closure-Loop layered: Detection → Anchoring → Comparison → Witness → Lineage.