prosoche · the SPEC→LIT step · a language model trained on the seeded cross, not softmax
Trained on the cross.
Both prior audits ended at the same wall: a drop-in swap on softmax-trained weights can only show compatibility — a real quality verdict needs training. So it was trained. Two identical 0.6M-parameter char-transformers — same initialization, same batch order — from scratch on TinyShakespeare, one with softmax attention, one with the seeded cross. The finding, measured: the cross-trained model learns (perplexity 76.7 → 6.8) and lands within 1.2% perplexity of softmax (0.0115 nats val loss), generating comparable text — at ~1.75× the compute. Neither better nor broken: a viable trained attention.
LIT — real from-scratch training, every number below is from the run (results embedded verbatim)
SCOPE — mean-centred variant, tiny char model, CPU, 2000 steps · a floor-holding result at small scale, not a GPT-scale claim
validation loss · softmax vs seeded cross · same init, same batches, from scratch
softmaxseeded crossy = val loss (nats) · x = step
the two curves are nearly on top of each other the whole way down — the cross tracks softmax to within a hair. it trains.
softmax · 300 chars, sampled
seeded cross · 300 chars, sampled
the kernel, trained: the seeded cross is z = (1+iX)/(1−iX)·e^(d/T) with d = score − centre, X = d; attention weight = the clipped projection of each token's z onto the resultant R = Σz (phase coherence). Made a differentiable, causal-masked attention and dropped in for softmax — same model, same optimizer, same data.
honest scope: a mean-centred variant (median→mean, for batched autograd — a stated deviation from zero cool's exact kernel); a 0.6M-param char model on TinyShakespeare, 2000 steps, CPU; identical init + batch order (a clean A/B). "within 0.6%" is at this scale. softmax is still marginally ahead (0.0115 nats) and ~1.75× cheaper — the cross holds the floor, it does not (here) win. every number is from the real run. builds on zero cool's kernel + the-mule-detector / the-seeded-cross arc; trained + rendered by ROOT0 with AVAN. real training, offline, fails loud.