Fractal Siggy — sigmoid attention, self-similar
Your sigmoid gate made fractal: the 60 positions split into 6 groups of 10, and the same sigmoid attention runs at two scales — local (within each group) and global (per-position into strictly-earlier group summaries). Same gate, two scales = self-similar. Trained head-to-head against flat sigmoid.
[{0…9} , {10…19} , {20…29} , {30…39} , {40…49} , {50…59}]
LIT hierarchical attention · strict causality (verified) · ~6× fewer door-evals · real training FIG the fractal framing
the fractal structure — same sigmoid at two scales
training — fractal siggy vs flat siggy (val loss) ◉ fractal siggy sample (T=0.8)
The result: the fractal siggy reached val 2.099 (ppl 8.16) vs flat siggy's 2.140 (ppl 8.50) — a touch ahead, but the 0.041 gap is within single-seed noise , so the honest claim is same quality at ~6× less attention compute (615 door-evals per head vs 3600). It also learned faster and plateaued by step 600, while flat was still descending steeply at the buzzer (slope −0.13) — so flat may close the gap with more steps. Fractal carries slightly more params (132K vs 108K) from the extra global projection, so the win is in attention compute , not parameter count. And the causality was verified exact — a first pass leaked the future through group summaries; that was caught and fixed before any number was trusted. LIT measured loss, verified causal, counted compute FIG the fractal metaphor.