ono silo plain v5 · ROUTER
the <ono> router · v5

THE ROUTER

Why run every input through the expensive path? v5 puts a router up front — the <ono> (order / no-order) gate — that reads the input and decides: send it to Dave's cheap silo when order doesn't matter, or the plain order-aware expert when it does. Trained from scratch, gradient-checked. It matches the expensive expert while spending ~40% less attention on average, and reliably beats the cheap one — with honest caveats, stated below.

The gate

One router, two destinations

Each example is either a bag (label = the most frequent group; order doesn't matter) or an order example (label = the first token's group; order matters). A regime cue rides in the input, so the router can tell them apart. It emits a gate q = [q_silo, q_plain]; at inference we hard-route to the preferred expert and pay only that expert's attention.

input + cue tokens · regime marker <ono> router no order → order → CONTENT SILO cheap · K²=16 pairs PLAIN EXPERT order-aware · (N+1)²=81
The held-out results

Router vs. always-silo vs. always-plain

All three trained from scratch on the same mixed set, both arms to convergence (300 epochs). silo-only is cheap but can't do the order half; plain-only does both but always pays; the router sends each example to the right expert. Bars below are seed 0; the callout gives the range across 5 seeds. Accuracy bars (red line = chance); the orange bar is average attention pairs, relative to plain's 81.

◎ results.json — trained from scratch; the router-MoE backprop (with & without the compute penalty) is gradient-checked in selftest.py
The robust win is conditional computation: the router pays the plain N² only when order actually matters, so it saves ~40% average attention on every seed. Its small accuracy edge over plain (mean +3.3 pts, but −3 to +9 across seeds) comes from expert specialisation — each head trains almost only on its own regime and gets clean at it. On this world silo-only and plain-only tie on the bag task (0.915 each); the silo collapses on order (0.42) while plain handles it (0.92), which is why the always-cheap silo can't stand alone. Synthetic probe, tiny models, 5 seeds — reproducible with python train.py / python sweep.py. Not a leaderboard.
The colophon

What's real — and the honest caveats

What's real

The router is genuinely trained: selftest.py gradient-checks the full router-MoE backprop — with and without the compute penalty — against numerical gradients (< 1e-5). It routes the two regimes near-perfectly (seed 0: bag→silo 100%, order→plain 100%), and — verified across 5 seeds — spends ~40% less average attention than always-plain on every seed while staying within a few points of it. It reliably beats the always-cheap silo. Reproducible: python train.py, python sweep.py.

The caveats

Not a guaranteed accuracy win: router − plain runs −3 to +9 pts across seeds — it trails plain on 1 of 5. And a router can only route because the regime is detectable in the input (a cue carries it); if nothing carried it, no router could tell. The compute penalty is standard cost-aware routing, not magic; the router adds a second head + gate (428 vs 386 params) over a shared encoder. Synthetic probes, tiny models — a demonstration, not a benchmark.

SOURCE · github.com/DavidWise01/transformer-silo-v5 · model.py (shared encoder + silo & plain heads + the <ono> router, gradient-checked) · tasks.py (mixed-regime data + regime cue) · train.py → results.json · selftest.py. The pentalogy: build (v1) · measure (v2) · train one silo (v3) · two in parallel (v4) · route between them (v5).
⧉ the code (GitHub) ⌗ v1 ↗ v2 ▲ v3 ◆ v4 ◈ v6 · the cascade ⌂ ud0