architecture + format, fused · verified on distilgpt2

A transformer block, and the 4-bit weights flowing through it.

The architecture and the format, in one picture. This is a full GPT-2 transformer block rebuilt from scratch — attention plus MLP — verified against real distilgpt2 to machine precision. Then its weights are swapped to MXFP4 (4-bit) and the same block is run again, to see how the format flows through the architecture and where the error lands.

VERIFIED — from-scratch block matches distilgpt2 to 2e-7 MEASURED — MXFP4 weight error & block output drift, real weights BLEEDING EDGE — MXFP4 is the format; a real Tensor Core runs it, not this CPU

What the measurement shows: each weight matrix quantizes to ~20% error in MXFP4, but the block output drifts ~35% — the error compounds as it flows through attention, the MLP, and the residual adds. That's the honest cost of naive 4-bit on a single tiny model with no calibration. Real deployments recover most of this with rotations / GPTQ-style calibration (the open W4A4 research front) — this is the raw, uncalibrated floor.

the fusion: "transformer" = the architecture (attention+MLP block, verified here). "MXFP4" = the number format the weights sit in. "Tensor Core" = the Blackwell hardware unit that runs the 4-bit matmul. this instrument builds the first, applies the second, and names the third — it does not run on the third (numpy on CPU reproduces the numerics, not the silicon throughput).
method: GPT-2 block rebuilt in numpy from distilgpt2's real weights, verified vs the HF block (2e-7). weights quantized with the E2M1 + block-32 MXFP4 codec; block re-run; output drift measured. offline.