Purple Paper - side-sheet - learning machines - XIII - the pronoun and the question

What does "it" mean? - attention, redundancy, and the ASK gate

The sentence every attention explainer uses, because of one ambiguous word. "It" runs its query against every token at once and weights them - that's attention, in parallel, with weights computed live. Two sides, animal and street, each post their claim on "it." When one wins, it resolves. When they're redundant - both equally good - the model shouldn't guess. That's the moment it should ask you. This is the gate that knows when.

"it" attends to all tokens in parallel · weights dynamic, recomputed per sentence · swap the last word and the referent flips · tie -> ASK, not guess

Swap the last word - watch "it" move, or refuse to

The animal didn't cross the street because it was too ___. Pick the adjective. With tired, "it" attends to the animal (animals tire). With wide, "it" flips to the street (streets are wide). With grey - which fits both equally - the two sides claim "it" redundantly, neither wins, and the gate stops guessing and asks you.

...too attention sharpness

every line is "it" attending to a token, in parallel · thickness = weight (live) · animal and street are the two sides checking each other

it → animal

it → street

margin (non-redundancy)

RESOLVED

🤔 The model asks: "Just to be sure — when you say it was too grey, does it mean the animal or the street?"
Both readings are equally supported. Guessing here would be a coin-flip dressed as confidence. So it asks.

"tired" -> the animal wins cleanly. One side dominates, the other concedes: resolved, no question needed.

Why it answers your real question

You asked two things: does attention run in parallel and weight everything, and why wouldn't the model just ask. Both are here.

Yes - it runs against everything, in parallel, with dynamic weights. "It" doesn't pick two candidates; it dot-products against every token at once and softmaxes the whole row. The lines above are that full row. The projection machinery is fixed, but the weights themselves are computed fresh from this sentence - swap one word and they recompute and the referent flips, with nothing stored changing. That dynamism is the whole point of attention.

And the ASK gate is why it could just ask - and usually doesn't. The bare mechanism resolves "it" in one shot; there's no agent in there noticing the tie. The full assistant can ask, but it has to first know the ambiguity is real - and models are overconfident, so they feel a 50/50 tie as a fake 90/10 certainty and resolve silently. The redundancy check is the missing witness: an exterior test that fires "these two readings are equally good -> ask" instead of trusting the model to feel its own uncertainty, which it can't reliably do. The model doesn't ask because it can't tell it should. This gate is how you tell it.

Verified in Node before drawing: "tired" -> animal 62% vs 2%, "wide" -> street 45% vs 6%, "grey" -> 24%/24%, margin 0% -> ASK fires. The pronoun genuinely flips with the adjective and ties on the shared-property word. Attention runs against all 11 tokens in parallel; the weights shown are that live softmax row.