Learn without labels by pulling a positive pair together and pushing negatives apart. The InfoNCE loss makes that a categorical guess — which of these N candidates is the true match for the context? — and its value lower-bounds the mutual information the representation keeps: log N − L ≤ I(x;c). Down the center the batch flows: candidates go in, the critic scores and the softmax decides, the loss and the bound come out. The blue team builds and defends it; the red team tries to break it.
source van den Oord, Li & Vinyals, Representation Learning with Contrastive Predictive Coding (InfoNCE), 2018 — arxiv.org/abs/1807.03748; SimCLR: Chen et al., 2020. Rendered, not quoted.
InfoNCE is a categorical cross-entropy that must identify the positive among N candidates (1 positive drawn from p(x|c), N−1 negatives from p(x)):
L = −log es⁺ / ( es⁺ + Σ es⁻ )
The denominator is a genuine softmax — the N probabilities sum to 1. Score the positive highest and L is small; give no signal and L climbs to its maximum log N.
The optimal critic is proportional to the density ratio p(x|c) / p(x). On the constructed K-ary channel (K=4, q=0.7) that ratio is K·p(x|c), computed live:
| candidate | p(x|c) | f* = p(x|c)/p(x) |
|---|
Contrastive learning is the cross-entropy turned inside out: no fixed label set, just one true positive scored against sampled negatives. The positive comes free — two augmentations of one image, or nearby moments of one signal.
The InfoNCE mutual-information bound is the engine behind SimCLR (Chen et al., 2020) and CLIP (Radford et al., 2021): the same softmax-over-negatives, scaled to millions of pairs. Each sphere is the next one's premise.
The blue team's live check: recompute the invariants — no-signal loss must equal log N, a discriminating batch must fall below it, and the Monte-Carlo InfoNCE estimate must stay a valid lower bound on the channel's known I(x;c). If red tampers, this badge is where it shows.
A batch is one context c and N candidates: the single positive x⁺ ~ p(x|c) — the augmented view, the next frame — and N−1 negatives x⁻ ~ p(x), drawn from the rest of the batch.
The critic assigns each candidate a similarity s = sim(x, c) / τ — a dot product of embeddings over a temperature τ. High τ flattens the softmax; low τ sharpens it. Those scores are exactly what you feed the panel below.
"Positive" and "negative" are not labels a human wrote — they are which candidate actually came from this context. That is the whole self-supervised trick.
Raise the positive similarity or drop τ (the critic improves) → L falls → the MI bound rises toward its ceiling log N. Nothing is looked up; the softmax is computed on the spot.
| candidate | sim | score s | softmax p |
|---|
L = −log p(positive), computed from the live softmax over 1 positive + N−1 negatives.
What the machine produces, proven: the softmax is a valid probability (sums to 1); the loss L ≥ 0 is minimized by ranking the positive first; and log N − L is a genuine lower bound on the mutual information I(x;c) — verified against a constructed channel whose I is known in closed form. The bound tightens as the critic improves and its ceiling is log N, so more negatives raise the ceiling.
The blue team's witness (left) confirms these live; the red team (right) tries to make the bound lie.
It is a lower bound only: a weak critic makes it vacuous (→ 0). And the number depends on how the negatives are sampled AMBER — a biased negative sampler silently changes what "apart" means. Contrastive learning does not maximize MI; it maximizes a loose, batch-limited bound on it.
"Contrastive learning maximizes mutual information." Cut. It maximizes a lower bound (InfoNCE), capped at log N and often loose — computed, not assumed, in the machine.
"More negatives always help." Kept, corrected. More negatives raise the ceiling log N, but returns diminish and easy negatives add almost nothing — the ceiling grows only as log.
"You need labels to learn representations." Cut. The positive is generated free from augmentations or time; no human label is read.
The red team's move: drop the negatives from the denominator, leaving only the positive. The softmax is then trivially 1, so L = 0 for any input and the bound reports log N no matter how bad the critic is. The blue team's witness (window 7) is watching.
Remove the negatives and the loss collapses to 0 regardless of signal — the "loss reflects discriminating positive from negatives" check fails and the MI bound becomes vacuous. The witness recomputes, disagrees with the known channel, and turns red. Nothing is faked; the attack is real and it is caught.