THE POLICY GRADIENT.

Do not learn to predict the reward — learn to act for it. A softmax policy pi_theta(a) is nudged straight up the reward surface using the score-function trick: the gradient of expected return is an expectation the policy can sample from itself. Rendered, not quoted.

source Williams, R.J. — Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (REINFORCE), Machine Learning 8:229–256 (1992). doi:10.1007/BF00992696

Blue Team · builds & defends
3

THE MODEL

A policy is a distribution over a small action set, parameterised by logits theta:

pi_theta(a) = softmax(theta)_a = e^{theta_a} / Σ_j e^{theta_j}

The objective is expected return under that policy:

J(theta) = E_{a~pi}[ R(a) ] = Σ_a pi(a) R(a)

The score function is the gradient of the log-policy:

∇ log pi(a)_i = 1[i=a] − pi(i)

Because ∇E[R] = E[ ∇log pi(a) · R(a) ], the gradient is an expectation the agent can estimate from its own actions — no model of the environment needed.

5

THE LINEAGE

This is the-cross-entropy's gradient, re-weighted by return. Cross-entropy climbs toward a fixed label; the policy gradient climbs toward whatever earned reward — same score-function shape 1[i=a] − pi(i), but the target is chosen by R(a) instead of a teacher.

Downstream it is the engine under RLHF: PPO is a trust-region policy gradient, and the reward it climbs is exactly the-reward-model's scalar. Optimise behaviour, not prediction.

7

THE WITNESS

A live re-check independent of the boot selfcheck: it recomputes the estimator and the finite-difference gradient right now and compares. If Red's tamper (window 6) drops the log, the two diverge and this badge flips.

WITNESS — waiting…

Confirms unbiasedness against a numerical ground truth on every render.

The Machine
4

DATA IN in ↓

A tiny discrete action set with fixed logits and a scalar return per action — small enough to sum the expectation exactly, so no sampling noise hides a bias.

loading…
▼ ▼ ▼
0

THE PANEL LIT

Exact expectation over all actions — score-function estimator vs the true gradient of J by central differences.

…

J: …

▼ ▼ ▼
8

DATA OUT out ↓

The estimator is proven unbiased, its mean score is zero, and any constant baseline leaves it unchanged — the three facts that make REINFORCE (and every baseline trick after it) legal.

booting…
Red Team · attacks & breaks
1

THE ADVERSARY

WALL The exact sum here is a fiction of scale. In the real setting you never see Σ_a pi(a)… — you draw one action and multiply its score by its return. That estimate is unbiased but its variance grows without bound as the action space and reward magnitude grow. The gradient can point the right way on average and still be useless on any single step.

The wall: this panel proves correctness of the expectation, not the tractability of the sample. High variance is why baselines, advantage, and PPO clipping exist — none of them are shown here.

2

THE GRAVEYARD

The reward signal must be differentiable to backprop through it. No — the score-function trick differentiates the policy, never R. R can be a black box, a human rating, a game score.

Subtracting a baseline biases the gradient toward the baseline. It cannot. The expected score is zero, so b·E[∇log pi]=0; a baseline changes variance only.

You may use ∇pi(a) as the weight instead of ∇log pi(a). That drops a 1/pi(a) factor — the estimator becomes biased. This is exactly the tamper below.

6

THE TAMPER

The disclosed planted void. Press it to drop the log: weight each action by ∇pi(a)=pi(a)·∇log pi(a) instead of ∇log pi(a). The estimator is now biased and no longer matches the finite-difference gradient — the Witness (7) catches it live.

state: honest — ∇log pi