A policy that acts, guided by a value that judges. The actor is a softmax policy π; the critic is a state value V(s). Their difference is the advantage A(s,a)=Q(s,a)−V(s) — how much better than average an action turned out. Down the center the terms go in, the engine computes the policy gradient with and without the critic as a baseline, and the proven result comes out: the baseline is free of bias yet cuts the variance. The blue team builds it; the red team tries to break it.
source A. G. Barto, R. S. Sutton & C. W. Anderson, Neuronlike adaptive elements that can solve difficult learning control problems, IEEE Trans. Systems, Man & Cybernetics, SMC-13 (1983) 834–846 — scholarworks.umass.edu/cs_faculty_pubs/362. Rendered, not quoted.
The actor is a softmax over a small action set: π(a|s)=softmax(z)_a. Its score is exact — ∂log π(a)/∂zi = [i=a] − πi. The critic is the state value V(s)=Σa π(a)Q(s,a). The policy gradient is g = Ea~π[ score(a)·(Q(a)−b) ].
Two facts do all the work: Ea~π[score(a)] = 0, so any action-independent b leaves g untouched; and the exact gradient collapses to gi = πi·A(s,ai). Live, for the current state:
| a | π(a) | A(a)=Q−V | gi=πiAi |
|---|
Neighbour sphere: the-policy-gradient. Plain REINFORCE pushes on score(a)·Q(a) — unbiased but noisy, because Q swings with the raw return.
The actor-critic hands that same estimator a learned baseline: subtract V(s) so the actor is steered by advantage, not by absolute return. The critic tells the actor how much better than average an action was — the noise the plain gradient suffers is exactly what the baseline removes. Each sphere is the next one's premise.
The blue team's live check: re-derive the exact gradient with and without the baseline, and re-measure both variances. It confirms unbiased + variance-reduced. If red makes the baseline depend on the action, this badge catches it and turns red.
A single state s with a small action set (K=3). Two things are fed in: the actor's logits z (which set the policy π=softmax(z)) and the action values Q(s,·) the critic estimates. Everything else is computed.
| symbol | meaning | role |
|---|---|---|
| z | policy logits | ACTOR |
| π=softmax(z) | the policy | ACTOR |
| Q(s,·) | action values | CRITIC |
| V(s)=Σπ·Q | state value | CRITIC |
| A=Q−V | advantage | the baseline term |
The critic supplies V(s) as the baseline; a legal baseline may depend on the state s but never on the action a. That single rule is the whole game — and it is what you feed the panel below.
Actor-critic: subtract the critic's V(s). Same gradient, less noise.
Move any slider — π, V, the advantage, the gradient and both variances are computed on the spot from the score function, never looked up.
What the machine proves, exactly and live: (1) the advantage has zero mean under π — Eπ[A]=0 to 1e-12; (2) subtracting the state-value baseline leaves the gradient unbiased — identical to REINFORCE to 1e-9; (3) it strictly reduces the estimator variance. At the default state: Var drops 0.803 → 0.252 (a 68.7% cut), matched by a fixed-seed Monte-Carlo.
The blue team's witness (left) re-derives these live; the red team (right) tries to make the baseline cheat.
Worse in practice: the critic is itself learned and biased — a lagging or wrong V(s) can raise variance and, when the advantage is bootstrapped (TD), inject bias the unbiasedness proof here does not cover. This sphere proves the constant-baseline case exactly; the deep on-policy actor-critic inherits all the critic's estimation error on top.
"Subtracting a baseline changes the gradient direction." Cut. The expectation is identical (E[score]=0); only the variance moves. Proven live in the panel — the gradient vector does not budge.
"The baseline must be the value function." Cut. Any action-independent baseline is unbiased. V(s) is merely convenient and near-optimal — a constant, or an average return, works too.
"An action needs positive Q to be reinforced." Kept, corrected. It needs positive advantage: a below-average action with Q>0 is pushed down, an above-average action with Q<0 is pushed up.
The red team's move: make the baseline depend on the action — set b(a)=Q(a) itself. Now the subtracted term correlates with the score, E[score·b]≠0, and the gradient is biased (it collapses to zero). The blue team's witness (window 7) is watching.
An action-dependent baseline is illegal: it no longer estimates ∇J. The witness re-derives the gradient, finds it disagrees with REINFORCE, and turns red. Nothing is faked — the attack is real and it is caught.