How much a single sample tells you about a parameter — and the hard floor it sets on any estimator's variance. Fisher's insight: the curvature of the log-likelihood at its peak is information. A sharp peak pins θ down; a flat one leaves it loose. Define it two ways — the mean-square of the score, or the negative expected second derivative — and they are equal. From that equality falls the Cramér–Rao bound: no unbiased estimator can beat 1/(n·I(θ)). Terms go in, the engine computes I two ways and the bound, the proof comes out.
source R. A. Fisher, "Theory of Statistical Estimation," Proc. Camb. Phil. Soc. 22 (1925) 700–725 — cambridge.org/core · 1925. Rendered, not quoted.
Write the log-likelihood ℓ(θ)=log p(x;θ). Two objects live on it:
score = ∂ℓ/∂θ — the slope. Its expectation is zero at the truth (∫∂p = ∂∫p = ∂1 = 0). Its mean-square is the information: I(θ)=E[(∂ℓ/∂θ)²].
curvature = ∂²ℓ/∂θ² — bends downward at the peak. Differentiate the zero-score identity once more and you get the second face: I(θ)=−E[∂²ℓ/∂θ²]. The two faces are the same number.
For the current model, both faces computed live:
I(θ) is the curvature of the maximum-likelihood peak. The MLE sits where the score is zero; how sharply ℓ falls away there — the second derivative — is exactly the observed information, and its expectation is I(θ).
Sharper peak → more information → smaller variance floor. Multiply the samples by n and the curvature scales by n (independence adds information), so the bound shrinks as 1/(n·I). It is also the metric the natural gradient descends — distance in θ measured in units of information. Each sphere is the next one's premise.
The blue team's live check: re-run both definitions across a grid of parameters and confirm they agree, that the score is mean-zero, and that the sample mean achieves the Cramér–Rao bound. If red tampers, this badge is where it shows.
Pick a parametric family and a value of its parameter. Two textbook families carry it, both exponential-family models where the sample mean is the natural sufficient statistic:
| family | ℓ(θ) | score ∂ℓ/∂θ | I(θ) |
|---|---|---|---|
| Bernoulli(θ) | x·log θ + (1−x)·log(1−θ) | x/θ − (1−x)/(1−θ) | 1/(θ(1−θ)) |
| Gaussian mean μ, σ known | −(x−μ)²/2σ² + c | (x−μ)/σ² | 1/σ² |
Bernoulli's support is discrete — its expectations are exact finite sums over {0,1}. The Gaussian's are exact Gauss–Hermite quadratures (the integrands are low-degree polynomials, so the rule is exact). That is what you feed the panel below.
The log-likelihood for a fixed-seed sample of size n — its curvature at the peak is n·I(θ̂):
Move any control — I(θ) is recomputed from the two definitions on the spot, never looked up.
What the machine proves: the two definitions of I(θ) agree exactly, the score is mean-zero, and every unbiased estimator obeys Var ≥ 1/(n·I(θ)) — a floor the sample mean achieves with equality for both families. Current numbers:
The blue team's witness (left) confirms these live; the red team (right) tries to make them wrong.
It also assumes the model is true. Under misspecification the inverse information is replaced by Huber's sandwich. And it only binds unbiased estimators — James–Stein and ridge deliberately take bias to beat it in mean-squared error. I(θ) is a property of the model+parameter, not of the data.
"The MLE always achieves the Cramér–Rao bound." Cut. Only asymptotically in general. A finite-sample efficient estimator exists only for exponential families — which is exactly why the sample mean hits equality here and rarely elsewhere.
"Fisher information measures the information in the data." Corrected. It measures how sharply the likelihood constrains θ near a value. Reparametrize φ=g(θ) and it rescales by (dθ/dφ)² — it lives in the parameter's metric, not the sample.
"More data divides the bound by n by definition." Kept, corrected. I_n(θ)=n·I₁(θ) is a theorem — additivity of information under independence — not a definition. Correlated samples do not simply add.
The red team's move: define I as +E[∂²ℓ/∂θ²] — drop the minus sign on the curvature. Curvature is negative at a peak, so I turns negative, the two definitions split, and the bound 1/(n·I) goes meaningless. The witness (window 7) is watching.
Drop the minus and I(θ)<0: the mean-square-score form (always ≥0) and the curvature form no longer match, and a negative "floor" is no bound at all. The witness recomputes, disagrees, and turns red. Nothing is faked; the attack is real and it is caught.