Give me a linear plant and a quadratic bill for state and effort, and I will hand you the one best feedback law in the world: u = -Kx. Kalman showed the gain is not tuned by hand — it falls out of a single matrix equation, the algebraic Riccati equation. Solve it once and you own the optimal controller forever. This is the seed of reinforcement learning, in closed form. Down the center, data flows: the cost matrices go in, the Riccati solver runs, the gain and its proof come out. The blue team builds and defends it; the red team tries to break it.
source R. E. Kalman, Contributions to the Theory of Optimal Control, Boletín de la Sociedad Matemática Mexicana 5 (1960) 102–119 — no DOI / stable publisher link (AMBER); scanned copy at ee.iitb.ac.in/~belur/ee640/optimal-classic-paper.pdf. Rendered, not quoted.
Minimize the cost J = ∫₀∞ (xᵀQx + uᵀRu) dt for the plant ẋ = Ax + Bu. The optimum is linear state feedback:
u = -Kx, K = R⁻¹BᵀP, where P is the stabilizing solution of the algebraic Riccati equation
AᵀP + PA − PBR⁻¹BᵀP + Q = 0
This instrument uses a constructed, closed-form-checkable plant — a diagonal 2×2 with known eigenvalues, so P, K and the closed loop are all exact to 1e−8:
The Riccati equation is the Bellman equation for a linear plant and a quadratic cost. The value function is exactly V(x) = xᵀPx; Bellman's backup, specialized to that quadratic, collapses to the Riccati recursion — and value iteration on it converges to the same P (window 7 proves it live).
Where the Bellman equation is the general dynamic-programming law, the LQR is its one case that solves in closed form — the classical root of reinforcement learning. Each sphere is the next one's premise.
The blue team's live check: re-solve the Riccati equation, confirm A−BK is stable, that x₀ᵀPx₀ equals the actual closed-loop trajectory cost, and that the Bellman backup converges to the same P. If red drops the P factor, this badge is where it shows.
You do not design the gain; you design the cost. Two matrices and a start state are the whole input:
| knob | meaning | here |
|---|---|---|
| Q ≽ 0 | price on being off-target (state x) | diag(12, 8) |
| R ≻ 0 | price on control effort (u) | ρ · I |
| x₀ | initial state to regulate to 0 | [1, 1] |
Big R = expensive actuators → a gentle, low-gain law. Big Q = intolerance of error → an aggressive, high-gain law. Turn the ρ knob below and watch the gain answer.
The gain K = R⁻¹BᵀP is recomputed from the Riccati P on every change — never looked up. Raising ρ makes control dear, so K shrinks.
The optimal cost-to-go from x₀ is the quadratic form x₀ᵀPx₀ — and it is checked against a Simpson integral of the real closed-loop trajectory to 1e−9.
What the machine produces, proven at ρ=1: P = diag(12, 2), gain K = diag(6, 2), and a closed loop A−BK = diag(−2, −3) — the once-unstable plant driven to eigenvalues {−2, −3}, both in the left half-plane. The optimal cost from [1,1] is 14, matching the trajectory integral exactly.
The blue team's witness (left) confirms these numbers live; the red team (right) tries to make them wrong.
Worse for the legend of robustness: full-state LQR has guaranteed gain and phase margins, but the moment you add a Kalman filter to estimate the hidden state, the combined LQG controller can have arbitrarily small margins — Doyle's 1978 "Guaranteed Margins for LQG Regulators" is a half-page proof that the guarantee evaporates. Optimal is not the same as robust.
"LQR is always robust — 60° phase margin, guaranteed." Cut. That is the full-state loop only. Close it through an observer (LQG) and Doyle (1978) shows the margins can be made as small as you like. The guarantee is real, and narrow.
"The Riccati equation always has a solution." Cut. A stabilizing P exists only when (A,B) is stabilizable and (A,√Q) is detectable. Break those and the solver has no stabilizing answer.
"K minimizes cost by making the loop as fast as possible." Kept, corrected. K balances state error against control effort; raise R and the optimal loop deliberately gets slower — watch the gain shrink in the panel.
The red team's move: use K = R⁻¹Bᵀ — drop the P factor from the Riccati gain. It looks like a gain; it is not the optimal one. The blue team's witness (window 7) is watching.
Without P, the "gain" no longer stabilizes the plant — a closed-loop eigenvalue slides to +0.75, out of the left half-plane — and the cost-to-go stops equaling x₀ᵀPx₀. The witness recomputes, disagrees, and turns red. Nothing is faked; the attack is real and it is caught.