Optimizing many variables at once is hard — so don’t. Freeze all but ONE variable, slide it to its best value, then move to the next, and cycle. Each move is a trivial 1-D problem, and the path staircases down toward the minimum. It’s how LASSO regression and many large-scale fits are actually solved — simple sub-steps, no full gradient. Slide to alternate axes and watch it stairstep to the bottom.
Coordinate descent minimizes a multivariate function by optimizing ONE coordinate at a time, holding the rest fixed, and cycling through them. Each sub-problem is a cheap 1-D minimization (often closed-form), and the iterate staircases toward the optimum — no full gradient or step-size tuning needed. It CONVERGES for smooth convex functions and for separable-plus-smooth ones (the L1 penalty in LASSO, where its soft-thresholding step is exact), making it a default for high-dimensional regularized regression and many large sparse problems. A fail-loud self-check throws unless alternating axis-minimizations reach the known minimum. ◆ real optimization, node-verified.
It can STALL on non-smooth, non-separable functions (getting stuck where no single-axis move helps, though the joint optimum lies diagonally) and axis-aligned steps are slow on correlated variables. The 1-D exact-minimization steps are exact.