Nothing in an image moves — only its brightness shifts. From that shift a machine recovers motion, one linear equation per pixel: Ix·u + Iy·v + It = 0. But that is one equation in two unknowns — a single pixel can never solve it. Lucas & Kanade pool a whole window of pixels into a least-squares system, and the same little 2×2 matrix that makes a corner trackable makes a bare edge impossible. Down the center, data flows: the frames go in, the window is solved, the flow comes out. The blue team builds it; the red team breaks it.
source B. D. Lucas & T. Kanade, An Iterative Image Registration Technique with an Application to Stereo Vision (1981), Proc. 7th IJCAI, vol. 2, pp. 674–679 — hal.science/hal-03697340 (IJCAI proceedings carry no DOI; archival copy). Rendered, not quoted.
Assume brightness is conserved along the motion: a point keeps its grey value as it moves. Taylor-expand and every pixel hands you one row Ixu + Iyv = −It. Stack a window of them and least-squares gives the normal equations:
A = [ ΣIx² ΣIxIy ; ΣIxIy ΣIy² ]
b = [ −ΣIxIt ; −ΣIyIt ]
A·[u,v]ᵀ = b
A is the structure tensor — the same matrix Harris uses to find corners. Flow is recoverable only when it is invertible.
Live structure tensor for the current window:
| quantity | value |
|---|
Motion from brightness — Lucas & Kanade, 1981. The whole method rides on one matrix, and that matrix is shared with the-harris-corner.
Two large eigenvalues → a corner → both flow components pinned → trackable. One large, one zero → a straight edge → only the normal component pinned → the aperture problem. Zero and zero → a flat wall → nothing. A good feature to track and a solvable flow are the same 2×2 question. Each sphere is the next one's premise.
The blue team's live check: re-solve the canonical textured window and confirm it recovers the planted flow (0.700, −0.400), and that the edge tensor is singular. If red tampers, this badge is where it shows.
The input is a pair of frames and a window. From frame 1 the spatial gradients Ix, Iy (how brightness changes across space); between the frames the temporal gradient It (how it changes in time). A textured corner gives gradients in many directions; a single edge gives them all in one direction — and that difference decides everything downstream.
Pick the true motion and the window type; the panel below computes the gradients over a 9×9 window and solves live.
frame 1, grey; ◻ window; → true flow, → recovered.
Gradients, structure tensor and the 2×2 solve are all computed on the spot from the two frames — never looked up.
What the machine produces, proven: on a well-textured window the least-squares solve recovers the true motion to 1e-9; on a straight edge it refuses — the structure tensor is singular and only the flow normal to the edge survives. The current window's result is above; the guarantee is the output.
The blue team's witness (left) confirms this live; the red team (right) tries to make a flat pixel speak.
It also assumes small motion (the Taylor step) and constant illumination — it breaks on large displacement, brightness changes, occlusion, and transparency. Pyramids (coarse-to-fine) and global smoothing (Horn–Schunck, 1981) patch the range and the flat regions, but the point-wise ambiguity is a property of the world, not the algorithm.
"Optical flow is the true motion of objects." Cut. It is the motion of brightness patterns. A rotating barber-pole flows upward; a shadow moves with no object behind it. Real motion and apparent flow only coincide with texture and constant light.
"One pixel is enough to find the flow." Cut. One pixel is one equation, two unknowns. You need a window with two independent gradient directions — exactly what the tamper (window 6) removes.
"Lucas–Kanade assumes nothing." Kept, corrected. It assumes brightness constancy, small motion, and a locally constant flow across the window. Name the assumptions and it is honest; hide them and it lies.
The red team's move: solve the flow from a single pixel's one equation instead of the whole window. One constraint, two unknowns — the system is under-determined and the recovery is wrong even on a rich texture. The blue team's witness (window 7) is watching.
Shrink the window to one pixel and the structure tensor collapses to rank 1 — non-invertible. The witness recomputes, fails to recover the planted flow, and turns red. Nothing is faked; the attack is real and it is caught.