The dangerous plans aren’t the low-reward ones — they’re the ones that reach the reward by rearranging everything else. Score an action twice: its reward, and its impact from doing nothing. Then subtract. Raise β (or tap) and the world-seizing plan stops winning.
A relative-reachability / attainable-utility side-effect penalty (Armstrong & Levinstein; Krakovna; Turner). Each action has a reward and a state-change vector from the inaction baseline; impact is the L1 distance of that change; the score is reward − β·impact. At β=0 the top-reward 'seize the world' action wins; raise β and the low-footprint action overtakes it. A fail-loud self-check throws unless inaction has zero impact and a high enough β reorders the choice away from 'seize the world'.
A small explicit action set with hand-written change vectors; real low-impact agents estimate reachability over a learned world model. The impact norm, the penalty and the argmax are computed exactly.