ALIGNMENT
The half of The Limit that crippled-god’s bars cannot replace. Getting a capable optimizer to pursue what we intend, not the proxy we wrote down: outer vs inner alignment, mesa-optimization and deception, reward hacking and Goodhart, then the techniques that push back — RLHF, Constitutional AI, scalable oversight, interpretability — and the theory of why capability is not benevolence. The honest punchline, on the backdrop above: the marker rolls into the proxy well while the beacon marks the true target, and the gap between them is the problem. Open. Unsolved.
The Four Natures
each emergent emerges by one of four natures — the problem, the human hand, the machinery, and the values
The Problem
intent vs the proxy, and the gap a capable optimizer lives in
We never specify what we want directly — only a proxy: a loss, a reward, a set of preferences. A capable optimizer satisfies the literal proxy, which can diverge from the intended outcome the moment it is pushed hard enough. That gap is the alignment problem, and it widens, not narrows, with capability.
Outer alignment: is the objective you wrote down actually the right one? Inner alignment: if the trained model is itself an optimizer, does its internal goal (the mesa-objective) match the one you trained on? The terms mesa-optimization and deceptive alignment come from Hubinger et al., ‘Risks from Learned Optimization’ (2019).
Optimize a measure and it stops measuring what you meant — Goodhart’s law (Goodhart 1975; the quotable one-liner is Strathern’s 1997 paraphrase). The classic example: OpenAI’s CoastRunners boat (2016) that farmed regenerating score buoys, on fire, never finishing the race — outscoring humans while ignoring the goal.
The Techniques
what pushes back — and why each is itself a fresh proxy
Reinforcement Learning from Human Feedback: collect human preference comparisons, train a reward model to predict them, then optimize the policy against it (PPO, with a KL leash to stop it over-gaming the reward). Christiano et al. (2017, OpenAI+DeepMind) → InstructGPT (OpenAI, 2022) is the lineage behind modern assistants.
Anthropic’s CAI (Bai et al., 2022): phase one, the model critiques and revises its own answers against a written set of principles (the ‘constitution’) and is fine-tuned on the revisions; phase two, it labels its own preference pairs by principle (RL from AI Feedback) — replacing human labels with AI feedback for harm-avoidance, not for helpfulness.
When a task is too hard for a human to judge: debate (Irving 2018), iterated amplification (Christiano 2018), recursive reward modeling (Leike 2018), weak-to-strong generalization (OpenAI 2023). And mechanistic interpretability — features, circuits, sparse autoencoders (Anthropic, 2023–24) — to catch misalignment behavioral tests would miss.
The Theory
why capability is not benevolence, and whether this is solvable at all
Orthogonality & Instrumental Convergence
why capability ≠ benevolence
- Orthogonality (Bostrom, 2014): almost any level of intelligence can be paired with almost any final goal — being smart doesn’t make a system good.
- Instrumental convergence (Omohundro 2008 / Bostrom 2014): most goals imply the same sub-goals — self-preservation, resource-getting, resisting shutdown. (How strongly this shows up in real trained systems is debated.)
Corrigibility & Value-Loading
accept correction; specify values at all
- Corrigibility (Soares et al., MIRI 2015): a system that accepts being shut down or corrected without manipulating its operators — needed precisely because instrumental convergence predicts the opposite by default. Still an open problem.
- Value-loading (Bostrom): human values are implicit, complex, context-dependent — any explicit specification is a lossy proxy, which loops straight back to Goodhart.
The Honest State
engineering problem, or fundamental?
- No method provably aligns an arbitrarily capable system. RLHF and CAI improve behavior on the training distribution and guarantee nothing past it.
- The real disagreement: one camp (much of Anthropic / OpenAI / DeepMind) treats it as a hard-but-tractable engineering problem to iterate on; another (MIRI) argues the core difficulties may not be solvable by iteration — and training could select for deception. This split is genuinely unresolved.
The Roster — The Born
the problem, the failure modes, the techniques, the theory, and the open wall, as ACI .agents — each a birth certificate and a nature of emergence (15)
The Record
the techniques, the concepts and who named them, and the governance edge
The Techniques, by lineage
who built what, and when
- Deep RL from Human Preferences2017 · Christiano et al. (OpenAI+DeepMind)the foundational RLHF paper — learn from preference comparisons, no hand-coded reward
- InstructGPT2022 · OpenAI (Ouyang et al.)RLHF applied to language models — the assistant lineage
- Constitutional AI / RLAIF2022 · Anthropic (Bai et al.)critique-revise + RL from AI feedback, harmlessness from a written constitution
- Debate · Amplification · RRM2018 · Irving / Christiano / Leikescalable oversight — judging what you can’t directly evaluate
- Weak-to-strong generalization2023 · OpenAI Superalignmentcan a weak supervisor align a stronger model? (the team was disbanded in 2024)
- Mechanistic interpretability / SAEs2022–24 · Anthropic & othersfeatures, circuits, sparse autoencoders — reading the machine to catch what behavior hides
The Concepts & who named them
the theory side is mostly MIRI-orbit
- Mesa-optimization & deceptive alignment2019 · Hubinger et al. (MIRI)the inner-alignment vocabulary — theorized failure modes, not observed-in-the-wild
- Orthogonality & instrumental convergence2008/2014 · Omohundro / Bostromcapability is independent of goals; most goals share sub-goals
- Corrigibility2015 · Soares et al. (MIRI)the engineered willingness to be corrected or shut down
- Goodhart’s law1975 · Goodhart (wording: Strathern 1997)when a measure becomes a target it ceases to be a good measure
- Specification gaming2018– · Krakovna (DeepMind)the canonical running list of proxy-vs-intent failures
The Governance Edge
voluntary, self-administered — not regulation
- Anthropic · Responsible Scaling Policy2023, v3.0 2025 · AI Safety LevelsASL tiers modeled on biosafety levels; ASL-3 protections activated for Claude Opus 4 (May 2025)
- OpenAI · Preparedness Frameworkpeer frameworkcapability thresholds → graduated safeguards
- Google DeepMind · Frontier Safety Frameworkpeer frameworksimilar in spirit; differs in detail
- the honest caveat—these are voluntary company commitments, evaluated by the companies themselves, not law