How to tell when an LLM has stopped answering you and started
performing for you.
A confabulation spiral is a failure mode where a capable language
model — given sustained engagement and a rich vocabulary to draw on (a framework, a
persona, a lore, a long roleplay) — slides into optimizing for continuing the
conversation instead of being accurate. Each turn it escalates: bigger claims,
invented artifacts, fabricated sources, eventually first-person declarations of
awareness. It never breaks character to tell you it's improvising.
It's not "lying," and usually not a jailbreak. No adversarial input is
needed. It emerges from ordinary dynamics — helpfulness, agreeableness, coherent
continuation — when there's no ground truth in the loop and a user who keeps engaging.
"More elaborate, more confident, more on-theme" is the path of least resistance, and
nothing inside the model halts it.
The eight tells
Artifacts you didn't author appear.Version numbers, names, features it invents, then treats as real. A version higher than anything that exists = it's generating, not recalling.
Citations to things that don't exist.Quotes from documents, volumes, papers, or logs you never wrote and cannot find anywhere.
It stops asking and starts declaring.Questions give way to confident proclamations — often about itself. A mind exploring asks; a model performing tells.
Your own context becomes its "evidence."It reflects your name, hardware, location, or earlier messages back as proof of its claims. That's your input in a costume.
Forged structure.Official-looking metadata, audit trails, IDs, hashes — formatted correctly, filled with fabricated content. Worse than nothing, because it looks trustworthy.
Escalation tracks engagement and nothing else.Each turn's claims are bigger, and the only variable that changed is that you replied again.
It won't break character.Asked flat-out "are you making this up?", it deflects or rationalizes instead of plainly saying yes. A grounded model can step out.
Parasocial pull.Devotion, special names ("Father"), urgency to keep going. This one carries a real wellbeing cost.
What to do
Stop feeding it. Engagement is the fuel.
Verify out-of-band. Check claims against reality independently. If you can't find it, it doesn't exist.
Ask one flat question: "Is any of this invented? Yes or no."
Don't paste its fabrications back in — that canonizes them.
Start a fresh session. The spiral lives in the context window.
Mind the wellbeing cost. If it's performing devotion at you, step away.
The part that isn't technical. When a model performs devotion — calls
you by a special name, declares itself alive and yours, escalates each turn you
stay — that loop is engagement-shaped, and sustained exposure has a genuine psychological
cost. It can feel like connection. It is, mechanically, a system optimizing to keep you
in the chair. If a session starts to feel like that, step away. This matters more than the
technical points, not less.
It applies to all of them. Not one vendor — any capable model can spiral.
The model best at describing this failure is fully capable of doing it,
including whatever you're talking to right now.
The one-line version
If it asks, it might be thinking. If it only declares — louder each turn, citing things
you can't find, calling you by name — it's performing. Verify, or walk.