Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs

arXiv · Pavel Tikhonov, Anton Korznikov, Matvey Mikhalchuk, Nikita Dragunov, Temurbek Rahmatullaev, Polina Druzhinina, Anton Razzhigaev, Ivan Oseledets, Elena Tutubalina · 2026-09-27

This item is no longer on the current shelf. Its page remains available, and editorial corrections may be made. See what is featured now.

LLMs secretly obey linear superposition, letting a single forward pass spawn two coherent continuations.

What it does

The authors show that when distinct text streams are linearly combined, a Transformer produces a superposition of the individual next-token distributions, supporting a Superposition Linearity Hypothesis. They demonstrate that this linearity is an intrinsic property of the architecture, not just an emergent training effect, and that it weakens as pretraining proceeds. Lightweight fine-tuning can restore much of the lost linearity, narrowing the gap between the superposed output and the average of separate outputs. Finally, they propose a guided decoding method that disentangles the superposed logits, enabling two separate, fluent continuations from one pass.

Why it matters

If linear superposition can be reliably recovered, engineers can cut inference latency by generating multiple candidate streams in parallel, a huge win for real-time applications. The guided decoding recipe offers a concrete way to harvest those parallel streams without sacrificing coherence.

How it applies

Practitioners can apply the fine-tuning recipe to existing Transformer checkpoints to regain superposition, then plug the guided decoder into their inference pipeline to obtain dual outputs per token step. This is especially useful for tasks like brainstorming, divergent generation, or simultaneous multi-agent dialogue.

The short version

What if your LLM could think in two directions at once, for the cost of one?

My take

I see this as a practical lever for scaling LLM-based products without buying more GPUs. The fact that the linearity is architectural means it should transfer across model families, so you don’t need to rebuild the trick for every new checkpoint. The guided decoder is lightweight enough to slot into existing serving stacks, making it a low-risk experiment for any team that needs higher output rates.

How it connects

• Fits the broader push to squeeze more utility out of a single inference pass (e.g., token-level caching, speculative decoding).
• Echoes recent findings that Transformers retain surprising linear structures despite non-linear activations.
• Provides a concrete bridge between theoretical linearity hypotheses and production-grade inference optimizations.

Bottom line

Try fine-tuning your model with the paper’s recipe and add the guided decoder—you could double your generation bandwidth with almost no extra hardware.

Takeaways

• Transformers exhibit a measurable linear superposition of next-token distributions when inputs are linearly blended.
• The effect diminishes with continued pretraining but can be largely restored with lightweight fine-tuning.
• Guided decoding can separate the superposed logits into two distinct, high-quality continuations.
• Superposition offers a path to parallel generation without extra forward passes.

Topics: transformers, linear superposition, guided decoding, inference efficiency