Long-context training can shift knowledge toward context dependence
The Information Abundance Paradox reports an intermediate optimum and robustness tradeoff under tested training regimes.
The Information Abundance Paradox asks whether plentiful relevant training context reduces the incentive to store that information parametrically. In the paper’s pretraining comparisons, increasing context improves evaluated tasks up to an intermediate optimum, after which performance declines.
In supervised fine-tuning, additional relevant training context improves performance when supporting context is present but reduces robustness when it is missing or misleading. The authors analyze a shift in gradient pressure from feed-forward networks toward attention and test interventions on that mechanism.
This is not proof that every model must choose between knowledge and context, or that attention is universally cheaper than storage. The result concerns specific training regimes and distributions.
An adoption test should therefore separate supported-context performance from closed-book and misleading-context performance. Compare models trained or adapted under the intended evidence conditions; a larger context window alone is not a substitute for that evaluation.
Editorial correction (2026-10-13 UTC): Restored conditional training and evaluation scope rather than a universal knowledge/context prohibition.