MemFold: Learning Compact Soft Memory for Long-Context Personalization via On-Policy Optimization

arXiv · Jingxuan Wu, Yuzhe Yang, Yiqiao Huang, Chengzhi Liu, Qingni Wang, Chengxuan Qian, Shutong Wu, Jiawei Zhang, Xin Eric Wang · 2026-10-04

This item is no longer on the current shelf. Its page remains available, and editorial corrections may be made. See what is featured now.

MemFold compresses long-term user memory into a fixed-size soft vector bank, boosting personalization accuracy while staying within strict context budgets.

What it does

MemFold introduces a query-conditioned soft memory that stores user history in K continuous vectors, forming the reader’s memory interface. The reader is then trained on its own rollouts using two signals: group-relative rewards for task outcomes and confidence-gated on-policy distillation from a frozen textual-memory teacher. The teacher rescales sampled tokens without being sampled itself, so supervision stays on the student’s current distribution and adds no decoding overhead. At inference time the teacher is removed, leaving only the compact soft memory. Across three Qwen backbones, MemFold achieves the highest accuracy on PersonaMem-32K and PersonaMem-128K, with larger margins as history length grows, and transfers to PrefEval and LongMemEval without domain-specific fine-tuning.

Why it matters

Long-context personalization traditionally forces the model to re-read ever-growing text, inflating latency and memory use. By compressing history into a fixed-size latent bank that is directly optimized for downstream behavior, MemFold delivers higher accuracy without sacrificing speed. The approach also generalizes to other long-context tasks, suggesting a reusable memory module for any assistant that must remember across thousands of tokens.

How it applies

Implementers can swap the textual context window with a MemFold-style soft memory layer, training the downstream reader with on-policy rewards that reflect the actual task. Because the teacher is only needed during training, inference remains lightweight, fitting easily into existing deployment pipelines that already use fixed-size context windows.

The short version

Your assistant shouldn’t have to reread a novel every time you ask a follow-up.

My take

I think the biggest shift here is moving memory from a passive store to an active, task-driven component. When the memory is trained with the same reward signals that drive the final assistant, you get a tighter loop between what the user wants and what the model remembers. The fact that it works across multiple backbones and transfers without extra data makes it a plug-and-play upgrade for existing pipelines. In short, if you’re hitting context walls, MemFold gives you a way to keep the conversation going without re-architecting the whole model.

How it connects

• Trend toward modular, memory-augmented LLM architectures.
• Increasing demand for cost-effective long-context handling in chat assistants.
• Shift from reconstruction-based compression to behavior-oriented memory learning.

Bottom line

Replace raw text context with a MemFold-style soft memory to extend personalization horizons without expanding prompt size.

Takeaways

• Soft memory is optimized for the behavior it supports, not just reconstruction.
• Confidence-gated on-policy distillation provides supervision without extra decoding steps.
• MemFold outperforms baselines on PersonaMem-32K and PersonaMem-128K, with gains increasing for longer histories.
• The method transfers to PrefEval and LongMemEval without additional training data.
• Three Qwen backbones validate that the technique is backbone-agnostic.

Topics: long-context memory, soft memory, on-policy optimization, personalization, LLM efficiency