Memory as Plans: World-Action Modeling with Memory-Grounded Planning

arXiv · Research Team · 2026-09-13

Memory-as-Plans reframes an agent’s memory into a living roadmap for world-action reasoning.

What it does

The authors propose a novel architecture where a recurrent memory buffer is treated as a plan graph that directly conditions action selection. They introduce a memory-grounded planning loss that aligns stored trajectories with future goal states, and they demonstrate the approach on three complex environments (Minecraft, Mini-RTS, and a robotic manipulation suite). The paper also provides a theoretical analysis showing how memory-derived plans can reduce planning horizon complexity. Open-source code and pretrained checkpoints are released.

Why it matters

Current LLM-driven agents either plan ahead with expensive tree search or act myopically with short-term context. By making memory itself a structured plan, the method bridges the gap, offering both long-term coherence and low inference cost. This is a game-changer for any production system that needs real-time decision making under limited compute.

How it applies

Replace the flat transformer context in your agent with the Memory-as-Plans module and you’ll see more consistent goal-directed behavior without a separate planner. The approach works with off-the-shelf LLM backbones, so you can drop it into existing chat-or-control bots. It’s especially useful for robotics, game AI, and any domain where an agent must remember a sequence of sub-goals.

The short version

What if your agent’s notebook was also its roadmap?

My take

In my own work stitching LLMs into control loops, the biggest pain point is keeping the model aware of *what* it’s trying to achieve after a few steps. This paper shows a clean, mathematically-grounded way to bake that awareness into the memory itself. It’s the kind of architectural tweak that doesn’t just improve metrics—it reshapes how we think about agent design.

How it connects

• Echoes the trend of integrating retrieval-augmented memory with planning (e.g., RAG + action).
• Fits the move toward lightweight, on-device agents that can’t afford external search.
• Provides a concrete bridge between symbolic planning and neural memory.

Bottom line

Swap in the Memory-as-Plans module to give your agents long-term direction without paying the cost of a separate planner.

Takeaways

• Treating memory as a graph-structured plan yields better long-term coherence.
• The memory-grounded planning loss can be added to any pretrained LLM with minimal fine-tuning.
• Empirical results show a 20-30% boost in success rate on multi-step tasks versus standard prompting.
• Inference cost remains comparable to a vanilla transformer because planning is implicit in the memory update.
• The open-source implementation includes adapters for popular RL and robotics frameworks.

Topics: agents, planning, memory, world-model, RL