Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs

arXiv · Prakhar Khatri · 2026-09-04

This is no longer on the current shelf — shelves rotate as new material clears the bar. The analysis below is unchanged. See what is featured now.

A token‑budget playbook that lets multimodal LLMs actually watch hours‑long videos without blowing up the GPU.

What it does

The paper introduces a three‑stage pipeline – Select, Compress, Reinvest – that dynamically allocates visual tokens across a long‑video timeline. It studies how frame‑selection policies, token‑compression (e.g., pooling, low‑rank factorization), and token‑reinvestment (feeding saved budget back into high‑interest segments) affect downstream tasks like video QA and captioning. The authors run a controlled ablation on a 10‑hour video benchmark and release a lightweight token‑allocator library that plugs into any existing MLLM.

Why it matters

Long‑form video has been a blind spot for current MLLMs because the quadratic token cost forces aggressive down‑sampling, which throws away critical context. This work shows a principled way to keep the important moments while staying within a fixed compute budget, directly impacting product‑scale agents that need to understand surveillance feeds, tutorials, or meeting recordings.

How it applies

You can drop the token‑allocator into your existing vision‑LLM stack, set a global token budget, and let the system automatically decide which frames to keep raw, which to compress, and where to spend the saved tokens for deeper reasoning. The library also exports a simple API for custom importance scoring (e.g., motion, OCR, audio cues).

Takeaways

• Dynamic token allocation outperforms static uniform sampling by 12‑18% on long‑video QA.
• Compress‑then‑reinvest yields a better trade‑off than compress‑only or select‑only pipelines.
• A cheap importance model (motion + audio energy) is sufficient for most domains; you only need a heavy vision encoder for the reinvested segments.
• The method scales linearly with video length and fits in a single 24 GB GPU.
• Open‑source token‑allocator released under Apache 2.0.

Topics: multimodal, long‑video, token‑efficiency, LLM‑infrastructure