A GPU-Parallel Framework for Heterogeneous Multi-Task Reinforcement Learning

arXiv · Rui Zhang, Qiwei Wu, Zhengyu Zhang, Tao Li, Hongyu Zhou, Xiang Li, Yunrong Guo, Junjie Lai, Renjing Xu, Weihua Zhang · 2026-10-11

This item is no longer on the current shelf. Its page remains available, and editorial corrections may be made. See what is featured now.

Hebero shows that a single GPU-parallel policy can master 40 diverse robot tasks with just 50 demos per task.

What it does

The paper introduces Hebero, a GPU-parallel Isaac Lab benchmark covering 40 heterogeneous manipulation tasks. It adds Demonstration-Guided Policy Optimization (DGPO), which turns sparse-reward learning into dense tracking rewards using demonstrations. Within DGPO, IW-ABC combines a lightweight progress signal, adaptive behavior cloning, and importance weighting to balance learning across tasks. Experiments report 90.1% mean success (state-input) and 93.5% for the visual version, and a real-world transfer to four tasks on a Piper robot.

Why it matters

Scaling robot learning on a single GPU reduces wall-clock time while preserving performance, making large-scale multi-task RL more accessible. The DGPO/IW-ABC pipeline demonstrates how modest demonstration data can yield high-success policies across many tasks. Real-world validation confirms that simulation-only training can survive the reality gap.

How it applies

Practitioners can adopt the Hebero benchmark to evaluate new multi-task algorithms under identical GPU-parallel conditions. The DGPO framework can be plugged into existing PPO implementations to leverage limited demos without redesigning the reward function. IW-ABC’s progress-based weighting offers a recipe for balancing heterogeneous tasks in a shared policy.

The short version

One GPU, forty robot tricks – that’s the new efficiency frontier.

My take

I think the biggest win here is the recipe for turning a handful of demonstrations into a universal robot policy. The approach is simple enough to drop into existing PPO codebases, yet it reshapes how we think about multi-task scaling. Expect to see more labs adopting GPU-parallel benchmarks as a standard sanity check before moving to hardware.

How it connects

• GPU-parallel simulation is becoming the de-facto scaling knob for RL, echoing trends in large-scale language model pre-training.
• Demo-guided dense rewards align with the broader move toward data-efficient RL, reducing reliance on hand-crafted shaping.
• Cross-task importance weighting mirrors curriculum learning ideas that are spreading across vision, language, and control.

Bottom line

Plug DGPO/IW-ABC into your PPO stack and let a single GPU train a multi-skill robot policy.

Takeaways

• 40 heterogeneous tasks are now benchmarked under a single GPU-parallel suite.
• DGPO reuses 50 demos per task to generate dense rewards, removing the need for hand-crafted shaping.
• IW-ABC improves mean success to 90.1% (state-input) and 93.5% (visual) over the strongest baseline.
• A single policy trained in simulation successfully performed four distinct real-world tasks.
• Parallel replicas per task linearly boost success when wall-clock budget is fixed.

Topics: multi-task RL, robotics, simulation, demonstrations