OpenAI Astra and Recurrent Depth / Looped Transformers

Sebastian Raschka · 2026-09-02

▶ Play the video

This is no longer on the current shelf — shelves rotate as new material clears the bar. The analysis below is unchanged. See what is featured now.

Why it earned a slot

Sebastian's breakdown of the rumored recurrent depth design in Astra is the kind of architectural rabbit hole that keeps me up at night. The Nanbeige4.2-3B looped transformer comparison at the 3-minute mark is the strongest part — he actually walks through the mechanics instead of just waving hands at 'infinite context.' The low like rate is baffling given how dense this is; probably just the audience being too busy implementing to leave feedback.

The short version

Looped transformers might be the architecture that breaks the scaling wall.

Why it matters

If the Astra rumors are real, OpenAI is betting on recurrent depth as a way to scale test-time compute without exploding parameter count. This matters because it reframes the entire efficiency conversation — instead of bigger models, we might see models that think harder by reusing the same layers.

My take

The Mixture-of-Recursions paper and the Universal Transformer lineage are converging on something I've been tracking: dynamic depth as a first-class architectural choice, not a hack. When I build agentic reasoning loops, I'm essentially manually doing what these architectures automate — the question is whether latent reasoning through repeated layers actually produces better traces than chained inference. Sebastian's KV-cache implications section at the 10-minute mark is the practical crux.

How it connects

• Recurrent depth blurs the line between inference-time compute and model capacity — the next frontier might be 'how long does this think?' not 'how many parameters does it have?'
• If looped transformers work at scale, they fundamentally change the economics of running frontier models — cheaper per query, but potentially more expensive per training run.
• The recursive self-improvement angle at the 23-minute mark is where this gets genuinely unsettling for the industry.

Bottom line

Watch Sebastian's validation-loss comparison section closely — if looped depth actually beats standard transformers on downstream tasks, the entire architectural roadmap for 2027 shifts.

Topics: architecture, recurrent-depth, astra, transformers