Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone

arXiv · Chenyu Zhu, Ruoyu Zhao, Zhichao Lu · 2026-09-27

This item is no longer on the current shelf. Its page remains available, and editorial corrections may be made. See what is featured now.

Neural Spectral Capacity gives a closed-form, architecture-only score that outperforms Params and FLOPs for picking the best Transformer or CNN under a budget.

What it does

The paper defines Neural Spectral Capacity (NSC), a scalar derived from the singular-value spectrum of each weight matrix under random initialization. Using the Marchenko-Pastur law, NSC can be computed directly from a model's depth-width-head-FFN specification without instantiating the network, training data, or gradients. An exact dynamic-programming solver (NSC-DP) then finds the architecture that maximizes NSC under arbitrary resource constraints in seconds on a CPU. Empirical tests show NSC ranking correlates better than Params, FLOPs, or other training-free proxies across seven Transformer and CNN families, discovers a superior Transformer-XL on WikiText-103 in 2 s, and prunes LLaMA-7B to a 5.7 B model 5,900× faster than the strongest baseline.

Why it matters

For engineers who constantly juggle parameter budgets and compute limits, NSC offers a fast, theoretically grounded metric that respects architectural nuance. It removes the need for costly black-box searches or full model instantiation when exploring design spaces. The DP solver guarantees a globally optimal allocation, saving time and resources during early-stage model planning.

How it applies

When sizing a new LLM or vision model, you can compute NSC from the spec sheet and run NSC-DP to get the highest-capacity layout under your FLOP or memory budget. The same metric can guide pruning decisions, as shown with LLaMA-7B, without any calibration data. This lets teams iterate architecture ideas in minutes rather than days of training.

The short version

What if you could pick the best neural architecture without ever training a single model?

My take

I see NSC as the next-generation sizing tool for anyone constructing large-scale models. It lets you reason about capacity the way you’d reason about bandwidth in networking—precise, budget-aware, and provably optimal. The fact that it works across Transformers and CNNs suggests the underlying spectral view could become a universal design language for deep nets.

How it connects

• Spectral analysis aligns with recent work on lottery-ticket and pruning theory, hinting at a unified capacity-sparsity framework.
• Dynamic-programming solvers for architecture echo the rise of differentiable architecture search, but with zero gradient cost.

Bottom line

Compute NSC for your next model spec and let the DP solver hand you the best architecture before you write a single line of training code.

Takeaways

• NSC is a training-free, architecture-only capacity metric derived from random-init weight spectra.
• NSC-DP finds the globally optimal depth-width-head-FFN allocation under arbitrary constraints in seconds on a CPU.
• Across seven model families, NSC achieves τ = 0.505 on FlexiBERT pairs that differ by <10% in Params, where Params alone collapses to 0.082.
• NSC-DP discovers a Transformer-XL on WikiText-103 that beats the human-designed baseline in just 2 seconds.
• Pruning LLaMA-7B to a 5.7 B model across eight commonsense tasks is ~5,900× faster than the strongest training-free proxy.

Topics: model architecture, training-free proxy, capacity estimation, transformers