# SoftRank

The approximation rank of softmax attention. A research project of the Quantitative Research Society, Singapore, and Nanyang Technological University; NeurReps 2026 extended abstract (accepted) and arXiv 2608.28150.

- Overview: https://qrsntu.org/projects/softrank/
- How it works: https://qrsntu.org/projects/softrank/details/
- Paper (arXiv): https://arxiv.org/abs/2608.28150
- Extended abstract (OpenReview): https://openreview.net/forum?id=0zgsZ8FjVz

Plain-text summary for AI agents and readers, generated from the evidence file behind the pages; the pages are the full account.

## Design

SoftRank studies the approximation rank r_eps(A) of a softmax attention matrix A: the least rank of any real matrix within maximum-row-l1 error eps, which is exactly the rank needed to preserve every bounded attention output. Instead of measuring one query-key factorization (radii, spectra, the native product Gamma = beta rho_Q rho_K, which changes under Q -> QS, K -> KS^{-T} while A stays fixed), it reads geometry off the softmax-visible interaction C = P_m (log A) P_N. For a retained rank-r part T with projective residual tau(T) < eps and size kappa(T), the smallest radius product over all factorizations of T, the theorem gives r_eps(A) <= min{p, C_r (1 + kappa(T)/(eps - tau(T))^2)^(r/2)}, invariant to refactorization. The r/2 exponent is attained on a finite grid, and on sphere and ball supports the exponents are (d-1)/2 and d/2. On trained heads the rank is bracketed directly with certified lower and checked upper witnesses.

## Key results

- 9,978 brackets (3,326 operators at three tolerances) on BERT, GPT-2, Qwen2.5-1.5B and two ViT-Base checkpoints; on the 2,302-operator primary panel at eps 0.25 the median upper/lower ratio is 2.6-3.8 by family.
- Held-out panel of 1,152 operators: the best of six constructions needs 2.6-3.7 times the lower certificate (family medians); truncated SVD needs 1.62 times the rank of alternating l1.
- Qwen2.5-1.5B common-prefix blocks of width 64 to 512: fixed budgets of rank 8, 16 and 32 are shown feasible less often as the block widens. A blind run on Llama-3.2-1B with fresh documents shows the same decline at ranks 8 and 16; rank 32 is unresolved.
- The native-coordinate geometry adds no detectable held-out information beyond a calibrated baseline, and it moves under changes of coordinates that leave A fixed.
- Venues: NeurReps Workshop (Symmetry and Geometry in Neural Representations), NeurIPS 2026, extended abstract accepted; arXiv 2608.28150 (v2).

## Limitations

- Certificates are float64 computations without interval enclosure; validated numerics are not finished.
- Optimum ranks are intervals where the ends differ; BERT resolves only 219 of 512 operators.
- The computable intrinsic score does not predict trained-head complexity: its agreement with certified order is heterogeneous across families, and it adds nothing beyond a calibrated baseline.
- The constant C_r is unknown, so the theorem is a structural bound, not a numerical certificate for learned heads.
- Impossibility certificates are rare in the block-width study and many large-width operators stay unresolved.
- The theorem assumes one common positive support; causal masks need more theory.

## Not claimed

- No runtime, memory, sparsity, accuracy or computational-hardness conclusion: the results concern representation rank only.
- No claim that the intrinsic geometry tells, ranks or accounts for how much rank a trained head needs.
- No exact optimum rank where the lower and upper certificates differ; no midpoint of a bracket is reported as a rank.
- No scaling law in block width, and nothing about attention heads in general beyond the fixed head rosters.
- No numerical value for the dimension constant C_r of the theorem.

## Papers

- Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention (Yuhe Sui, Jianing Zhang, Yingzhi Tang), arXiv 2608.28150, v1 28 August 2026, v2 22 September 2026.
- The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension (Yuhe Sui, Jianing Zhang), NeurReps Workshop (Symmetry and Geometry in Neural Representations), NeurIPS 2026, extended abstract, accepted.

## A question to start with

Explain the main theorem to a newcomer, then give the three strongest criticisms of the evidence and one experiment that would settle each.