QRS research · Attention theory · Representation geometry
SoftRank
The approximation rank of softmax attention
How much rank does an attention head need? Measure the operator, not one choice of query and key coordinates.
A natural way to judge how complex a head is, is to look at its queries and keys: their radii and their spectra. But one attention matrix has many query–key factorizations, and those numbers change from one to the next. SoftRank asks how much rank is needed to reproduce the head’s output on every bounded set of values, and bounds it with the geometry that softmax itself can see, which is the same in every coordinate frame.
Details
Please help me understand SoftRank, a research project on the approximation rank of softmax attention.
Project page: https://qrsntu.org/projects/softrank/
How it works: https://qrsntu.org/projects/softrank/details/
Plain-text summary: https://qrsntu.org/projects/softrank/summary.md
Paper (arXiv): https://arxiv.org/abs/2608.28150
Extended abstract (OpenReview): https://openreview.net/forum?id=0zgsZ8FjVz
Design
SoftRank studies the approximation rank r_eps(A) of a softmax attention matrix A: the least rank of any real matrix within maximum-row-l1 error eps, which is exactly the rank needed to preserve every bounded attention output. Instead of measuring one query-key factorization (radii, spectra, the native product Gamma = beta rho_Q rho_K, which changes under Q -> QS, K -> KS^{-T} while A stays fixed), it reads geometry off the softmax-visible interaction C = P_m (log A) P_N. For a retained rank-r part T with projective residual tau(T) < eps and size kappa(T), the smallest radius product over all factorizations of T, the theorem gives r_eps(A) <= min{p, C_r (1 + kappa(T)/(eps - tau(T))^2)^(r/2)}, invariant to refactorization. The r/2 exponent is attained on a finite grid, and on sphere and ball supports the exponents are (d-1)/2 and d/2. On trained heads the rank is bracketed directly with certified lower and checked upper witnesses.
Key results
- 9,978 brackets (3,326 operators at three tolerances) on BERT, GPT-2, Qwen2.5-1.5B and two ViT-Base checkpoints; on the 2,302-operator primary panel at eps 0.25 the median upper/lower ratio is 2.6–3.8 by family.
- Held-out panel of 1,152 operators: the best of six constructions needs 2.6–3.7 times the lower certificate (family medians); truncated SVD needs 1.62 times the rank of alternating l1.
- Qwen2.5-1.5B common-prefix blocks of width 64 to 512: fixed budgets of rank 8, 16 and 32 are shown feasible less often as the block widens. A blind run on Llama-3.2-1B with fresh documents shows the same decline at ranks 8 and 16; rank 32 is unresolved.
- The native-coordinate geometry adds no detectable held-out information beyond a calibrated baseline, and it moves under changes of coordinates that leave A fixed.
- Venues: NeurReps Workshop (Symmetry and Geometry in Neural Representations), NeurIPS 2026, extended abstract accepted; arXiv 2608.28150 (v2).
Four queries and four keys, then the same vectors after a change of coordinates Q → QS, K → KS⁻ᵀ. The attention matrix does not change. The radius product of the two clouds grows from 1 to 26; the intrinsic size κ(C) stays at 1.Four queries and four keys after a change of coordinates Q → QS, K → KS⁻ᵀ. The attention matrix is the same as before it, while the radius product of the two clouds is 26 instead of 1. The intrinsic size κ(C) is 1 in every frame.
The rank-two example from the current text of the paper (it is not in arXiv v2), computed exactly.
SoftRank at a glance
The idea
Softmax cannot see anything that adds the same number to a whole row of logits, and the logits cannot see how they are split between queries and keys. What is left is the doubly centred log-attention . If a rank- part of leaves a residual , then
where is the smallest radius product over all factorizations of , and no smaller exponent than works in the worst case.
The smallest frame. The four queries and keys of the figure above, at logit scale β = 1, each cloud inside its smallest enclosing circle. Stretch or shear the coordinates over a range of frames: the attention matrix stays the same, while the product of the two radii moves. Because κ(C) is the smallest product over all factorizations, no frame goes below κ(C) = 1, which the original frame (dashed) reaches.After the shear t = 5 the product of the two radii is 26; the original frame (dashed) reaches κ(C) = 1, and no change of coordinates goes below it.Illustration: the rank-two example from the current text of the paper (it is not in arXiv v2), computed exactly, not a measurement.
Experiments
Learned attention heads tested
Certified rank brackets
9,978
Median of upper ÷ lower, five model arms
2.6–3.8
Best construction ÷ lower certificate
2.6–3.7×
BERT, GPT-2, Qwen2.5-1.5B and two ViT-Base checkpoints at ε = 0.25; the construction gap is on a separate held-out panel of 1,152 operators. On Qwen2.5-1.5B and, in a blind run on fresh documents, Llama-3.2-1B, fixed rank budgets of 8 and 16 are shown feasible less often as the block widens.
Validated numerics In progress
Interval-checked versions of the certificates, and brackets for κ(T) itself on learned heads, are not finished.
2,302 learned attention operators, each with a certified lower rank and a checked upper rank (ε = 0.25). Teal operators are resolved: the upper end is at most twice the lower end, or within two ranks or a tenth of full rank of it. More results →