← All projects

QRS research · Attention theory · Representation geometry

SoftRank

The approximation rank of softmax attention

How much rank does an attention head need? Measure the operator, not one choice of query and key coordinates.

A natural way to judge how complex a head is, is to look at its queries and keys: their radii and their spectra. But one attention matrix has many query–key factorizations, and those numbers change from one to the next. SoftRank asks how much rank is needed to reproduce the head’s output on every bounded set of values, and bounds it with the geometry that softmax itself can see, which is the same in every coordinate frame.

Details

Active

Quantitative Research Society, Singapore · Nanyang Technological University

NeurReps 2026: extended abstract accepted · arXiv 2608.28150 (v2)

One attention operator in two coordinate framesFour queries and four keys at plus and minus e1 and e2, sheared by S_t with t = 5. The attention matrix is the same for every t. The native query-key radius product is 1 + t squared = 26; the intrinsic size kappa(C) stays 1.Queries and keysquerykeyt = 5softmaxAttention A0.530.070.200.200.070.530.200.200.200.200.530.070.200.200.070.53unchanged in every frameSize of the interaction (log scale)native radius product ΓΓ = 26intrinsic size κ(C)κ(C) = 11101001 Factor2 Normalise3 Change frame4 Intrinsic size
One attention operator in two coordinate framesFour queries and four keys at plus and minus e1 and e2, sheared by S_t with t = 5. The attention matrix is the same for every t. The native query-key radius product is 1 + t squared = 26; the intrinsic size kappa(C) stays 1.Queries and keysquerykeyt = 5softmaxAttention A0.530.070.200.200.070.530.200.200.200.200.530.070.200.200.070.53unchanged in every frameSize of the interaction (log scale)native radius product ΓΓ = 26intrinsic size κ(C)κ(C) = 11101001 Factor2 Normalise3 Change frame4 Intrinsic size
Four queries and four keys, then the same vectors after a change of coordinates Q → QS, K → KS⁻ᵀ. The attention matrix does not change. The radius product of the two clouds grows from 1 to 26; the intrinsic size κ(C) stays at 1. Four queries and four keys after a change of coordinates Q → QS, K → KS⁻ᵀ. The attention matrix is the same as before it, while the radius product of the two clouds is 26 instead of 1. The intrinsic size κ(C) is 1 in every frame. The rank-two example from the current text of the paper (it is not in arXiv v2), computed exactly.

SoftRank at a glance

The idea

Softmax cannot see anything that adds the same number to a whole row of logits, and the logits cannot see how they are split between queries and keys. What is left is the doubly centred log-attention C=P(logA)P. If a rank-r part T of C leaves a residual τ(T)<ε, then

rε(A)≤Cr(1+κ(T)(ε−τ(T))2)r/2,

where κ(T) is the smallest radius product over all factorizations of T, and no smaller exponent than r/2 works in the worst case.

The smallest frame: the radius product of the queries and keys over changes of coordinatesFour queries and four keys from the rank-two example at logit scale beta = 1, each cloud inside its smallest enclosing circle in the current frame. A marker on a log scale shows the product of the two radii. The dashed circles are the original frame, whose product is 1: the intrinsic size kappa(C), the smallest over all factorizations, which no change of coordinates goes below. The attention matrix, recomputed from the moved vectors, is the same in every frame.Queries · radius ρQKeys · radius ρKRadius product ρQ · ρK in this frameκ(C) = 1: no frame goes lower110Γ = 26Attention A, recomputedfrom the moved vectors:the same in every frame
The smallest frame: the radius product of the queries and keys over changes of coordinatesFour queries and four keys from the rank-two example at logit scale beta = 1, each cloud inside its smallest enclosing circle in the current frame. A marker on a log scale shows the product of the two radii. The dashed circles are the original frame, whose product is 1: the intrinsic size kappa(C), the smallest over all factorizations, which no change of coordinates goes below. The attention matrix, recomputed from the moved vectors, is the same in every frame.Queries · radius ρQKeys · radius ρKRadius product ρQ · ρKin this frameκ(C) = 1: the floor110Γ = 26Attention A, recomputedfrom the moved vectors:the same in every frame
The smallest frame. The four queries and keys of the figure above, at logit scale β = 1, each cloud inside its smallest enclosing circle. Stretch or shear the coordinates over a range of frames: the attention matrix stays the same, while the product of the two radii moves. Because κ(C) is the smallest product over all factorizations, no frame goes below κ(C) = 1, which the original frame (dashed) reaches.After the shear t = 5 the product of the two radii is 26; the original frame (dashed) reaches κ(C) = 1, and no change of coordinates goes below it. Illustration: the rank-two example from the current text of the paper (it is not in arXiv v2), computed exactly, not a measurement.

Experiments

Learned attention heads tested

Certified rank brackets9,978
Median of upper ÷ lower, five model arms2.6–3.8
Best construction ÷ lower certificate2.6–3.7×

BERT, GPT-2, Qwen2.5-1.5B and two ViT-Base checkpoints at ε = 0.25; the construction gap is on a separate held-out panel of 1,152 operators. On Qwen2.5-1.5B and, in a blind run on fresh documents, Llama-3.2-1B, fixed rank budgets of 8 and 16 are shown feasible less often as the block widens.

Validated numerics In progress

Interval-checked versions of the certificates, and brackets for κ(T) itself on learned heads, are not finished.

Brackets on 2,302 learned attention operators at ε = 0.25Five panels, one per model family. Each operator is a vertical segment from its certified lower rank to its checked upper rank on a log scale; resolved operators are teal, unresolved grey.141664256BERT-baseresolved 42.8% · U/L 3.82GPT-2resolved 99.7% · U/L 3.00Qwen2.5-1.5Bresolved 77.1% · U/L 2.63ViT-Base (ImageNet-1k)resolved 59.0% · U/L 3.41ViT-Base (ImageNet-21k)resolved 61.9% · U/L 3.08rank (log scale)Each column is one operator, sorted by its upper end; the segment runs from the certified lower rank to the checked upper rank. Dashed: full matrix rank.
Brackets on 2,302 learned attention operators at ε = 0.25One panel per model family; each operator is a segment from its certified lower rank to its checked upper rank (log scale); resolved teal, unresolved grey.141664256BERT-baseresolved 42.8% · U/L 3.82141664256GPT-2resolved 99.7% · U/L 3.00141664256Qwen2.5-1.5Bresolved 77.1% · U/L 2.63141664256ViT-Base (ImageNet-1k)resolved 59.0% · U/L 3.41141664256ViT-Base (ImageNet-21k)resolved 61.9% · U/L 3.08
2,302 learned attention operators, each with a certified lower rank and a checked upper rank (ε = 0.25). Teal operators are resolved: the upper end is at most twice the lower end, or within two ranks or a tenth of full rank of it. More results →

Limitations and what didn’t work →

Papers

  • Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention

    Yuhe Sui, Jianing Zhang, Yingzhi Tang

    arXiv 2608.28150 · v1 28 August 2026, v2 22 September 2026 [arXiv]

Accepted Research

  • The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension

    Yuhe Sui, Jianing Zhang

    NeurReps Workshop, NeurIPS 2026 (extended abstract) · 2026 · Accepted [OpenReview]

QR code for qrsntu.org/projects/softrank/
Scan to open this page