Measure what softmax can see, in no particular coordinates

1Why not measure the queries and keys?

An attention head computes logits $H=\beta Q{K}^{\top }$ and normalises each row with softmax. The obvious way to say how complex the head is, is to measure the two point clouds that produce the logits: how far the queries and keys spread (their radii, combined into $\Gamma =\beta {\rho }_Q{\rho }_K$), or the spectrum of $W_q^{\top }{W}_k$. A bound written in those terms is correct for the factorization it is given.

But the factorization is not unique. For any invertible matrix $S$, replacing $Q$ by $QS$ and $K$ by $K{S}^{-\top }$ leaves every logit, and so the whole attention matrix, unchanged. The point clouds do change. In the paper’s rank-two example, four queries and keys at $\pm e_1,\pm e_2$ give $\Gamma =1$; after the shear $S_t$ with $t=2,5,10$ the same attention matrix has $\Gamma =5,26,101$. A measure that can be made as large as we like without touching the operator describes the coordinates, not the head.

SoftRank changes both ends. The target is the operator’s own approximation rank. The geometry is read off the part of the logits that softmax can see, and its size is minimised over every factorization at once, so that no frame is privileged.

2Three questions, three objects

Put together, they give the central theorem: whenever $\tau (T)<\epsilon$,

Here $p=min(m,N)$ is the trivial cap. The bound is never worse than the native bound with the same retained dimension, and it does not change under any invertible refactorization of the queries and keys.

3Coordinates against the operator

The two sliders separate what changes the coordinates from what changes the head. The shear $t$ moves the queries and keys; the matrix below is recomputed from the moved vectors each time and does not change. The logit scale $\beta$ changes the head itself: the matrix sharpens, and the intrinsic size grows with it.

Here $\kappa (C)=1$ exactly at $\beta =1$: the original factorization reaches a radius product of 1, and the rectangular contrast $C_{11}-C_{12}-C_{21}+C_{22}=4$ shows that no factorization can do better.

4Where the exponent comes from

5Measuring the rank directly

The constant $C_r$ in the theorem is not known numerically, so on trained heads the rank is measured rather than predicted. Any explicit matrix of rank $k$ that is checked to lie within $\epsilon$ proves $r_{\epsilon }(A)\le k$; the upper search uses reweighted SVD and alternating-${\ell }_1$ factorization. Lower certificates quantify over every real competitor: Frobenius tails, truncated pseudoinverses, residual-aware bounds and contractive decoders. A failed attempt to approximate is never used as a lower bound. Numerical attention matrices carry a slack $\chi$: $L_{\epsilon +\chi }(\hat{A})\le r_{\epsilon }(A)\le U_{\epsilon -\chi }(\hat{A})$.

Every result therefore says one of three things about a rank budget $k$: feasible (a checked construction of rank $k$ exists), impossible (a lower certificate exceeds $k$), or unresolved (neither). Certified here means a float64 computation without interval enclosure.

6Results: brackets on learned heads

The primary panel holds 2,302 operators: BERT-base at 256 × 256, the common-prefix blocks of GPT-2 and Qwen2.5-1.5B at 128 × 128, and two ViT-Base checkpoints at 197 × 197, from 64 fixed text sources and 64 ImageNet images. At three tolerances and with secondary decoder cells, the certificate pass produced 9,978 brackets. An independent recomputation found no failure, none of 458 targeted attacks on the lower ends succeeded, and 119 of 119 reconstructed upper witnesses passed the same check.

GPT-2, Qwen2.5 and both ViT arms meet the pre-specified identification criterion (resolved share, several disjoint bands, ordering coverage). BERT does not. At a common 128 × 128 shape the encoder and vision operators have larger checked upper budgets than the decoder blocks, and the two ViT checkpoints order difficulty almost identically while fine-tuning on ImageNet-1k raises the upper ends.

7Constructions against certificates

A held-out panel of 1,152 operators (32 text sources and 32 images) asks how close practical low-rank constructions come to the lower certificate. Six fixed implementations were run at ε = 0.25; alternating ${\ell }_1$ reached the lowest rank in every cell.

The gap is between a construction and a certificate; which end is loose is not identified. The bracket closes exactly in 0%, 17.7%, 19.3%, 0.4% of BERT, GPT-2, Qwen2.5 and ViT-1k operators.

8Rank budgets as the block widens

For Qwen2.5-1.5B the operator is the renormalised N × N common-prefix block of one 1,024-token pass, for 6 fixed heads and 32 arXiv documents, at widths N = 64 to 512. The question is a fixed budget: is rank 8, 16 or 32 enough at ε = 0.25?

On Qwen2.5 the pre-specified class is a decline at all three budgets: rank 16 drops from 98% to 47% of operators and rank 32 from 100% to 51%. The bracket widens with N, the lower ends nearly flat (median 2.5 → 3.5) and the first SVD-type upper ends growing (8.5 → 72), so the operators lose their feasible verdict to unresolved rather than to impossible.

On Llama-3.2-1B the class is a decline at budgets 8 and 16 (feasible share down by 19.9 points and 12.1) and unresolved at 32. Compared with Qwen2.5, budgets 8 and 16 replicate and 32 does not resolve. Two records of that comparison exist and are reported together in the limitations.

9Other attention regimes

The same certificates run unchanged on rectangular and encoder–decoder attention.

10Native coordinates as a predictor

The earlier, native-coordinate version of the geometry (native $\Gamma$, SVD-selected interaction dimensions and shape diagnostics) was tested as a predictor of the checked upper budget on held-out sources. Against a weak structural baseline it carries coarse information: held-out error falls by 0.0279 [0.0254, 0.0300] cap units. Against a calibrated baseline of logit, concentration and weight-space features, it adds nothing detectable: the aggregate gain in held-out error is −0.00005 cap units and the relative gain is −0.0035 [−0.0422, +0.0179].

The same native quantities move when the coordinates move. In a gauge audit of 144 operators, all 576 operator–gauge records pass the measurement’s own invariance check, while native $\Gamma$ changes by more than 10% in every operator under the diagonal and dense gauges, by up to 21×.

11Limitations and what didn’t work

Scope

• The panels are fixed checkpoints with fixed head rosters: one BERT encoder, GPT-2, Qwen2.5-1.5B, two ViT-Base checkpoints, FLAN-T5-base, Llama-3.2-1B and the BLIP-2 Q-Former. The GPT-2 and Qwen2.5 panels use six of eight pre-specified non-local heads. Nothing is claimed for other models, larger language models or long context.
• The theorem assumes one common positive support. Causal masks need more theory, so decoder results come from common-prefix blocks, and the native-causal brackets are measurements, not uses of the theorem.
• “Block width” is the width of a common-prefix block, not a context length. The query tokens change with N, so there is no causal length intervention, and each model was run on one dataset.
• The results concern representation rank. No runtime, memory, sparsity, accuracy or hardness conclusion follows.

What the evidence does not show

• Exact optimum ranks. Where the two ends differ the rank is an interval, and no midpoint is reported. The formal identification gate of the bracketing study was not met: BERT resolves 219 of 512 operators (42.8%), with layer 0 at 0/128.
• How much rank a trained head needs, read from its geometry. The computable intrinsic score agrees with certified rank order in BERT (0.996) and ViT-1k (0.995) but not in GPT-2 (0.159) or Qwen2.5 (0.391); FLAN-T5 and BLIP-2 align, though in FLAN-T5 the plain logit scale does better in the encoder (−0.068) and decoder (−0.076). Added to the calibrated baseline, the intrinsic block adds nothing (gain −0.000116 cap units) and makes leave-one-head-out prediction 47.2% worse. That test was retrospective: its targets and baseline were known before its protocol was frozen.
• A numerical value for the theorem. $C_r$ is not estimated; with the explicit constant the bound returns the trivial cap at the tested dimensions for $r\ge 2$. The experiments measure the rank instead, and their size proxy is the balanced-factor surrogate ${\kappa }_{\mathrm{b}\mathrm{a}\mathrm{l}}$, which can exceed $\kappa$ by an unbounded factor. The computable specialisations (balanced-factor size, canonical logit-SVD family) passed one internal proof check, the second with conditions on how they are computed; they do not solve the optimisation that defines $\kappa$.
• Impossibility at budget. Rank-$k$ impossibility certificates are rare: on Qwen2.5 they reach 6.9% of operators at rank 8 and N = 512 and never appear at 16 or 32, and Llama has none. At N = 512 63.3%, 53.4%, 49.3% of Qwen2.5 operators stay unresolved at ranks 8, 16 and 32. “Degrades” is a statement about these certificates and constructions, not about the true rank.
• The native null as an equivalence. The calibrated baseline’s error was already below the pre-set resolution in several families, so the test rules out a large relative gain, not every effect.
• Claims of the NeurReps abstract that the newer work dropped. The per-instance law there used the native $\Gamma =\beta {\rho }_Q{\rho }_K$, which the intrinsic $\kappa (T)$ replaces. Its 84-head BERT calibration (a smaller effective dimension in 41.9% of head–temperature cells; Spearman 0.574 and 0.606 with constructive upper certificates) is no longer offered as support for the geometry, and the reading that effective geometry tracks low-rank demand did not survive the held-out test above.

Implementation and reproducibility

• No validated numerics yet. Certificates are float64 without interval enclosure: computed singular values carry no accuracy bounds, the pseudoinverse witness’s residual is not charged, the common-prefix slack omits the rounding of renormalisation, and upper checks have no evaluation-error bounds. The validated re-check and the κ(T) brackets were started and are not finished; no result from them exists.
• Replication records. The blind Llama comparison has a pre-registered record in which every label is a defect, caused by a line-ending fault in the comparing code, and a corrected post-hoc reading (replicated at 8 and 16, unresolved at 32) that changes no statistic. Its experiment audit failed on closing-record wording while reproducing every statistic. An earlier Llama replication was not blind: its outcomes were seen before its design was pinned.
• Verification. Independent recomputations ran in isolated contexts from the same model vendor; there is no cross-vendor replication, and some verifiers covered a sample of operators rather than all of them. The intrinsic finite-grid sharpness result has one proof pass and one independent re-check; the native form of the $r/2$ sharpness was verified independently.
• Release. Code and per-operator records are to be released on publication and are not public yet.

12Where this is going proposed

1. Validated certificates

   A post-processing pass with rounding-error and residual bounds over the stored witnesses and constructions, so impossible, unresolved and feasible hold without the float64 qualifier.

2. κ(T) on learned heads

   Primal and dual brackets for the intrinsic size on a representative subset, to test the theorem’s own quantity rather than its balanced-factor surrogate.

3. Beyond one common support

   Theory for causal masks; and the open questions of the paper: equality of the intrinsic and native optima, joint sharpness in κ, residual and ε, and an efficient global algorithm for the profile.

13Definitions and the theorem

For an attention matrix $A\in {ℝ}^{m\times N}$ and $0<\epsilon <1$,

On one common positive support, with $P_m=I-\frac{1}{m}\mathbf{1}{\mathbf{1}}^{\top }$, the visible interaction and the projective residual of a centred $T$ are

The residual rests on the sharp inequality, with $\sigma$ the softmax map,

The size is a minimum over factorizations, attained in rank dimension:

The proof replaces $A$ by the softmax of $T$ plus a common column term, which costs at most $\tau (T)$ in row-${\ell }_1$; translating an optimal factorization to the centres of its enclosing balls turns that matrix into a rank-$r$ weighted Gibbs family of radius product $\kappa (T)$; the weighted Gibbs-row cover at budget $\epsilon -\tau (T)$ then gives the bound. For comparison, the native bound uses the radii of one factorization, $\Gamma =\beta {\rho }_Q{\rho }_K$, in the same place; at full interaction, minimising $\Gamma$ over invertible gauges recovers $\kappa (C)$.

14Support-geometry laws

For self-attention $A_{\beta }(X{)}_{ij}\propto e^{\beta ⟨x_i,x_j⟩}$ with $n$ tokens, write $\Re$ for the worst case of $r_{\epsilon }$ over configurations. For fixed $d\ge 2$ and $\epsilon$, on the sphere, and on the ball when $\beta \ge {\beta }_0(d,\epsilon )$ and $n\ge C_d{e}^{\beta /8}$:

The lower bounds come from decoding constructions (well-separated sphere centres; a discrete Gaussian on a lattice in the ball) whose attention is close to a block projection, so any good approximant must be invertible on the messages. The ball law needs exponentially many tokens, so it is a worst-case capacity statement, not a prediction for ordinary context lengths. The finite-grid family of section 4, $H_{ab}=2s{a}^{\top }b-s\Vert b{\Vert }^2$ on $\{0,\dots ,L-1{\}}^r$ with $s=\log \frac{8r}{1-\epsilon }$, has $rankC=r$, $\kappa (C)=sr(L-1{)}^2/2$ and $r_{\epsilon }(A)=L^r$.

15Full tables

16Reproducibility notes

Certificate pass

3,326 operators at tolerances 0.15, 0.25 and 0.35: 9,978 brackets, every upper end an explicit matrix checked in maximum-row-${\ell }_1$; independent recomputation with no failure; 0 of 458 targeted lower-end attacks succeeded; 119/119 reconstructed upper witnesses pass.

Numerics

Certificates and constructions in float64 without interval enclosure, with the slack $\chi$ of each numerical attention matrix charged to both ends.

Protocols

Each study fixed its panel, tolerances and decision rules in a frozen protocol before its run of record, and a separate verifier rechecked its numbers against the stored per-operator records.

Exact checks

The gauge example and the grid family are computed exactly from their definitions; the small exact oracle closes six rational cells, one with unrestricted rank one and row-stochastic rank two.

17In brief

18Sources

Paper: Intrinsic Interaction Geometry Controls the Low-Rank Complexity of Softmax Attention, Yuhe Sui, Jianing Zhang, Yingzhi Tang, arXiv 2608.28150 (v1 28 August 2026, v2 22 September 2026). [arXiv]

Extended abstract: The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension, Yuhe Sui, Jianing Zhang, NeurReps Workshop (Symmetry and Geometry in Neural Representations), NeurIPS 2026, accepted. [OpenReview]

Numbers on this page come from the project’s study records: the bracketing study, the held-out construction benchmark, the block-width studies on Qwen2.5 and Llama, the FLAN-T5 and BLIP-2 studies, and the native and intrinsic predictor tests. Where a newer study or the current text of the paper supersedes the extended abstract, this page follows the newer one.

Back to top ↑