The idea, the experiments, and the maths

1Why adapt relations?

Our working hypothesis is that relations matter most when representations are constrained: frozen or compressed features, small adaptation budgets, or similarity-critical tasks. Where representations are flexible, ordinary fine-tuning may absorb the same change. Either outcome is informative.

2Same points, different relations

Changing the geometry changes what counts as similar. Hold one query and seven keys fixed. The dot product prefers the key furthest along the query’s direction. Euclidean distance prefers the nearest key. Taxicab (L1) distance prefers neighbours along the axes. An anisotropic Mahalanobis metric prefers neighbours along a direction it treats as cheap. The winner changes every time, and no point has moved.

3The method in one picture

GeoAFT keeps the pretrained score and adds a small, trainable geometry term on top of it:

• Starts from the pretrained model. Each gate ${\alpha }_h$ is initialized at zero, so training begins from exactly the pretrained attention.
• Small and structured. What is trained is the geometry itself, a metric, an exponent or a projection, not a generic learned scorer.
• Leaves representations alone. $q$ and $k$ are not modified, so GeoAFT can be compared directly with matched Q/K adaptation.

4Synthetic experiments controlled

We first test GeoAFT where the right answer is known. A synthetic teacher generates attention from a known compatibility function over fixed representations, and every method adapts the same frozen dot-product base toward it. The score is oracle-gap recovery $R$: 0% means no improvement over the base, 100% means matching the oracle. It is not classification accuracy.

On the $L_p$ and Mahalanobis targets, Q/K-LoRA stays 39.9–41.7 points behind GeoAFT at the tested budgets, and the tested pair-MLP scorers 6.4–12.2 points; each range spans both targets.

5What the results mean

A geometry update can look different from a weight update and still compute the same function. So before asking whether geometry adaptation helps, we ask which kind of change it is.

1. Exact reparameterization

   ✓ Established analytically

   The geometry describes a function a conventional scorer already represents. Mahalanobis is the example.

2. Economical parameterization

   ◐ Supported in controlled experiments

   The same useful interaction, or a close approximation, with far fewer trained parameters.

3. Distinct adaptation capacity

   ○ Under investigation

   Changes a restricted conventional family cannot reach. Whether this survives downstream is the open question.

6In progress and future directions

Real benchmarks are in progress. Evaluation of the current method on natural tasks is ongoing, and no results from it are reported here. The programme ahead is organised around three research questions.

1. When is geometry adaptation actually different?

   The aim is not a larger metric menu. Each geometry family will be paired with a containment control that asks whether it can be compiled into an ordinary representation or score update. Theory will separate exact reparameterization, lower-complexity parameterization of an equivalent class, and genuinely distinct capacity.

   Now dotcosineL1 / L2learned $L_p$Mahalanobisrational-quadratic

   Next sphericalhyperbolicmixed curvaturelocal metriclearned kerneldynamic geometry

2. Can geometry become conditional?

   A single global metric will be generalized so that the geometry can depend on the input, layer, head or local statistics, through low-rank coordinate maps, kernel mixtures, local metrics, learned curvature and small selectors. Projected-dot and generic learned-score controls will isolate geometry from added projection capacity or search breadth.

3. Does it survive beyond attention and under compression?

   In JEPA-style latent prediction, a fixed distance between predicted and target latents will be replaced by a structured, trainable one. The same framework will be studied on encoder, vision and decoder-only models, with model size, token budget, latent dimension, adaptation budget and BF16 → INT8 → INT4 precision as probes. Stronger representations may give geometry more to exploit, or make it redundant because conventional PEFT absorbs the mismatch.

How it will be judged

1. Tested, syntheticIdentifiability Can the geometry recover a known relation?
2. PlannedFunctional necessity Does changing the relation improve prediction when features are frozen or constrained?
3. PlannedComplementarity Do gains survive equal-budget conventional PEFT, projected-dot, fixed-geometry, exact-containment and generic-score controls?
4. PlannedRegime dependence How do the answers change with architecture, scale and compression?

Experiments will move from controlled targets to geometry-critical frozen-feature, prototype and retrieval tasks, and then to pretrained Transformer, vision and JEPA systems. Model, split, comparator family, search budget, seeds and success margins will be frozen before final testing, and task quality will be reported with trainable and stored state, wall time, throughput and peak memory.

Geometry is unlikely to help uniformly. Similarity-critical, frozen or compressed regimes may benefit, while flexible networks may absorb the same change through ordinary PEFT. Both outcomes are useful: the intended result is a regime map of where adaptation capacity should be allocated.

7Notation and definition

For token states $x_i$, one attention head forms queries $q_i=W_Q{x}_i$ and keys $k_j=W_K{x}_j$, and scores each pair with the scaled dot product

where $𝒱(i)$ is the set of keys that query $i$ may attend to (its causal or padding mask). Most PEFT changes $(W_Q,W_K)\to (W_Q+\Delta W_Q,W_K+\Delta W_K)$ and leaves $s$ unchanged. GeoAFT adds a structured, trainable residual to the score:

with $S_{ij}=-\infty$ for $j\notin 𝒱(i)$ and $A_{ij}={softmax}_j{S}_{ij}$. Here:

• $P_q,P_k\in {ℝ}^{m\times d}$ are low-dimensional projections into the space where the geometry acts;
• $g_{\varphi }$ is a structured geometry family with parameters $\varphi$, such as a metric or an exponent;
• $c>0$ is a fixed calibration scale, estimated once without labels in the implementation, so the branch starts on a logit scale comparable to the base score;
• ${\alpha }_h$ is a per-head gate, initialized at $0$.

GeoAFT can be used on its own, with $W_Q$ and $W_K$ frozen, or alongside a conventional Q/K adapter. Comparing those configurations under matched budgets is the point of the experiments.

Development note. The earliest prototype mixed a bank of fixed operators per head through a router initialized at the dot product, then compiled the selection into a static model. Later versions use the gated residual above, which adds a trainable geometry instead of selecting among fixed ones.

8Families, centring and masks

With $u=P_{q}q$ and $v=P_{k}k$, the families used in the controlled experiments are:

The implementation also includes dot, cosine, L1, squared-L2 and rational-quadratic scores, several of them as reducibility controls rather than candidate contributions. The size of the menu is not the contribution.

Identity at initialization. With ${\alpha }_h=0$, $S_{ij}=s_{\mathrm{b}\mathrm{a}\mathrm{s}\mathrm{e}}(q_i,k_j)$ for every pair, so the adapted model computes the pretrained attention exactly, up to floating-point round-off. The research implementation verifies this with tests. This is a property of the implementation, not evidence that geometry adaptation improves prediction.

Support centring. Row-wise softmax ignores a constant added to all valid logits in a row, ${softmax}_{j\in 𝒱(i)}(S_{ij}+b_i)={softmax}_{j\in 𝒱(i)}(S_{ij})$. Centring the geometry branch over $𝒱(i)$ removes exactly this unidentifiable component. Masked positions are excluded from the mean, so centring never mixes valid and invalid keys.

9Q/K versus score adaptation

Q/K adaptation does change attention scores, through the bilinear form $W_Q^{\top }{W}_K$. With low-rank updates of rank $r_Q$ and $r_K$,

which has rank at most $r_Q+r_K$. A structured geometry is not restricted in this way: for $N$ distinct points on a line, the row-centred L1 score matrix $[-|x_i-x_j|]$ has rank $N-1$, the maximum possible after centring.

An exact equivalence. Not every geometry is new capacity. Squared Mahalanobis expands as

and the last term is constant within a softmax row, so it cancels. The overlay is softmax-equivalent to a bilinear score plus a key-quadratic bias, a known score family. Squared L2 is the case $M=I$, and cosine is a dot product of normalized vectors.

These are local statements about one head’s score matrix on fixed representations, not whole-Transformer impossibility results. A score that Q/K cannot reach is not necessarily useful for the task.

10Protocol, controls and full results

The controlled experiments test identifiability: can a method recover a known relation? Each fixes a set of representations and a target compatibility function. Methods are compared in paired cells on held-out synthetic queries and keys, and trainable parameter counts are recorded for every method.

$L_{\text{base}}$ is the loss of the unadapted model and $L_{\text{oracle}}$ is the oracle reference loss. Normalizing by the gap makes results comparable across targets whose raw KL scales differ.

Comparison rules and success margins were fixed before final testing. Not all registered superiority gates passed: the three caveats in §4 are the gates that did not.

11Limitations

• Synthetic scope. The results recover known relations on fixed representations. They are not natural-task results, and they say nothing about state-of-the-art performance.
• Natural tasks. Earlier natural-task experiments with previous versions of the method were negative. On hierarchical text classification and a GLUE MRPC replication, tuned Q/K-LoRA and full Q/K adaptation outperformed GeoAFT. Later coordinate-based variants did not exceed matched continued-training controls. This is why the project now asks when geometry adaptation helps rather than whether it helps in general.
• From score to task. A score change that Q/K cannot represent can still be erased by the value and output path, so non-absorbability in the score matrix does not imply task usefulness.
• Efficiency. Trainable-parameter counts are not inference cost. No accuracy–compute frontier is claimed until compatible measurements of time, throughput and memory are complete.

12Sources

The numbers on this page and in the figures come from the project’s current evidence summary. Each quantity is stored once in the figure-generation data, so the page and the figures cannot drift apart.

Back to top ↑