← All projects

QRS research programme · Fine-tuning · Attention

GeoAFT

Geometric Attention Fine-Tuning

Adapting how representations relate to one another, not only the representations themselves.

A pretrained Transformer learns representations and a rule for comparing them. Fine-tuning usually changes the first. GeoAFT adapts the second. On controlled tasks it recovers a known comparison rule almost exactly. On real tasks its margins over controls without the geometry are small, and a mechanism study found no geometry-specific improvement in the settings it tested.

Details

Active

Lead: Yuhe Sui

Workshop paper: AutoMLR Workshop, NeurIPS 2026

Illustration: the same query and keys under two compatibility rules. Dashed gray lines show pretrained dot-product attention, solid teal lines show attention after adapting the geometry; the points never move. A 36% B C D E F 49% G q
Rings and grid are the ruler; colour is compatibility with q. The geometry changes, attention re-ranks, the points stay put. Pretrained dot-product attention concentrates on key B. After the geometry is adapted, attention moves to F and A. The points do not move. Illustrative 2-D toy.

GeoAFT at a glance

The idea

LoRA-style methods change how queries and keys are produced. Geometric attention fine-tuning changes how they are compared:

Sij=sbase(qi,kj)+αhg¯ϕ(Pqqi,Pkkj)

gϕ is a small trainable geometry; the gate αh starts at zero (figure below).

Experiments

Synthetic tested

Learned Lp99.983%
Mahalanobis99.991%
Nonlinear99.934%

Recovery of three known target relations, not classification accuracy.

Real tasks tested

Remote-sensing few-shot: a scoped result of 84.48 / 88.20 / 91.47 at 1/2/4 shots, averaged over 8 of the 10 RSFS datasets, against best published rows of 82.33 / 86.58 / 89.53; a projected-dot control also clears them.

Mechanism study: no geometry-specific improvement was demonstrated on development data. On fixed class keys, with the strongest fitted host, a class bias alone came within 0.08 pp of the finalist overlay.

Open text keys (retrieval) Coming soon

Illustration of the gate: one query and seven fixed keys. Teal level lines show the combined score, teal lines and bars show attention from the query, and dark ticks mark pretrained dot-product attention, which the gate reproduces exactly at zero. Opening the gate moves attention from key B to keys F and A; the keys do not move. A B C D E F G q Attention from q A 34% B 0% C 19% D 9% E 0% F 37% G 0% pretrained (gate at 0) αh = 1.00
Illustration of the gate: one query and seven fixed keys. Teal level lines show the combined score, teal lines and bars show attention from the query, and dark ticks mark pretrained dot-product attention, which the gate reproduces exactly at zero. Opening the gate moves attention from key B to keys F and A; the keys do not move. A B C D E F G q Attention from q A 34% B 0% C 19% D 9% E 0% F 37% G 0% pretrained (gate at 0) αh = 1.00
What the gate αh changes. One query q and seven keys that never move. Teal lines are the ruler: keys on one line get equal attention, and from one line to the next attention falls by a factor of e, about 2.7. With the gate at zero the score is the pretrained scaled dot product (dashed, and the dark ticks). Opening it adds a small trainable geometry gϕ, here minus a squared stretched distance, and attention moves from B to F and A. The toy’s two scales are chosen for legibility, not taken from a model. Move the gate. Illustrative 2-D toy computed from the equation in The idea, not measured output.

Accepted Research

Related projects

AI Native Workflow, GeoAttention