The idea
LoRA-style methods change how queries and keys are produced. Geometric attention fine-tuning changes how they are compared:
is a small trainable geometry; the gate starts at zero (figure below).
QRS research programme · Fine-tuning · Attention
Geometric Attention Fine-Tuning
Adapting how representations relate to one another, not only the representations themselves.
A pretrained Transformer learns representations and a rule for comparing them. Fine-tuning usually changes the first. GeoAFT adapts the second. On controlled tasks it recovers a known comparison rule almost exactly. On real tasks its margins over controls without the geometry are small, and a mechanism study found no geometry-specific improvement in the settings it tested.
The idea
LoRA-style methods change how queries and keys are produced. Geometric attention fine-tuning changes how they are compared:
is a small trainable geometry; the gate starts at zero (figure below).
Experiments
Recovery of three known target relations, not classification accuracy.
Remote-sensing few-shot: a scoped result of 84.48 / 88.20 / 91.47 at 1/2/4 shots, averaged over 8 of the 10 RSFS datasets, against best published rows of 82.33 / 86.58 / 89.53; a projected-dot control also clears them.
Mechanism study: no geometry-specific improvement was demonstrated on development data. On fixed class keys, with the strongest fitted host, a class bias alone came within 0.08 pp of the finalist overlay.
Open text keys (retrieval) Coming soon
What these results do and do not show: Limitations and what didn’t work →