# GeoAFT

Geometric Attention Fine-Tuning: adapting how a pretrained model compares representations, not only the representations themselves. A research project of the Quantitative Research Society, Singapore and Nanyang Technological University; workshop paper accepted at the AutoMLR Workshop, NeurIPS 2026.

- Overview: https://qrsntu.org/projects/geoaft/
- How it works: https://qrsntu.org/projects/geoaft/details/
- Paper (OpenReview): https://openreview.net/forum?id=9PwlVQ3gyj

Plain-text summary for AI agents and readers, generated from the evidence file behind the pages; the pages are the full account.

## Design

GeoAFT (Geometric Attention Fine-Tuning) adapts the rule a pretrained model uses to compare a query with a key, instead of only the maps that produce queries and keys (as LoRA-style methods do). It adds a small, gated, identity-initialised residual g(P_q q, P_k k) to the native attention or class score, for example a projected L1 distance, a learned L_p distance, a Mahalanobis form or a small pair network; the gate starts at zero, so training begins from exactly the pretrained model.

## Key results

- Controlled synthetic tasks with a known target relation: oracle-gap recovery 99.983% (learned L_p target) and 99.991% (Mahalanobis target); on a nonlinear target 46 trainable parameters reached R = 0.9993429 against 0.9953917 for an 8,705-parameter pair MLP. The pre-registered superiority gates were not met.
- Remote-sensing few-shot (RSFS-8, score-level overlay on the ProKeR host, GeoRSCLIP ViT-B/32, 8 of 10 datasets, re-drawn splits): 84.48 / 88.20 / 91.47 at 1/2/4 shots against the best published rows 82.33 / 86.58 / 89.53; a projected-dot control without the geometry also cleared them (84.17 / 87.99 / 90.81). A scoped result, not a full RSFS result: our zero-shot sat +2.93 points above the published zero-shot, inside the +-3.0 sanity gate by 0.07.
- CLIP few-shot on the 11 CoOp datasets: the finalist stayed 0.33 to 0.44 points below the published cached-feature bars (per shot 1/2/4/16: 0.82 / 0.32 / 0.06 / 0.36 below).
- Terminal nulls on GLUE MRPC, domain generalisation and biomedical few-shot; on GLUE RTE the comparator could not be reproduced, so no comparison was run.
- Mechanism study on development data (descriptive; no GeoAFT method scored on a test split): scoped development negative: no geometry-specific improvement was demonstrated in the tested development settings, and the mechanisms behind the remaining increments are not identified.

## Limitations

- The RSFS-8 comparison is point estimates against published rows on re-drawn splits (8 of 10 datasets, 1/2/4 shots, one backbone); our zero-shot was +2.93 points above the published zero-shot, inside the +-3.0 sanity gate by 0.07, and the overlay led the projected-dot control by only +0.27 points.
- On fixed class keys (in-sample CMPt host) a class-bias-only arm came within 0.08 and 0.04 pp of the finalist; the projected-dot increment over the bias lay within +-0.30 pp (90% bounds [-0.04, +0.20] and [-0.09, +0.17], against total gains of +0.54 and +0.36 pp).
- On the last attention layer of CLIP ViT-B/16, an L1 overlay trained alone gained +5.32 pp (DTD) and +30.31 pp (EuroSAT) over zero-shot, but the gain was coordinate-specific, not geometry-specific: random coordinates removed 91% of it, replacing L1 by the dot 15%.
- Post hoc, an equal-parameter Q/K-LoRA was ahead in point estimate (-3.72 [-10.03, +2.58] and -2.49 [-10.32, +5.33] pp); the overlay stopped at its step budget more often, so under-training is not excluded. No overlay gain on top of Q/K-LoRA was observed.
- A DINOv2 proxy of the LC-TIM+DINO product graph did not meet the product-compatibility criterion (product minus sum graph -0.05 pp [-0.10, +0.01]); the published DINOv3 method was not reproduced (gated weights).
- The synthetic tasks favour the matching geometry by construction, and flexible generic learners also approximated the targets well.

## Not claimed

- state of the art on any benchmark
- an improvement on a real task that needs the geometry itself
- parity with, or an advantage over, Q/K-LoRA on a real model
- universal superiority over LoRA
- an accuracy-compute trade-off
- new Mahalanobis expressivity beyond the equivalent score family
- an identified mechanism for the remaining increments
- any test-split result for a GeoAFT method from Phase 14

## Coming soon

- Open text keys: remote-sensing image-text retrieval (RSICD, RSITMD): recommended first study for the next phase, after a leakage audit and a per-document bias control
- LC-TIM+DINO with the published DINOv3 SAT-493M encoder: design frozen; runs once access to the gated weights is granted
- Host redundancy and optimisation: design frozen (Study M2) on development data

## A question to start with

Explain the idea to a newcomer, then say which experiment would most convincingly show a benefit that is specific to the geometry, and why the ones run so far did not.