QRS research · Reinforcement learning · Transformers
PolicyAttention
Softmax attention as the policy-improvement step of an in-context controller
Softmax attention already has the form of a policy-improvement update. Build the controller around it, then find where it breaks.
Earlier Transformer constructions for in-context control improve a policy with linear attention, while constructions with standard softmax attention stop at evaluating one. PolicyAttention starts from a classical identity: the policy mirror descent update is a softmax, so one attention row can be the actor. We build a causal actor–environment–critic loop around it, bound the loss of the policy it returns, and map where finite attention makes a stronger update backfire.
Details
Please help me understand PolicyAttention, a research project on softmax attention as a policy-improvement step for in-context control.
Project page: https://qrsntu.org/projects/policyattention/
How it works: https://qrsntu.org/projects/policyattention/details/
Plain-text summary: https://qrsntu.org/projects/policyattention/summary.md
arXiv: https://arxiv.org/abs/2609.30500
PTA workshop paper (NeurIPS 2026): https://openreview.net/forum?id=zbGWnk0kY3
EvoRobust workshop paper (NeurIPS 2026): https://openreview.net/forum?id=VgswuGrqp6
Design
Negative-entropy policy mirror descent (PMD) updates a policy by pi_new(a|s) proportional to pi(a|s) exp(eta Q(s,a)), which is exactly softmax(log pi + eta Q). PolicyAttention uses one softmax attention row, with scores log pi + eta Q, as the actor of an in-context controller. A causal loop then runs actor, environment (fresh samples under the new policy) and a softmax critic (TD evaluation). A fixed 2-block, 2-head normalization-free decoder runs both calls, and its routing, critic and sampling errors propagate to a bound on the loss of the policy the loop returns, assuming a fresh per-state-action generative oracle. The robustness paper shows where finite attention breaks this: other states' memories keep a little mass, and a larger step can reverse improvement.
Key results
- Exact two-state family (margin kappa = 8, gamma = 0.8): raising eta from 0.8 to 2.4 raises ideal PMD gain from 0.950 to 2.084 but turns the implemented gain from +0.916 to −2.009; the sign flips at eta*c = (1-gamma)*kappa. A state-local mask makes the decoder equal PMD in all 1,803 tested cells (840 negative without it, 0 with it).
- Coverage relaxed, same decoder, 64 small MDPs: a single on-policy stream of the oracle's size leaves 3.86 of 12 pairs unvisited per round and raises median normalized returned-policy loss from 0.009 to 0.029.
- Learned pre-LN Transformer trained without direct PMD labels: held-out row-L1 error to PMD 0.0357 (closest other fixed rule 0.112), TD MAE 0.121.
- Repeated control, 5 separately trained actors with an exact critic, 20 rounds: loss ratio to an exact PMD reference 0.918 to 1.396, median 1.052, all within the registered 1.5x band.
- Learned stress screen: on low-entropy inputs mean row error 0.059 but sample maximum 1.977; a matched audit shows large row error need not mean value loss.
One round, step by step: a softmax row turns the memories of state s1 into its new policy, the environment samples under that policy, the critic returns the next Q, and with a finite margin the memories of another state leak into the row.A softmax row turns the memories of state s1 into its new policy; the environment samples under it and the critic returns the next Q. Dashed ochre links: another state’s memories, which keep some attention when the margin is finite.Illustrative numbers.
PolicyAttention at a glance
The idea
Negative-entropy policy mirror descent (PMD) moves a policy towards actions with high value , with step size , state by state:
What one step does. Each point is a mix of three actions, here three exposures. Grey arrows: one step of the rule from many starting mixes, towards the highest-value action but staying near where it started. Dashed: repeated steps. Within one state, adding the same amount to every value moves nothing; across states, with finite attention, it can. Drag today’s mix, change the step size, or raise every value.Illustrative values: what the rule predicts, not a measurement.
An attention row whose scores are computes this exactly. The environment then supplies fresh samples under the new policy, a softmax critic evaluates it, and the errors of both add up to a bound on the loss of the policy the loop returns.
Experiments
Exact decoder tested · EvoRobust, PTA
Ideal / implemented gain, η = 0.80.950 / +0.916Ideal / implemented gain, η = 2.42.084 / −2.009
EvoRobust paper: a larger step makes ideal PMD better but the implemented update worse once ; a state-local mask removes this. PTA paper: with coverage relaxed, one on-policy stream raises the returned-policy loss from 0.009 to 0.029.
Learned Transformer tested · PTA
Row error to PMD, held out0.0357Repeated control, loss ratio to exact PMD1.052
The next-closest fixed rule is at 0.112. Ratio: median of 5 trained actors after 20 rounds on held-out random MDPs, all within a 1.5× band.
Without the oracle Not yet tested
The guarantee assumes fresh samples for every state–action pair each round. A learned critic on ordinary trajectory data is not yet tested. A learned closed loop with actor and critic errors checked every round has been run only in the IAFM paper, in one small synthetic allocation problem and still with fresh samples for every pair.
Finance ICAIF 2026
Two ICAIF 2026 workshop papers study the same update rule in synthetic allocation and market-making settings: market-making pages.
A stronger step can reverse improvement. Exact value gain in a two-state family through the fully specified decoder (margin κ = 8, γ = 0.8). Ideal PMD keeps improving as the step grows; without a mask the implemented gain crosses zero at η = 1.6, as predicted; with a state-local mask it equals PMD. More results →
A learned critic run on ordinary trajectory data, where some pairs go unvisited and samples are dependent.
Closed-loop audits of learned models
Learned rollouts whose actor and critic errors are measured against the policy they actually execute, beyond the one small allocation problem of the IAFM paper.
Routing failures in general models
Whether the reversal seen in the exact decoder occurs in learned Transformers that see other states’ memories.