← All projects

QRS research · Reinforcement learning · Transformers

PolicyAttention

Softmax attention as the policy-improvement step of an in-context controller

Softmax attention already has the form of a policy-improvement update. Build the controller around it, then find where it breaks.

Earlier Transformer constructions for in-context control improve a policy with linear attention, while constructions with standard softmax attention stop at evaluating one. PolicyAttention starts from a classical identity: the policy mirror descent update π+∝πeηQ is a softmax, so one attention row can be the actor. We build a causal actor–environment–critic loop around it, bound the loss of the policy it returns, and map where finite attention makes a stronger update backfire.

Details
PolicyAttention: one round of the loopIllustrative. Memories for state s1 carry log pi plus eta Q; one softmax row turns them into the new policy at s1. The environment samples under the new policy and the critic computes the next Q by TD evaluation. With a finite margin, memories of another state keep some attention and shift the row towards that state's preferred action.Memoriesstate · action · log π + ηQs1 · a1s1 · a2s1 · a3s2 · a1other states2 · a2other states2 · a3other statesoftmaxone rowPolicy at s1before (outline), after (filled)a1a2a3with leak: a3 gainsEnvironmentfresh samples under the new policyCriticTD evaluation of the new policynext QActorEnvironmentCriticMargin κ
One round, step by step: a softmax row turns the memories of state s1 into its new policy, the environment samples under that policy, the critic returns the next Q, and with a finite margin the memories of another state leak into the row. A softmax row turns the memories of state s1 into its new policy; the environment samples under it and the critic returns the next Q. Dashed ochre links: another state’s memories, which keep some attention when the margin is finite. Illustrative numbers.

PolicyAttention at a glance

The idea

Negative-entropy policy mirror descent (PMD) moves a policy π towards actions with high value Q, with step size η, state by state:

πk+1=softmax(logπk+ηQk)
One step of the rule on the triangle of mixesEach point in the triangle is a mix of three actions: all defensive, all balanced, all aggressive. Grey arrows show one step of the rule from many starting mixes: each leans towards the aggressive corner, which has the highest value, but stays near where it started. A navy arrow shows one step from today’s mix, and a dashed line shows where repeated steps go: the highest-value corner.today’s mixone stepall defensiveall balancedall aggressivehighest value
What one step does. Each point is a mix of three actions, here three exposures. Grey arrows: one step of the rule from many starting mixes, towards the highest-value action but staying near where it started. Dashed: repeated steps. Within one state, adding the same amount to every value moves nothing; across states, with finite attention, it can. Drag today’s mix, change the step size, or raise every value. Illustrative values: what the rule predicts, not a measurement.

An attention row whose scores are logπ+ηQ computes this exactly. The environment then supplies fresh samples under the new policy, a softmax critic evaluates it, and the errors of both add up to a bound on the loss of the policy the loop returns.

Experiments

Exact decoder tested · EvoRobust, PTA

Ideal / implemented gain, η = 0.80.950 / +0.916Ideal / implemented gain, η = 2.42.084 / −2.009

EvoRobust paper: a larger step makes ideal PMD better but the implemented update worse once ηc>(1−γ)κ; a state-local mask removes this. PTA paper: with coverage relaxed, one on-policy stream raises the returned-policy loss from 0.009 to 0.029.

Learned Transformer tested · PTA

Row error to PMD, held out0.0357Repeated control, loss ratio to exact PMD1.052

The next-closest fixed rule is at 0.112. Ratio: median of 5 trained actors after 20 rounds on held-out random MDPs, all within a 1.5× band.

Without the oracle Not yet tested

The guarantee assumes fresh samples for every state–action pair each round. A learned critic on ordinary trajectory data is not yet tested. A learned closed loop with actor and critic errors checked every round has been run only in the IAFM paper, in one small synthetic allocation problem and still with fresh samples for every pair.

Finance ICAIF 2026

Two ICAIF 2026 workshop papers study the same update rule in synthetic allocation and market-making settings: market-making pages.

Value gain against step size in the matched two-state familyIdeal PMD gain rises with the step size. The implemented gain of the decoder without a mask falls below zero once eta passes (1 - gamma) kappa / c, marked by a dot. With a state-local mask the decoder equals ideal PMD at every step.−2−10120123η = 1.6step size ηvalue gain at s−ideal PMDdecoder, no maskdecoder, state-local mask
Value gain against step size in the matched two-state familyIdeal PMD gain rises with the step size. The implemented gain of the decoder without a mask falls below zero once eta passes (1 - gamma) kappa / c, marked by a dot. With a state-local mask the decoder equals ideal PMD at every step.−2−10120123η = 1.6step size ηvalue gain at s−ideal PMDdecoder, no maskdecoder, state-local mask
A stronger step can reverse improvement. Exact value gain in a two-state family through the fully specified decoder (margin κ = 8, γ = 0.8). Ideal PMD keeps improving as the step grows; without a mask the implemented gain crosses zero at η = 1.6, as predicted; with a state-local mask it equals PMD. More results →

Limitations and what didn’t work →

Accepted Research

  • PolicyAttention: Robustness and Failure Modes of Softmax Policy Improvement

    Yuhe Sui, Yingzhi Tang, Xinyue Wu

    EvoRobust Workshop, NeurIPS 2026 · 2026 · Accepted [OpenReview]

  • PolicyAttention: When Softmax Attention Becomes Policy Improvement

    Yuhe Sui

    PTA Workshop, NeurIPS 2026 · 2026 · Accepted [OpenReview]

Preprint and ICAIF 2026

  • PolicyAttention: Softmax Attention Implements Policy Mirror Descent for Closed-Loop Control

    Yuhe Sui, Yingzhi Tang, Shufang Chen

    arXiv 2609.30500 · 2026 [arXiv]

  • PolicyAttention: A Certifiable and Testable Update Rule for Attention-Based Allocation Controllers

    IAFM'26 workshop, ICAIF 2026 · 2026 · Accepted

  • Where Does Learned In-Context Policy Mirror Descent Pay Off in Market Making? A Component Diagnosis

    ICAIF 2026 Workshop on AI-Driven Market Microstructure · 2026 [Project pages]

Affiliations: Quantitative Research Society, Singapore · Nanyang Technological University (on the NeurIPS 2026 workshop papers).

QR code for qrsntu.org/projects/policyattention/
Scan to open this page