← PolicyAttention

PolicyAttention · ICAIF 2026 workshop paper · Market making

Where does learned in-context PMD pay off in market making?

A component diagnosis of the actor and the critic under fee, volatility, liquidity and toxicity stress

When an in-context market maker does badly after a regime shift, its returns cannot tell you which learned part failed. In a market we can solve exactly, we swap the learned actor and the learned critic, one at a time, for their exact counterparts. After 20 rounds the learned actor adds little; the critic stage carries the excess normalized loss, and in a follow-up an untrained average of the critic's own six samples loses about as much.

How it works → See the evidence

Authors: Xinyue Wu, Yuhe Sui, Shufang Chen, Yingzhi Tang

Poster 03 · ICAIF 2026 Workshop on AI-Driven Market Microstructure · Milan, 14 November 2026

exact criticlearned criticexact actorExactexact TD-PMDreferencemedian ℓ = 0.133 on G0CriticOnlylearned critic+0.0219above 0 on all 9 familieslearned actorActorOnlylearned actor+0.0020interval includes 0FullLoopboth learned+0.0235the critic carries almost all
exact criticlearned criticexact actorExactexact TD-PMDreferenceℓ = 0.133 on G0CriticOnlylearned critic+0.0219> 0 on all 9 familieslearned actorActorOnlylearned actor+0.0020interval includes 0FullLoopboth learned+0.0235mostly the critic
Paired excess normalized loss over exact TD-PMD after 20 rounds on nominal markets (lower is better): the median of 320 paired values, 5 runs × 64 markets. The learned actor's 95% interval is [−0.0002, +0.0043].

The paper at a glance

The idea

Policy mirror descent (PMD) improves a policy π against action values Q with a step η:

πk+1=softmax(logπk+ηQk)

One frozen Transformer runs both halves in context: an actor that takes this step and a critic that estimates the next Q from 6 sampled transitions per state–action pair. Because our dealer market is exactly solvable, each half has an exact counterpart, and a failure becomes a set of measured component costs.

Experiments

Synthetic market tested

Learned actor+0.0020
Learned critic+0.0219
Both learned+0.0235

Excess normalized loss over exact TD-PMD on nominal markets. In a follow-up, an untrained average of the critic's six samples loses 0.70–1.10 times as much as the learned critic, across all 9 families.

Against desk rules

The loop beats HBEST, the best nominal rule, under severe fee (+0.1661 in value capture) and toxicity (+0.1247) stress, but loses to it on nominal, volatility and illiquidity markets, and to the order-flow rule HIMB under toxicity.

Real markets Not tested

Value capture on a 15-state synthetic market, not trading P&L.

Paper

Paper and poster PDFs: not yet posted. This work is part of PolicyAttention.