The swap, the evidence, and the details

1Why take the controller apart?

A market maker keeps re-setting its quote skew as fees, volatility, liquidity and order-flow toxicity change. In-context reinforcement learning offers a controller that adapts without being re-engineered for each regime: a sequence model with frozen weights adapts at inference time by conditioning on recent experience. When such a controller does badly after a regime shift, the practitioner needs to know which learned component failed before deciding what to fix.

End-to-end scores mix three things: how well the policy-improvement step is learned, how well the value estimate it relies on is learned, and how hard the new regime is. A fixed rule also looks best in the regime it was designed for.

In an exactly solvable market, each learned component can be swapped for its exact counterpart on the same market and the same initial policy. A failure then becomes a set of measured component costs under each stress.

2A market we can solve exactly

The dealer holds inventory $q$ from −2 to 2 and observes order-flow imbalance $z\in \{-1,0,+1\}$, giving 15 states. Each step it sets a quote skew $u\in \{-1,0,+1\}$; at most one fill occurs. Buy pressure raises both the chance that the dealer's ask is hit and the chance that the price then rises: the adverse-selection channel, carried here by an observed state. The reward is spread income minus fees, plus mark-to-market, minus a mean–variance inventory penalty, with discount 0.8.

There are 9 families of 64 paired markets: nominal (G0) and fee, volatility, liquidity and toxicity stress (F, V, L, I), each at a moderate (1) and a severe (2) dose. Every nominal market has a twin in each stress family that re-draws one group of parameters and keeps the same random initial policy, so every comparison is paired market by market. At the severe dose, fees rise from [0, 0.2] to [0.6, 0.8], volatility from [0.8, 1.2] to [1.8, 2.2], and liquidity falls from [1, 2] to [0.3, 0.6].

The family is not trivial. On nominal markets (G0) the inventory rule HINV reaches a median value capture of 0.678 and exact PMD from the initial policy reaches 0.883 after 20 rounds, and severe stress changes at least one optimal action in 95–100% of markets.

3The loop and the swap in one picture

• Actor. Reads the policy ${\pi }_k$, the action values $Q_k$ and the step $\eta$, and returns ${\pi }_{k+1}$. It is trained toward the PMD step, $\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(\log {\pi }_k+\eta Q_k)$.
• Critic. Reads $Q_k$, the reward table and 6 sampled successors per state–action pair, and returns $Q_{k+1}$. It is trained toward the average backup of those six samples.
• The arms. Exact runs exact PMD with the exact backup (exact TD-PMD, the reference). ActorOnly learns only the actor, CriticOnly only the critic, FullLoop both. The follow-up arm SampleAvg replaces the critic network by the plain six-sample average it is trained to output, computed from the same samples.

The controller is PolicyAttention: one Transformer (4 layers, 4 heads, width 64) trained on nominal markets only, in 5 runs. Its weights stay frozen across rounds, so it adapts only through its context. Every arm starts from the same random policy and its exact action values, and runs 20 rounds.

4What each part costs after 20 rounds

Each cost is a paired difference in normalized loss against exact TD-PMD on the same market: the median of 320 paired values (5 runs × 64 markets) with a 95% hierarchical-bootstrap interval. Lower is better.

So a critic that fitted its six-sample target perfectly would keep at least 70% of the cost: improving the critic stage means improving what it is trained toward, not only how well it fits. On L1 the network adds +0.0025 [+0.0005, +0.0045] beyond the average, a label that becomes NOT_SEPARATED under a crossed bootstrap.

5By round and by family

Paired excess normalized loss over exact TD-PMD at round 20, median of 320 paired values with its 95% hierarchical-bootstrap interval (paper Table 1). SC1 and SC2 are follow-up comparisons. *: the interval excludes 0; †: NOT_SEPARATED under a crossed bootstrap; ×: the full loop fails the holds rule (see Protocol and controls). CriticOnly also fails it on V2 (ratio 1.514, 1 of 5 runs at or below 1.5).

6Where the loop pays off against desk rules

HBEST is the best of 45 linear skew rules on 256 separate nominal markets (a = 1, b = −1, c = −0.5; mean capture 0.872 there). It leans against inventory and order flow and is not re-tuned per regime. HIMB skews against order flow alone ($u=-z$).

• Fee and toxicity stress break HBEST. Its median capture falls from 0.931 nominally to 0.677 on F2 and 0.714 on I2, while the loop keeps 0.878 and 0.849. The paired median gain over HBEST is +0.1661 [+0.1363, +0.2159] on F2 and +0.1247 on I2.
• Volatility and illiquidity do not. There HBEST is close to optimal (0.958–0.989 on V2, L1 and L2) and the loop reaches 0.893, 0.778 and 0.740. Exact TD-PMD shows the same signs, so these losses are not the learned parts' alone: 20 rounds from a random start do not reach a rule that is already near-optimal for those markets.
• Under toxicity, order flow is enough. HIMB beats the loop (−0.0761 on I1, −0.0910 on I2), and exact TD-PMD too (−0.0530, −0.0547).

On the fee families the change reaches the loop through the exact reward table it is given, which the rules never see, so part of the fee gain may reflect that information advantage.

7Adapting after a shift

Is adapting in context better than keeping the old optimum? CARRY applies the nominal twin's exact optimal policy unchanged. From a random start, 20 rounds do not reach CARRY on 6 of 8 stress families. The loop is not separated from it on V2 (+0.0180) and ahead only on I2 (+0.0218 [+0.0101, +0.0306]), a label that becomes NOT_SEPARATED under the mean (verifier-reported) or a crossed bootstrap; the registered median governs. Exact TD-PMD shows the same sign on seven of the eight families, so on five of the six NEGATIVE families most of the shortfall (exact TD-PMD's gap is 0.68–0.85 of the loop's) comes from comparing 20 rounds from a random start with an oracle that starts converged; on F2 (0.37) the learned components account for most of it.

The regime-switch test gives the loop a warmer start: 20 rounds on the nominal twin, then 20 on the severe market. The loop then ends above CARRY on I2 (+0.0328 [+0.0148, +0.0420]), is not separated from it on F2 (−0.0020) and V2 (+0.0137), and ends below it on L2 (−0.0548). Under toxicity CARRY is a weak reference: the order-flow rule HIMB ends above the loop after the I2 switch (−0.0714 [−0.0827, −0.0562], descriptive). After the switch the learned actor also costs capture on F2 (+0.0054), V2 (+0.0080) and L2 (+0.0101), in a descriptive follow-up.

8What the results mean

• Improve the critic stage first, including the backup it is trained toward. In this 15-state testbed with a generous information budget, its excess is positive on every family, while the learned actor stays within the holds margin of exact PMD.
• Fix the target, not only the fit. In the follow-up, the untrained six-sample average loses about as much as the learned critic, so fitting that target better cannot remove most of the cost.
• Desk rules stay strong. The loop beats the best nominal rule where fees or toxic order flow break it (on F1 and I1 not under a crossed bootstrap), not where that rule is already near-optimal, and under toxicity the order-flow rule HIMB beats the loop.

9Open questions

1. Can a better critic target close the gap?

   Only one sample budget was tested, and the design does not split the critic's excess between the noise of its six samples and the network's error. Exact component swaps with untrained controls can check each improvement before adaptation is claimed.

2. Does the picture hold from a converged start?

   The adaptation test compares 20 rounds from a random policy with an old-regime oracle; no test starts from a converged nominal policy, and every test stops at 20 rounds.

3. Why the critic and not the actor?

   The decomposition is measured, not derived. No theory yet explains why the critic, not the actor, limits control.

10Notation and metrics

Negative-entropy policy mirror descent improves a full-support policy ${\pi }_k$ against an action-value estimate $Q_k$ with step $\eta >0$, in every state $s$:

whose solution is the softmax step the actor is trained toward:

Pairing it with a one-step evaluation $Q_{k+1}={𝒯}^{{\pi }_{k+1}}{Q}_k$ gives temporal-difference PMD (TD-PMD), where

The learned critic is trained toward the six-sample average $r(s,u)+\gamma \frac{1}{6}{\sum }_j{Q}_k(s_j^{\prime },u_j^{\prime })$, with $u_j^{\prime }~{\pi }_{k+1}(·\mid s_j^{\prime })$. The actor is fed $\eta =0.8$; its effective step lies in [0.795, 0.815].

Two metrics. Value capture is 0 for the uniform policy and 1 for the optimum, with $V^{\pi }$ the value of $\pi$ averaged over the 15 states; it is a control metric on a synthetic market, not trading P&L. The normalized loss is the sup-norm gap of the returned policy's action values, relative to the initial gap (lower is better):

Information budget: in every round the critic receives the exact expected reward table and six sampled successors for every state–action pair (270 samples per round), and every arm starts from the exact $Q^{{\pi }_0}$. Fee and volatility stress change only the reward table, so on F and V the regime change reaches the controller through exact rewards.

11Protocol and controls

Pre-registration. The holds and class rules were committed before any learned result, the tests and statistics code before any locked output was read, and the follow-up protocol before any of its rows was computed. SampleAvg was registered after the main results were known.

Holds rule. An arm holds on a family if its per-run ratio of median losses to exact TD-PMD is at most 1.5 in the median run and in at least 4 of 5 runs, fewer than 2 runs are systematically stalled (over 25% of markets stalled), and all numerical audits pass. On nominal markets ActorOnly, CriticOnly and FullLoop reach 1.008, 1.253 and 1.254, all 5 runs at or below the threshold, so the full loop holds. It fails on V2 (3 of 5 runs at or below 1.5) and on L2, where 4 of 5 runs have 17 of 64 markets stalled against a threshold of 16; the L2 failure reverses under the stall rule of the original PolicyAttention protocol (verifier-reported), and the paper does not interpret it either way.

Labels. Each comparison is a paired difference on 5 runs × 64 markets. Its statistic is the median of the 320 pooled values, with a 95% percentile interval from a hierarchical bootstrap that resamples runs, then markets within runs (10,000 replicates). The label is POSITIVE if the interval lies above 0, NEGATIVE if below, and NOT_SEPARATED otherwise, with no multiplicity adjustment. A follow-up crossed bootstrap that shares each market resample across runs changes 5 of 119 registered labels and 5 follow-up labels (PC4 and the holds and class labels not re-run), all to NOT_SEPARATED; the registered labels govern.

Controls. On G0 and the four severe families, zeroing the prior $\log {\pi }_k$ at inference raises the loss by +0.540 to +0.685, and zeroing the value input by +0.689 to +0.860: the network depends on both inputs, though the ablation is out of distribution and does not show that it computes the PMD step. Training the actor by cross-entropy instead of the proximal loss is not separated from the main models on any family (every interval inside [−0.0052, +0.0037]). Market-specific training helps: on G0 and each severe family the main models beat a model trained on random MDPs of the same size by +0.022 to +0.139.

Training. Nominal markets only (24 markets, 2,048 examples, 51,200 AdamW steps); 5 runs give 5 models.

12Limitations

• The market is stylized and exactly solvable (15 states, observed toxicity), and value capture says nothing about trading P&L. The controller also gets exact rewards, a generative sampler and an exact initial value table.
• The adaptation test starts cold: it compares 20 rounds from a random policy with an old-regime oracle, and every test stops at 20 rounds.
• Illiquidity is the weakest axis: the largest critic excess, losses to HBEST and HINV (the L1 HINV loss not under a crossed bootstrap), and the regime-switch test NEGATIVE on L2.
• No learned critic tested here closes the adaptation gap: after a switch the full loop does not clear CARRY on F2 and V2, where the exact-critic arm does (descriptive), and neither arm clears it on L2.
• The critic's excess combines the noise of its six samples with its approximation error; the design does not split them, and only one sample budget was tested.
• No theory explains why the critic, not the actor, limits control.
• The desk rules are fixed (HBEST is selected on nominal markets, not re-tuned per regime), and labels are pointwise, with no multiplicity adjustment.

13Sources

Numbers on these pages come from the paper's text and Table 1; the two charts are replotted from the paper's own figure data. The paper and poster PDFs are not yet posted.