# In-Context PMD in Market Making: How It Works

> How the ICAIF 2026 component diagnosis works: an exactly solvable market-making family, the in-context PMD loop and its exact swap, what the actor and the critic cost, the payoff against desk rules, and the limitations.

Web page: https://qrsntu.org/projects/policyattention/icaif/details/ (content last changed 2026-11-07)
Site index for agents: https://qrsntu.org/llms.txt

[← ICAIF 2026 overview](https://qrsntu.org/projects/policyattention/icaif/)

PolicyAttention · ICAIF 2026 · How it works

Start with the idea and the pictures. Definitions, the test protocol, the full table and the limitations are collected at the end.

Authors: [Xinyue Wu](https://qrsntu.org/people/xinyue-wu/), [Yuhe Sui](https://qrsntu.org/yh/), [Shufang Chen](https://qrsntu.org/people/shufang-chen/), [Yingzhi Tang](https://qrsntu.org/yz/)

Contents

1. The idea
2. Why take the controller apart?
3. A market we can solve exactly
4. The loop and the swap
5. The evidence
6. What each part costs
7. By round and by family
8. Against desk rules
9. Adapting after a shift
10. What the results mean
11. Open questions
12. Technical details
13. Notation and metrics
14. Protocol and controls
15. Limitations
16. Sources

Part 1 · The idea

## 1Why take the controller apart?

A market maker keeps re-setting its quote skew as fees, volatility, liquidity and order-flow toxicity change. In-context reinforcement learning offers a controller that adapts without being re-engineered for each regime: a sequence model with frozen weights adapts at inference time by conditioning on recent experience. When such a controller does badly after a regime shift, the practitioner needs to know which learned component failed before deciding what to fix.

End-to-end scores mix three things: how well the policy-improvement step is learned, how well the value estimate it relies on is learned, and how hard the new regime is. A fixed rule also looks best in the regime it was designed for.

In an exactly solvable market, each learned component can be swapped for its exact counterpart on the same market and the same initial policy. A failure then becomes a set of measured component costs under each stress.

## 2A market we can solve exactly

The dealer holds inventory $q$ from −2 to 2 and observes order-flow imbalance $z ∈ { − 1,0, + 1}$, giving 15 states. Each step it sets a quote skew $u ∈ { − 1,0, + 1}$; at most one fill occurs. Buy pressure raises both the chance that the dealer's ask is hit and the chance that the price then rises: the adverse-selection channel, carried here by an observed state. The reward is spread income minus fees, plus mark-to-market, minus a mean–variance inventory penalty, with discount 0.8.

There are 9 families of 64 paired markets: nominal (G0) and fee, volatility, liquidity and toxicity stress (F, V, L, I), each at a moderate (1) and a severe (2) dose. Every nominal market has a twin in each stress family that re-draws one group of parameters and keeps the same random initial policy, so every comparison is paired market by market. At the severe dose, fees rise from [0, 0.2] to [0.6, 0.8], volatility from [0.8, 1.2] to [1.8, 2.2], and liquidity falls from [1, 2] to [0.3, 0.6].

The family is not trivial. On nominal markets (G0) the inventory rule HINV reaches a median value capture of 0.678 and exact PMD from the initial policy reaches 0.883 after 20 rounds, and severe stress changes at least one optimal action in 95–100% of markets.

## 3The loop and the swap in one picture

**The in-context loop and the exact swap.** Redrawn after Fig. 1 of the paper, with the formulas in words; they are written out under Notation and metrics.

- **Actor.** Reads the policy $π_k$, the action values $Q_k$ and the step $η$, and returns $π_{k + 1}$. It is trained toward the PMD step, $softmax( log π_k + ηQ_k )$.
- **Critic.** Reads $Q_k$, the reward table and 6 sampled successors per state–action pair, and returns $Q_{k + 1}$. It is trained toward the average backup of those six samples.
- **The arms.** Exact runs exact PMD with the exact backup (exact TD-PMD, the reference). ActorOnly learns only the actor, CriticOnly only the critic, FullLoop both. The follow-up arm SampleAvg replaces the critic network by the plain six-sample average it is trained to output, computed from the same samples.

The controller is [PolicyAttention](https://qrsntu.org/projects/policyattention/): one Transformer (4 layers, 4 heads, width 64) trained on nominal markets only, in 5 runs. Its weights stay frozen across rounds, so it adapts only through its context. Every arm starts from the same random policy and its exact action values, and runs 20 rounds.

Part 2 · The evidence

### Tested synthetic, pre-registered

- Each component's cost on all 9 families: 5 trained runs × 64 paired markets.
- The full loop against desk rules and against the carried old-regime optimum.
- Holds rules committed before any learned result; two-sided tests and statistics code before any locked output was read.
- Follow-up, registered after those results: the untrained SampleAvg arm.

### Not tested open

- Real markets and trading P&L.
- A converged starting policy, or more than 20 rounds.
- Any sample budget other than six successors per state–action pair.

## 4What each part costs after 20 rounds

Each cost is a paired difference in normalized loss against exact TD-PMD on the same market: the median of 320 paired values (5 runs × 64 markets) with a 95% hierarchical-bootstrap interval. Lower is better.

### Learned actor

Pre-registered · nominal markets

+0.0020

interval [−0.0002, +0.0043], which includes 0. Within the holds margin of exact TD-PMD on all nine families (ratios 0.990–1.086).

Small but positive under severe volatility (+0.0035) and severe toxicity (+0.0062).

### Learned critic

Pre-registered · nominal markets

+0.0219

interval [+0.0186, +0.0248]. Positive on every family, from +0.0191 (F1) to +0.0580 (L2). The full loop's excess, +0.0235, is almost all the critic's.

In a descriptive follow-up, no interaction between the two parts is separated from 0 on eight families (NEGATIVE on F2, −0.0006, not under a crossed bootstrap).

### Untrained average

Follow-up · all nine families

0.70–1.10×

the learned critic's excess, for SampleAvg, the plain average of the same six samples (+0.0197 to +0.0450 over exact TD-PMD).

The learned critic is not separated from it on eight families. The design does not split the excess into sampling noise and network error.

So a critic that fitted its six-sample target perfectly would keep at least 70% of the cost: improving the critic stage means improving what it is trained toward, not only how well it fits. On L1 the network adds +0.0025 [+0.0005, +0.0045] beyond the average, a label that becomes NOT_SEPARATED under a crossed bootstrap.

## 5By round and by family

**Excess loss over exact TD-PMD, round by round.** Median of 320 paired values per round; descriptive, with no per-round intervals. Medians of paired differences do not subtract: on L2 the CriticOnly and SampleAvg curves end 0.017 apart, while their paired difference is +0.0042 (NOT_SEPARATED). Replotted from the paper's figure data (paper Fig. 2).

| Family | ℓ(Exact) | ActorOnly | CriticOnly | FullLoop | SC1 · SampleAvg − Exact | SC2 · CriticOnly − SampleAvg |
|---|---|---|---|---|---|---|
| G0 nominal | 0.133 | +0.0020 [−0.0002, +0.0043] | +0.0219 [+0.0186, +0.0248] | +0.0235 [+0.0188, +0.0277] | +0.0220* | −0.0007 |
| F1 fee, moderate | 0.113 | +0.0018 [−0.0010, +0.0042] | +0.0191 [+0.0143, +0.0212] | +0.0200 [+0.0164, +0.0254] | +0.0197* | −0.0002 |
| F2 fee, severe | 0.100 | +0.0018 [−0.0009, +0.0043] | +0.0221 [+0.0177, +0.0275] | +0.0234 [+0.0194, +0.0297] | +0.0222* | +0.0003 |
| V1 volatility, moderate | 0.118 | +0.0026 [−0.0003, +0.0081] | +0.0260 [+0.0213, +0.0315] | +0.0312 [+0.0230, +0.0395] | +0.0272* | −0.0010 |
| V2 volatility, severe | 0.086 | +0.0035 [+0.0003, +0.0118] | +0.0212 [+0.0170, +0.0266] | +0.0298^× [+0.0201, +0.0418] | +0.0218* | +0.0007 |
| L1 illiquidity, moderate | 0.190 | +0.0023 [−0.0013, +0.0063] | +0.0518 [+0.0401, +0.0691] | +0.0595 [+0.0404, +0.0754] | +0.0450* | +0.0025*† |
| L2 illiquidity, severe | 0.243 | +0.0024 [−0.0022, +0.0066] | +0.0580 [+0.0442, +0.0714] | +0.0602^× [+0.0496, +0.0748] | +0.0406* | +0.0042 |
| I1 toxicity, moderate | 0.152 | +0.0013 [−0.0019, +0.0048] | +0.0247 [+0.0190, +0.0327] | +0.0282 [+0.0218, +0.0325] | +0.0271* | −0.0002 |
| I2 toxicity, severe | 0.130 | +0.0062 [+0.0014, +0.0133] | +0.0263 [+0.0175, +0.0322] | +0.0287 [+0.0191, +0.0385] | +0.0287* | +0.0020 |

Paired excess normalized loss over exact TD-PMD at round 20, median of 320 paired values with its 95% hierarchical-bootstrap interval (paper Table 1). SC1 and SC2 are follow-up comparisons. *: the interval excludes 0; †: NOT_SEPARATED under a crossed bootstrap; ×: the full loop fails the holds rule (see Protocol and controls). CriticOnly also fails it on V2 (ratio 1.514, 1 of 5 runs at or below 1.5).

## 6Where the loop pays off against desk rules

HBEST is the best of 45 linear skew rules on 256 separate nominal markets (a = 1, b = −1, c = −0.5; mean capture 0.872 there). It leans against inventory and order flow and is not re-tuned per regime. HIMB skews against order flow alone ($u = − z$).

**Value-capture difference $C(FullLoop) − C(rule)$ at round 20**, with 95% hierarchical-bootstrap intervals; every label shown is separated from 0 (†: F1 and I1 against HBEST are not under a crossed bootstrap). Black dashes: exact TD-PMD in the loop's place (descriptive reference). Replotted from the paper's figure data (paper Fig. 3); the comparison with HINV is in the paper.

- **Fee and toxicity stress break HBEST.** Its median capture falls from 0.931 nominally to 0.677 on F2 and 0.714 on I2, while the loop keeps 0.878 and 0.849. The paired median gain over HBEST is +0.1661 [+0.1363, +0.2159] on F2 and +0.1247 on I2.
- **Volatility and illiquidity do not.** There HBEST is close to optimal (0.958–0.989 on V2, L1 and L2) and the loop reaches 0.893, 0.778 and 0.740. Exact TD-PMD shows the same signs, so these losses are not the learned parts' alone: 20 rounds from a random start do not reach a rule that is already near-optimal for those markets.
- **Under toxicity, order flow is enough.** HIMB beats the loop (−0.0761 on I1, −0.0910 on I2), and exact TD-PMD too (−0.0530, −0.0547).

On the fee families the change reaches the loop through the exact reward table it is given, which the rules never see, so part of the fee gain may reflect that information advantage.

## 7Adapting after a shift

Is adapting in context better than keeping the old optimum? CARRY applies the nominal twin's exact optimal policy unchanged. From a random start, 20 rounds do not reach CARRY on 6 of 8 stress families. The loop is not separated from it on V2 (+0.0180) and ahead only on I2 (+0.0218 [+0.0101, +0.0306]), a label that becomes NOT_SEPARATED under the mean (verifier-reported) or a crossed bootstrap; the registered median governs. Exact TD-PMD shows the same sign on seven of the eight families, so on five of the six NEGATIVE families most of the shortfall (exact TD-PMD's gap is 0.68–0.85 of the loop's) comes from comparing 20 rounds from a random start with an oracle that starts converged; on F2 (0.37) the learned components account for most of it.

The regime-switch test gives the loop a warmer start: 20 rounds on the nominal twin, then 20 on the severe market. The loop then ends above CARRY on I2 (+0.0328 [+0.0148, +0.0420]), is not separated from it on F2 (−0.0020) and V2 (+0.0137), and ends below it on L2 (−0.0548). Under toxicity CARRY is a weak reference: the order-flow rule HIMB ends above the loop after the I2 switch (−0.0714 [−0.0827, −0.0562], descriptive). After the switch the learned actor also costs capture on F2 (+0.0054), V2 (+0.0080) and L2 (+0.0101), in a descriptive follow-up.

## 8What the results mean

- **Improve the critic stage first,** including the backup it is trained toward. In this 15-state testbed with a generous information budget, its excess is positive on every family, while the learned actor stays within the holds margin of exact PMD.
- **Fix the target, not only the fit.** In the follow-up, the untrained six-sample average loses about as much as the learned critic, so fitting that target better cannot remove most of the cost.
- **Desk rules stay strong.** The loop beats the best nominal rule where fees or toxic order flow break it (on F1 and I1 not under a crossed bootstrap), not where that rule is already near-optimal, and under toxicity the order-flow rule HIMB beats the loop.

## 9Open questions

1. ### Can a better critic target close the gap?
   Only one sample budget was tested, and the design does not split the critic's excess between the noise of its six samples and the network's error. Exact component swaps with untrained controls can check each improvement before adaptation is claimed.
2. ### Does the picture hold from a converged start?
   The adaptation test compares 20 rounds from a random policy with an old-regime oracle; no test starts from a converged nominal policy, and every test stops at 20 rounds.
3. ### Why the critic and not the actor?
   The decomposition is measured, not derived. No theory yet explains why the critic, not the actor, limits control.

Part 3 · Technical details

## 10Notation and metrics

Negative-entropy policy mirror descent improves a full-support policy $π_k$ against an action-value estimate $Q_k$ with step $η > 0$, in every state $s$:

$$π_{k + 1} ( · ∣s) = argmax_{ν ∈ Δ_A } η⟨ν,Q_k (s, · )⟩ − KL(ν‖π_k ( · ∣s))$$

whose solution is the softmax step the actor is trained toward:

$$π_{k + 1} ( · ∣s) = softmax( log π_k ( · ∣s) + ηQ_k (s, · )).$$

Pairing it with a one-step evaluation $Q_{k + 1} = 𝒯^{π_{k + 1} } Q_k$ gives temporal-difference PMD (TD-PMD), where

$$(𝒯^π Q)(s,u) = r(s,u) + γ𝔼_{s^′ } ⟨π( · ∣s^′ ),Q(s^′ , · )⟩.$$

The learned critic is trained toward the six-sample average $r(s,u) + γ\frac{1}{6}∑_j Q_k (s_j^′ ,u_j^′ )$, with $u_j^′ ~π_{k + 1} ( · ∣s_j^′ )$. The actor is fed $η = 0.8$; its effective step lies in [0.795, 0.815].

Two metrics. **Value capture** is 0 for the uniform policy and 1 for the optimum, with $V^π$ the value of $π$ averaged over the 15 states; it is a control metric on a synthetic market, not trading P&L. The **normalized loss** is the sup-norm gap of the returned policy's action values, relative to the initial gap (lower is better):

$$C(π) = \frac{V^π − V^{UNIF} }{V^{OPT} − V^{UNIF} }$$

$$ℓ(π) = \frac{‖Q^⋆ − Q^π ‖_∞ }{‖Q^⋆ − Q^{π_0 } ‖_∞ }$$

Information budget: in every round the critic receives the exact expected reward table and six sampled successors for every state–action pair (270 samples per round), and every arm starts from the exact $Q^{π_0 }$. Fee and volatility stress change only the reward table, so on F and V the regime change reaches the controller through exact rewards.

## 11Protocol and controls

**Pre-registration.** The holds and class rules were committed before any learned result, the tests and statistics code before any locked output was read, and the follow-up protocol before any of its rows was computed. SampleAvg was registered after the main results were known.

**Holds rule.** An arm holds on a family if its per-run ratio of median losses to exact TD-PMD is at most 1.5 in the median run and in at least 4 of 5 runs, fewer than 2 runs are systematically stalled (over 25% of markets stalled), and all numerical audits pass. On nominal markets ActorOnly, CriticOnly and FullLoop reach 1.008, 1.253 and 1.254, all 5 runs at or below the threshold, so the full loop holds. It fails on V2 (3 of 5 runs at or below 1.5) and on L2, where 4 of 5 runs have 17 of 64 markets stalled against a threshold of 16; the L2 failure reverses under the stall rule of the original PolicyAttention protocol (verifier-reported), and the paper does not interpret it either way.

**Labels.** Each comparison is a paired difference on 5 runs × 64 markets. Its statistic is the median of the 320 pooled values, with a 95% percentile interval from a hierarchical bootstrap that resamples runs, then markets within runs (10,000 replicates). The label is POSITIVE if the interval lies above 0, NEGATIVE if below, and NOT_SEPARATED otherwise, with no multiplicity adjustment. A follow-up crossed bootstrap that shares each market resample across runs changes 5 of 119 registered labels and 5 follow-up labels (PC4 and the holds and class labels not re-run), all to NOT_SEPARATED; the registered labels govern.

**Controls.** On G0 and the four severe families, zeroing the prior $log π_k$ at inference raises the loss by +0.540 to +0.685, and zeroing the value input by +0.689 to +0.860: the network depends on both inputs, though the ablation is out of distribution and does not show that it computes the PMD step. Training the actor by cross-entropy instead of the proximal loss is not separated from the main models on any family (every interval inside [−0.0052, +0.0037]). Market-specific training helps: on G0 and each severe family the main models beat a model trained on random MDPs of the same size by +0.022 to +0.139.

**Training.** Nominal markets only (24 markets, 2,048 examples, 51,200 AdamW steps); 5 runs give 5 models.

## 12Limitations

- The market is stylized and exactly solvable (15 states, observed toxicity), and value capture says nothing about trading P&L. The controller also gets exact rewards, a generative sampler and an exact initial value table.
- The adaptation test starts cold: it compares 20 rounds from a random policy with an old-regime oracle, and every test stops at 20 rounds.
- Illiquidity is the weakest axis: the largest critic excess, losses to HBEST and HINV (the L1 HINV loss not under a crossed bootstrap), and the regime-switch test NEGATIVE on L2.
- No learned critic tested here closes the adaptation gap: after a switch the full loop does not clear CARRY on F2 and V2, where the exact-critic arm does (descriptive), and neither arm clears it on L2.
- The critic's excess combines the noise of its six samples with its approximation error; the design does not split them, and only one sample budget was tested.
- No theory explains why the critic, not the actor, limits control.
- The desk rules are fixed (HBEST is selected on nominal markets, not re-tuned per regime), and labels are pointwise, with no multiplicity adjustment.

## 13Sources

- ### Where Does Learned In-Context Policy Mirror Descent Pay Off in Market Making? A Component Diagnosis
  [Xinyue Wu](https://qrsntu.org/people/xinyue-wu/), [Yuhe Sui](https://qrsntu.org/yh/), [Shufang Chen](https://qrsntu.org/people/shufang-chen/), [Yingzhi Tang](https://qrsntu.org/yz/)
  ICAIF 2026 Workshop on AI-Driven Market Microstructure in Centralized and Decentralized Finance · Milan · Poster 03 [[Workshop listing]](https://icaif2026.org/workshop-papers/#paper-ai-market-microstructure-1245) [[Workshop]](https://icaif-ai-driven.github.io/)

Numbers on these pages come from the paper's text and Table 1; the two charts are replotted from the paper's own figure data. The paper and poster PDFs are not yet posted.