5.47 DRL Market Making With a Periodic Signal and Real Latency
A tick-level DRL market maker with real submit and cancel latency: PPO beats DQN and Avellaneda-Stoikov, an alpha signal nearly doubles it, and slower cancels raising profit is a risk trap.
Most reinforcement-learning market-making papers cheat in the same two places, and once you know where to look you stop believing their equity curves. First, they take event-driven tick data, the actual stream of order placements, cancellations, and fills, and resample it onto a fixed clock, one snapshot every second or every hundred ticks, which throws away the exact microstructure that a market maker lives and dies on. Second, they assume your order hits the book the instant you decide to send it, which is a fantasy in a business where the whole game is measured in milliseconds. Jiafa He, Cong Zheng, and Can Yang at HKUST built a market maker that refuses both shortcuts: it runs tick-by-tick in physical time on a reconstructed level-3 order book from Coinbase BTC-USD, and it carries explicit latency for both submitting and cancelling orders.
The headline results are what you would hope for. Proximal Policy Optimization beats Double, Dueling, and Noisy DQN, and beats a fixed-spread quoter and a simplified Avellaneda-Stoikov model. Bolting a periodical alpha signal onto the tick-level agent nearly doubles PPO's return on real data. But the interesting parts are the two things that should make you cautious: a latency result that looks like good news and is actually a risk artifact, and a simulator the authors themselves admit is missing the one behavior that matters most. The old article "The Three Pillars of Market Making: Fair Price, Spread, Skew" laid out what a quote actually is; the old article "Why Market-Making Simulations Don't Work" laid out why most of these backtests lie. This paper is a good chance to hold both in your head at once.
The two risks you are actually fighting
A market maker posts a bid and an ask and hopes to earn the spread on the round trip. Two risks stand between the maker and that spread. The first is inventory risk: you accumulate a position you did not want because only one of your two quotes kept filling. The paper's own example: if the true fair price is 90 dollars and you misjudge it as somewhere between 95 and 105, you quote a bid at 90 and an ask at 105, only your bid fills, and now you are long an asset that is about to fall. The second is adverse selection: your resting limit orders are public, so a faster, better-informed trader picks off the stale one before you can move it. The old article "The Three Pillars of Market Making: Fair Price, Spread, Skew" made the point that skew's real job is not managing ordinary variance but blocking informed traders from loading you with toxic inventory. Everything in this paper is an attempt to control those two risks tick by tick: the tick-level agent fights inventory risk with fast reactions, and the periodical alpha signal fights adverse selection by predicting where price is about to go.
The simulator: tick intervals and a random-walk mid
Before touching real data, the authors build a simulator calibrated to real Coinbase top-of-book data so they can test in controlled conditions. Two random pieces drive it. The time between consecutive ticks is drawn from a log-normal distribution.
$$ \Delta T = e^{\mu + \sigma Z} $$