Shield-Consistent Double Dueling Q-Learning with Auditable Inventory Risk Budgets for Trade Execution
DOI:
https://doi.org/10.54097/pzzyx492Keywords:
Optimal execution, safe reinforcement learning, Double DQN, action shielding, implementation shortfall, conditional value-at-riskAbstract
Large-order execution agents often propose actions that are subsequently changed by a risk-control layer. This creates a training-deployment mismatch: the value network estimates outcomes for unfiltered orders while actual execution follows the shield. We study shield-consistent Double Dueling Q-learning (SC-DDQN), which trains on the transitions generated by the same observable execution shield used at deployment, and extend it with an adaptive inventory-exposure envelope (BSC-DDQN). The shield operationalizes flow-sensitive urgency, schedule recovery, and deadline floors; the new envelope tightens the permissible post-trade inventory as observed volatility rises. We implement a finite-horizon, six-action execution environment with concave temporary impact and four persistent latent liquidity regimes. Three independently trained seeds are evaluated on 720 paired held-out synthetic paths against deterministic schedules, unshielded DDQN, post-hoc shielding, and shield-consistent ablations. SC-DDQN attains 15.05 basis points (bps) mean implementation shortfall, compared with 18.67 bps for time-weighted execution and 15.88 bps for post-hoc shielded DDQN; the risk-budget extension instead reaches 16.24 bps but reduces inventory exposure. A seed-level sensitivity analysis yields a 95% interval of [-0.50, 2.16] bps for the 0.83-bps advantage over post-hoc shielding; only three independent fits are available. We report tail losses, intervention frequency, and impact-coefficient sensitivity. All outcomes are generated by a disclosed simulator; they measure the interaction between learning and explicit execution constraints rather than realized venue profitability.
Downloads
References
[1] Bertsimas, D., & Lo, A. W. (1998). Optimal control of execution costs. Journal of Financial Markets, 1(1), 1–50. https://doi.org/10.1016/S1386-4181(97)00012-8
[2] Almgren, R., & Chriss, N. (2001). Optimal execution of portfolio transactions. Journal of Risk, 3(2), 5–39. https://doi.org/10.21314/JOR.2001.041
[3] Gatheral, J. (2010). No-dynamic-arbitrage and market impact. Quantitative Finance, 10(7), 749–759. https://doi.org/10.1080/14697680903373692
[4] Cont, R., Kukanov, A., & Stoikov, S. (2014). The price impact of order book events. Journal of Financial Econometrics, 12(1), 47–88. https://doi.org/10.1093/jjfinec/nbt003
[5] Nevmyvaka, Y., Feng, Y., & Kearns, M. (2006). Reinforcement learning for optimized trade execution. Proceedings of the 23rd International Conference on Machine Learning, 673–680. https://doi.org/10.1145/1143844.1143929
[6] Ning, B., Lin, F. H. T., & Jaimungal, S. (2021). Double Deep Q-Learning for optimal execution. Applied Mathematical Finance, 28(4), 361–380. https://doi.org/10.1080/1350486X.2022.2077783
[7] Lin, S., & Beling, P. A. (2020). An end-to-end optimal trade execution framework based on proximal policy optimization. Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, 4548–4554. https://doi.org/10.24963/ijcai.2020/627
[8] Fang, Y., Yu, L., Liu, D., & Zou, B. (2021). Universal trading for order execution with oracle policy distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(1), 107–115. https://doi.org/10.1609/aaai.v35i1.16083
[9] Alshiekh, M., Bloem, R., Ehlers, R., Könighofer, B., Niekum, S., & Topcu, U. (2018). Safe reinforcement learning via shielding. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2669–2678. https://doi.org/10.1609/aaai.v32i1.11797
[10] Qin, Y., Zou, J., Guo, Z., & Zi, B. (2026). An adaptive deep reinforcement learning framework for intelligent transaction execution in internet finance. World Scientific Research Journal, 12(10), 16–31. https://wsr-j.org/index.php/ojs/article/view/152
[11] Almgren, R. F. (2003). Optimal execution with nonlinear impact functions and trading-enhanced risk. Applied Mathematical Finance, 10(1), 1–18. https://doi.org/10.1080/135048602100056
[12] Almgren, R., & Lorenz, J. (2011). Mean-variance optimal adaptive execution. Applied Mathematical Finance, 18(5), 395–422. https://doi.org/10.1080/1350486X.2011.560707
[13] Gabaix, X., Gopikrishnan, P., Plerou, V., & Stanley, H. E. (2006). Institutional investors and stock market volatility. The Quarterly Journal of Economics, 121(2), 461–504. https://doi.org/10.1162/qjec.2006.121.2.461
[14] Toth, B., Lillo, F., Farmer, J. D., & Mantegna, R. N. (2011). Anomalous price impact and the critical nature of liquidity in financial markets. Physical Review X, 1, Article 021006. https://doi.org/10.1103/PhysRevX.1.021006
[15] Gould, M. D., Porter, M. A., Williams, S., McDonald, M., Fenn, D. J., & Howison, S. D. (2013). Limit order books. Quantitative Finance, 13(11), 1709–1742. https://doi.org/10.1080/14697688.2013.803148
[16] Sirignano, J., & Cont, R. (2019). Universal features of price formation in financial markets: perspectives from deep learning. Quantitative Finance, 19(9), 1449–1459. https://doi.org/10.1080/14697688.2019.1622295
[17] Makarov, I., & Schoar, A. (2020). Trading and arbitrage in cryptocurrency markets. Journal of Financial Economics, 135(2), 293–319. https://doi.org/10.1016/j.jfineco.2019.07.001
[18] Macri, A., & Lillo, F. (2024). Reinforcement learning for optimal execution when liquidity is time-varying. Applied Mathematical Finance, 31(5), 312–342. https://doi.org/10.1080/1350486X.2025.2490157
[19] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533. https://doi.org/10.1038/nature14236
[20] van Hasselt, H., Guez, A., & Silver, D. (2016). Deep reinforcement learning with Double Q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1), 2094–2100. https://doi.org/10.1609/aaai.v30i1.10295
[21] Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., & Freitas, N. (2016). Dueling network architectures for deep reinforcement learning. Proceedings of the 33rd International Conference on Machine Learning, 48, 1995–2003. https://proceedings.mlr.press/v48/wangf16.html
[22] Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2016). Prioritized experience replay. International Conference on Learning Representations. https://arxiv.org/abs/1511.05952
[23] Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., & Silver, D. (2018). Rainbow: Combining improvements in deep reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 3215–3222. https://doi.org/10.1609/aaai.v32i1.11796
[24] Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. International Conference on Learning Representations. https://arxiv.org/abs/1412.6980
[25] Ntakaris, A., Magris, M., Kanniainen, J., Gabbouj, M., & Iosifidis, A. (2018). Benchmark dataset for mid-price forecasting of limit order book data with machine learning methods. Journal of Forecasting, 37(8), 852–866. https://doi.org/10.1002/for.2543
[26] Zhang, Z., Zohren, S., & Roberts, S. (2019). DeepLOB: Deep convolutional neural networks for limit order books. IEEE Transactions on Signal Processing, 67(11), 3001–3012. https://doi.org/10.1109/TSP.2019.2907260
[27] Rockafellar, R. T., & Uryasev, S. (2000). Optimization of conditional value-at-risk. Journal of Risk, 2(3), 21–41. https://doi.org/10.21314/JOR.2000.038
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Mathematical Finance and Risk Management
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
