Reinforcement Learning Discovers Price Manipulation More Effectively Than Model-Based Methods.

The research investigates the effectiveness of model-free Reinforcement Learning (RL) agents in uncovering and capitalizing on price manipulation within financial markets. This is compared against conventional model-based strategies that assume a correct data-generating process but are susceptible to noisy parameter estimations. The study uses a single-asset market framework, specifically the Almgren-Chriss model, which incorporates non-linear permanent and linear temporary market impacts. The authors first established the existence of price-manipulative strategies in a discrete-time setting and calculated an optimal benchmark strategy using Sequential Least Squares Quadratic Programming under full information. They then compared two learning approaches: a model-based procedure that estimates impact parameters from simulated execution data, and an agnostic RL approach using Deep Deterministic Policy Gradient, trained on the same data volume. Results indicate that for intermediate volatility levels, the RL agent successfully identifies profitable manipulative strategies without explicit knowledge of the underlying market model, even with limited training data. Crucially, RL consistently outperformed the model-based approach when parameter estimates were affected by sampling error. However, for high volatility, neither method succeeded, and for low volatility, the model-based approach was superior. These findings underscore RL's potential in complex control problems and highlight the inherent risks of deploying learning algorithms in financial markets without proper safeguards.