Anticipatory RL Reduces Lag in Industrial Trajectory Tracking

Deep Reinforcement Learning (DRL) in industrial control often struggles with latency and overshooting because it typically reacts only to the current tracking error. To overcome these limitations and enable anticipatory control without excessive computational demands, a new predictive formulation has been developed. This approach enhances the DRL state space by incorporating target velocities and future reference horizons. The method was tested using Proximal Policy Optimization (PPO) on a simulated 1-degree-of-freedom (1-DoF) helicopter. Simulation results showed a remarkable nine-fold reduction in tracking error, decreasing the mean absolute deviation from 2.73 degrees to 0.31 degrees. However, direct transfer of these models to physical hardware revealed a "sim-to-real" gap. Interestingly, a simpler configuration, using a single, further look-ahead horizon, achieved the best real-world performance, matching the most complex model. This suggests that highly granular predictive data might not always be necessary for effective physical transfer, highlighting the importance of practical validation beyond simulation.