New Automata-Based Rewards Boost RL for STL Specifications

Designing controllers for complex real-world systems, especially those lacking precise models, is a significant challenge. Signal Temporal Logic (STL) offers a formal language to specify real-time properties for such systems, along with a quantitative robustness score to monitor their satisfaction. While using STL robustness scores as rewards in reinforcement learning (RL) has been explored, it often leads to intractable state space expansion for long and complex specifications due to the history-dependent nature of robustness.This paper presents an innovative automata-based solution to this problem. The approach constructs a timed alternating automaton directly from the given STL specifications. This automaton then augments the RL state space with its locations and clock valuations, enabling the derivation of efficient Markovian rewards based on the automaton's acceptance conditions. This mechanism provides a more efficient memory for handling complex temporal logic.Empirical evaluations demonstrate that this new approach allows RL frameworks to learn control policies that achieve significantly higher robustness scores and satisfaction rates. This improvement is particularly notable when compared to existing methods that rely solely on robustness-based rewards, highlighting the benefit of the automata-based memory mechanism for complex, long-horizon specifications.