Themis: XAI Framework for RL with Human Feedback

Training safe Reinforcement Learning (RL) systems presents inherent challenges, particularly in guaranteeing the avoidance of undesirable behaviors. Two highly effective strategies for mitigating these risks are enhancing transparency through explainability (XAI) and ensuring alignment via human feedback. Despite their individual promise, a comprehensive, publicly available framework that integrates both capabilities has been lacking. To bridge this gap, researchers introduce Themis, an XAI-enabled testing and evaluation framework specifically designed for Reinforcement Learning from Human Feedback (RLHF). Themis is highly versatile, supporting over 200 widely used environments, and is easily configurable for various experiments in RL, transparency, and alignment. The framework demonstrates its effectiveness by training reward models that either match or surpass the true reward signal of an environment using human preferences. Additionally, Themis includes a user-friendly, auto-scalable cloud-based platform for efficiently collecting human feedback and managing experiments, capable of supporting large participant groups across multiple experiments without requiring extra development effort.