Key Factors for Strong Lightweight Game-Playing AI Agents Identified

Developing strong reinforcement learning agents for complex games, especially those with imperfect information like card games, presents significant challenges. Traditional evaluation methods often fall short, as agents quickly outperform random opponents and merely tie with copies of themselves. This research addresses the evaluation problem by creating a robust, fixed, rule-based expert for Gin Rummy, used solely as a performance yardstick rather than for training. Through over a hundred experimental runs, the study systematically isolated the factors contributing to a lightweight agent's strength. Key findings indicate that trust region updates, precise reward shaping, a curriculum of progressively tougher opponents, warm starting, and retaining the best performing checkpoint are all highly beneficial. Combining these elements significantly boosted a self-play champion's performance against the expert. Conversely, several commonly explored ideas proved ineffective or too resource-intensive for lightweight agents. These included short-term and long-term reward shaping, learned state embeddings, imitation learning (DAgger), and using a live large language model as an opponent. The research also compared various neural network encoders, concluding that increased model capacity did little to overcome performance ceilings, suggesting that information availability, rather than network size, is the primary limiting factor. The findings were validated on Leduc Hold'em, and a reusable, game-agnostic recipe for training competitive lightweight agents is provided.