Diagnosing RL Challenges for Clinical AI Agents in FHIR Environments

The application of Reinforcement Learning (RL) to clinical protocol execution, such as checking lab values or placing FHIR orders, holds great promise due to the availability of verifiable rewards from clinical SMEs. However, this paper investigates the practical challenges of deploying RL in such environments, specifically auditing MedAgentBench v1/v2. The audit revealed a substantial "silent-finish ceiling" where inaction becomes the dominant RL strategy, leading to the creation of an improved MedAgentBench-v3 (MAB-v3) with a much lower ceiling. Training a Qwen3-8B model on MAB-v3 exposed two critical structural barriers: a "capability ceiling" where the base model has zero performance on many task types, preventing gradient-based learning, and a "format-knowledge barrier" where exact clinical codes are required but undiscoverable through exploration. Pure RL achieved only 18.2% pass rate compared to 34.1% for rule-based Supervised Fine-Tuning (SFT), with this gap entirely attributed to these barriers. The research introduces a taxonomy for decision, format-knowledge, and lookup tasks, which helps predict RL learnability. It concludes that a hybrid approach is necessary: SFT should be used to inject required codes and foundational knowledge, while RL can then be applied to learn complex conditional logic. This combined strategy is essential for effective clinical AI agent development.