RLVR Boosts LLM Tool-Use in Atlassian Workflows

Large language models (LLMs) are primarily trained for next-token prediction, which often leads to "silent failures" when applied to complex tool-use tasks within enterprise SaaS APIs. These failures include dropped required fields, hallucinated tools, or premature stops. This research explores whether Reinforcement Learning with Verifiable Rewards (RLVR) can bridge this gap by training models directly on the desired outcomes in the target environment. As a proof of concept, a suite of five synthetic environments was built, accurately emulating Jira REST v3 and Confluence v2 APIs. Rewards were computed solely from tool-call traces, eliminating the need for live APIs, learned judges, or human labels. The study evaluated prompted Qwen3-1.7B and Qwen3.5-4B models. Results showed that RLVR-trained policies dramatically improved average rewards, from a baseline range of 0.35-0.92 to 0.95-1.00, on four out of five scenarios with non-degenerate rewards. The most significant gain was observed in Confluence page creation, improving from 0.35 to 1.00. This suggests RLVR is a promising step towards developing outcome-optimized small models for niche enterprise APIs, though the current method of hand-crafting verifiable rewards may not scale easily.