New Benchmark Evaluates AI Coding Agents for Payment Integration

Developing AI coding agents capable of handling complex, repository-level software tasks like payment integration is challenging. These tasks demand agents to select products, implement client-server flows, verify outcomes, and maintain data consistency. To address this, the Alipay-PIBench has been created as a new benchmark specifically for evaluating coding agents on realistic Alipay payment integration scenarios. It features nine product-specific projects and 18 task instances, covering both basic functional completion and advanced risk-aware hardening. The benchmark uses detailed rubrics for static, unit, integration, and end-to-end checks, complemented by LLM-assisted semantic assessment. Initial evaluations of six coding agents show that "skill" access significantly improves performance, highlighting areas where models excel or struggle in payment domain requirements.