Telco-GAIA Benchmarks Bilingual AI Agents for Telecom

Evaluating the capabilities of tool-using AI agents, especially in specialized domains, presents a significant challenge. To address this, a new benchmark called Telco-GAIA has been introduced, specifically designed for the telecommunications sector. This benchmark is bilingual, supporting both English and Arabic, and multi-modal, incorporating various data types. Telco-GAIA consists of 100 human-verified question-answering tasks, each demanding multi-hop reasoning, averaging 4.2 hops. These tasks require agents to process information from three heterogeneous sources: a static website snapshot (including HTML, images, and linked PDFs), a synthetic relational SQL database, and external web archives, spanning text, image, and tabular modalities. The benchmark is delivered as a sandboxed Docker environment, ensuring objective, deterministic, and reproducible evaluation through normalized exact string matching, without relying on LLM-as-a-Judge metrics. Initial evaluations of a purpose-built reference agent across twelve commercial and open LLMs reveal that Telco-GAIA is highly challenging. Even the strongest model solved only 71% of tasks, dropping to about 40% under a moderate cost budget. Visually grounded categories proved particularly difficult, with an average backend score below 30%, indicating substantial room for improvement in document and image understanding for AI agents.