AISE-Bench Challenges LLM Agents in Academic Information Seeking

Large language models (LLMs) augmented with tools are increasingly used as autonomous agents for complex, long-horizon tasks, including information seeking. However, existing benchmarks for evaluating these agents on academic knowledge graphs often rely on simplified scenarios, synthetic templates, or narrow tasks, failing to capture the full complexity of real-world user intent, multi-step API planning, and grounded answer generation. To address these limitations, AISE-Bench has been introduced as a real-world, full-cycle annotated benchmark for information seeking on academic knowledge graphs. It comprises 1,133 question-answer pairs, complete with query taxonomies, detailed API execution trajectories, validated parameters, and source-grounded answers linked to references. A customized agent workflow was designed to facilitate high-quality annotation, enabling annotators to efficiently plan, execute, and revise complex API workflows. The benchmark includes a comprehensive evaluation protocol that measures answer quality, reference grounding, API-planning correctness, and execution success. In tests involving 14 different methods, even the most advanced model (PLAY2PROMPT with Gemini-3-Pro) achieved only moderate performance, frequently encountering difficulties with API planning and execution. AISE-Bench thus serves as a challenging new testbed for rigorously evaluating and improving the stepwise correctness, grounded summarization, and traceable reasoning abilities of multi-step API-using LLM agents.