New Tool Benchmarks LLM Spreadsheet Creation Abilities

Evaluating the capability of large language models (LLMs) to interact with and manipulate spreadsheets is a complex task. This research presents a novel pipeline called the "workbook time machine," designed to automate the creation of benchmarks for assessing LLMs' proficiency in generating derived spreadsheet objects. These objects include formulas, charts, pivot tables, and conditional formatting, covering a range of complexity. By applying this pipeline to public workbook corpora, the researchers generated `wtmcorpus`, a comprehensive collection of input workbook, output workbook, and query triples. From this larger corpus, they curated `wtmbench`, a focused benchmark comprising 150 tasks with queries at three distinct levels of specificity. Initial evaluations using `wtmbench` on existing spreadsheet manipulation agents and baseline models revealed several critical factors influencing LLM performance. These include the specificity of the query, the orchestration strategy of the agent, and the particular API used to control the spreadsheet. The findings highlight that these elements play a significant role in how effectively LLMs can perform Excel-related tasks.