IFCMemoryBench Evaluates LLM Agent Long-Term Memory in BIM.

While long-term memory is becoming a critical feature for LLM-based agents, current evaluation methods often focus on conversational recall in general settings. A new benchmark, IFCMemoryBench, proposes a more rigorous test by evaluating agents' ability to retain and apply information from past interactions within a live, structured, and domain-specific environment: Building Information Modeling (BIM). This professional engineering workflow requires agents to query large IFC models while also drawing upon project specifications, client decisions, and engineering conventions often discussed in conversations but not explicitly in the model. IFCMemoryBench comprises 143 multi-session tasks across 19 projects, incorporating 4,016 prior sessions. These tasks are derived from incomplete-information questions, where agents must combine remembered context from earlier conversations with live IFC queries to provide answers. The evaluation framework breaks down memory performance into ingestion, retrieval, and utilization, using expert-validated LLM judges for both answer and memory quality. Initial evaluations of vector-, graph-, and file-based memory systems show that even the strongest system achieves only 32.4% accuracy under realistic conditions, highlighting a significant gap in agent memory for domain-specific professional tasks.