New Benchmark Evaluates VLM Memory Beyond Simple Accuracy

The memory capabilities of Large Language Models (LLMs) and Vision-Language Models (VLMs) are a recognized challenge, typically evaluated by their accuracy on long-context tasks. However, this narrow focus on accuracy overlooks crucial aspects vital for real-world, long-horizon applications. A new benchmark, ECCBench, has been developed to provide a more comprehensive evaluation of VLM memory. ECCBench measures memory along three axes: Efficiency (computational cost to retrieve information), Compression (ability to remember compressible inputs more effectively), and Calibration (the system's ability to express uncertainty and manage error costs). This protocol aims to reveal deeper insights into how VLMs handle and utilize information over extended periods. Initial findings using ECCBench indicate that while pre-trained VLMs can compress memory for text, they do not exhibit similar compression for video data. Furthermore, these models generally show poor calibration across both modalities. Interestingly, some non-Transformer architectures demonstrated superior compression-calibration trade-offs compared to RoPE Transformers, suggesting alternative architectural components might be more suitable for agents operating with long-term memory requirements.