RENDER Benchmark Controls LLM Memory Evaluation Artifacts

Evaluations of Large Language Model (LLM) memory and Retrieval-Augmented Generation (RAG) systems often overlook how the input is presented to the answering model. Systems can render conversational history in various formats, such as memory entries, summaries, typed records, or raw excerpts. The new RENDER benchmark addresses this by fixing the conversation content while systematically varying the "reader-facing artifact" – the specific format in which the history is provided to the LLM. RENDER employs a five-level packet ladder to localize when answer-bearing content enters the input, alongside deterministic templates that mimic common formats like ChatGPT-style entries, LangChain summaries, and MemGPT-style typed records. Testing across 500 LongMemEval questions and nine different models demonstrated that presenting history as matched-budget resolved packets significantly improved performance, beating recency-truncated raw dialogue by 42.4-72.6 points. Furthermore, among deployed-style templates, the best-worst performance spread was 24.6-48.8 points per model. ChatGPT-style entries generally showed higher point estimates than raw conversation for most models. The study also found that models scoring 0% on formal ledger packets could answer the same facts from natural-language entries at 45.4-53.4% accuracy. This effect persists even with retrieval noise and transfers to other benchmarks, underscoring the critical importance of how memory/RAG evaluations present information to the LLM.