GitOfThoughts Introduces Version Control for LLM Agent Reasoning

This research introduces GitOfThoughts, a novel approach to bring version control capabilities to the reasoning processes of large language model (LLM) agents. By treating each thought as a Git commit, scores as notes, and outcomes as tags, the system allows for the replay, auditing, and merging of an agent's decision-making history. This addresses the current ephemeral nature of LLM reasoning, where chains of thought and search branches typically vanish. The paper also investigates the effectiveness of memory in improving LLM accuracy across various formats, including markdown, vector, graph, and Git. Surprisingly, the findings indicate that memory does not reliably enhance accuracy for novel problems. Significant gains are observed only when the retrieved information is a near-duplicate of the current problem, suggesting memory primarily aids in direct answer retrieval rather than method transfer. Ultimately, GitOfThoughts emphasizes auditability, provenance, and mergeability as its primary benefits, offering these advantages at accuracy parity. The study highlights the importance of rigorous evaluation, even documenting a retracted result and a refuted hypothesis to demonstrate its commitment to scientific standards.