Mimeo Tool Compiles Expert Knowledge for AI Agents

Providing AI agents with specific knowledge about named experts can offer several benefits, such as access to specialized information, the ability to adopt a distinct persona, or even influencing the agent's decision-making. However, the extent to which these benefits are realized, particularly regarding the transfer of expert judgment, requires careful testing. Mimeo is an open-source tool designed to address this by finding and compiling a person's public work into a file that an AI agent can load. The tool rigorously checks each extracted quotation against its source text, ensuring accuracy. In tests, mimeo significantly enhanced knowledge access, enabling agents to answer obscure, quotation-heavy questions that closed-book conditions could not. It also improved grounding, preventing agents from misstating documented positions when adopting personas. While mimeo successfully provided agents with expert knowledge and helped maintain consistent personas, the transfer of expert judgment proved inconclusive. Both mimeo-enhanced agents and baseline agents performed equally well on engineering tasks and application scenarios, hitting a performance ceiling. Furthermore, AI judges showed variability in identifying expert-like responses, cautioning against relying on a single AI judge. The evidence suggests mimeo is valuable for providing inspectable reference material and consistent personas, but not yet for demonstrating a transfer of complex judgment.