LivingArena: LLMs Evaluate Each Other to Find Knowledge Gaps.

A novel evaluation framework called LivingArena has been introduced to address the challenges of assessing frontier large language models (LLMs), which often suffer from benchmark saturation and contamination. This framework operates on the principle of "peer-probing," where LLMs actively challenge each other by formulating questions that they believe their opponents cannot answer correctly. In this dynamic setup, questioners are rewarded for successfully identifying and exploiting an opponent's knowledge gaps, while answerers are rewarded for correct responses. To ensure the objectivity of the questions, a panel of powerful LLMs acts as judges, validating the questions and penalizing questioners for invalid prompts. This method generates a stable Elo leaderboard and provides insights into models' abilities to not only recall facts but also to strategically probe and understand the limitations of other AI systems, offering a scalable and cost-effective approach to continuous evaluation.