IdeaGene-Bench Benchmarks AI for Scientific Lineage Reasoning.

A novel benchmark, IdeaGene-Bench (IG-Bench), has been developed to assess how well AI systems can understand and generate scientific ideas based on their evolutionary lineage. The premise is that scientific concepts, much like biological genomes, inherit mechanisms, address limitations, and recombine elements from prior work, a process largely unaddressed by current AI benchmarks. IG-Bench is structured around the "IdeaGene" framework, which represents each scientific paper or proposal as a collection of "Idea Genome" objects. These objects are minimal, typed, and evidence-grounded. A "GenomeDiff" mechanism then aligns these objects to record six operational evolutionary dynamics: inheritance, mutation, loss, external import, and novel insertion. The benchmark comprises 1,961 golden lineage traces, 1,085 curated Idea Genome objects, and 920 pairwise GenomeDiff records spanning ten scientific domains. The benchmark supports two evaluation types: IG-Exam, which tests closed-form lineage reasoning across 42 task types, and IG-Arena, which evaluates lineage-conditioned idea generation using a Population-Evolution Score (PES). Initial experiments with 14 LLM-based systems revealed a significant compositional bottleneck, with the strongest system achieving only 27.3% exact accuracy on lineage reasoning. This suggests that while structured lineage context can reshuffle system rankings, it doesn't uniformly improve all participants, highlighting a critical area for AI development.