New Benchmark Improves LLM Factuality and Reasoning Robustness

Large language models (LLMs) are increasingly tasked with generating responses based on diverse, user-provided knowledge, including structured data like tables and knowledge graphs, alongside unstructured text. A significant challenge is ensuring these models can perform multi-hop reasoning accurately across these heterogeneous data types and remain consistent even when the input order changes. To address this, a new benchmark called TKFQA has been developed. It features over 10,000 question-answering pairs, each constructed with explicit counterfactual reasoning chains. This allows for a comprehensive evaluation of an LLM's ability to provide correct answers, trace accurate reasoning paths, and maintain robustness against variations in how the input knowledge is presented. Initial evaluations of 14 state-of-the-art LLMs revealed that they often struggle with reasoning chain accuracy and are sensitive to input order. In response to these findings, the researchers propose ORLF (Order-Robust Latent Fusion), an LLM-agnostic training framework. ORLF models the topological relationships between different knowledge contexts using knowledge-specific latent vectors. By integrating context-wise position encoding, a latent-bridge attention mask, and topological knowledge bias, ORLF helps LLMs better preserve knowledge-specific information and encode semantic relationships. Experiments show ORLF significantly improves exact match and reasoning-chain accuracy while reducing performance variance due to input order.