Understanding LLM Knowledge-Using Gap in Finetuning

When Large Language Models (LLMs) are fine-tuned with new information, they often quickly memorize the facts but struggle to apply this knowledge to downstream reasoning tasks. This phenomenon, termed the "Knowing-Using Gap," is characterized by a delay and an accuracy difference between when knowledge is memorized and when it can be generalized. To understand this gap, researchers fine-tuned LLMs with novel knowledge and monitored how this knowledge permeated internally using a technique called self-patching. This method identified specific activation locations where modifying representations significantly improved cases of generalization failure. The findings support a "knowledge-circuit misalignment hypothesis," suggesting that while memorized representations exist within the model, they might not be correctly routed to the computational layers that can effectively utilize them for reasoning. A simple heuristic strategy based on this diagnostic finding was shown to recover a substantial portion of the potential improvement in generalization failures, demonstrating the practical implications of this mechanistic understanding.