FADE Method Reduces Hallucinations in Vision-Language Models

Large Vision-Language Models (LVLMs) often generate content that doesn't align with input images, a phenomenon known as hallucination. Recent studies suggest this issue stems from language priors dominating visual inputs. New research delves into the mechanistic origins, finding that transformer attention modules consistently gather visual evidence, but Feed-Forward Network (FFN) modules in critical layers introduce language priors that can override this visual information, leading to incorrect outputs. Based on this insight, a novel, training-free method called FADE (FFN Attenuation for DEcoding) has been introduced. FADE works by attenuating the outputs of the FFN modules, thereby reducing the influence of these language priors. This approach aims to rebalance the model's reliance on visual versus linguistic information during decoding. Evaluations across various benchmarks (POPE, CHAIR, MME) and LVLMs (LLaVA-1.5, mPLUG-Owl2, InstructBLIP) confirm that FADE effectively reduces hallucinations. Crucially, it achieves this without requiring additional training and maintains inference efficiency, offering a practical solution to a significant challenge in multimodal AI.