New Method Improves LLM Steering by Integrating Neighboring Features

Sparse autoencoders (SAEs) are widely used to disentangle and interpret model activations in large language models (LLMs), enabling "steering" or influencing model behavior. Current SAE-based steering methods typically select features using a top-k filter based on statistical scores, assuming that higher-scoring features have a stronger steering effect. However, new research indicates this assumption is often flawed, leading to suboptimal feature selection. Analysis reveals that effective steering features can be distributed within representationally adjacent, semantically similar groups, which arise from feature splitting within SAEs. Within these groups, features may have comparable steering influence despite exhibiting disparate statistical scores. This means that simple score-based selection can inadvertently overlook important features. To address this, researchers propose Neighbor Integrated Feature Selection (NIFS), a novel plug-and-play strategy that improves feature selection by incorporating representation similarity. NIFS has been evaluated across multiple SAE-based steering methods and tasks, consistently demonstrating performance gains over conventional top-k selection. This advancement offers a more nuanced and effective way to identify and utilize features for steering LLMs, potentially leading to more precise and controllable AI behavior.