Domain-Conditional Offsets Improve LLM Cold Start Accuracy

Autoregressive language models often struggle with accuracy at the very beginning of a sequence, a phenomenon known as the "cold-start penalty." This occurs because little context is available, forcing the model to rely heavily on a generic pretraining prior, which may not be optimal for a specific domain. This initial inaccuracy can impact tasks requiring early in-domain signal. Researchers have developed a novel, lightweight solution: domain-conditional position offsets. This method involves adding a single learned vector to the embedding activation at the first few sequence positions, while keeping all other model weights frozen. This offset can be trained rapidly, often in minutes, using only about a hundred documents, and can be switched between domains without adding sequence state or measurable latency overhead. Across various Mamba, GPT-NeoX, and Llama models ranging from 410M to 8B parameters, this technique reduced held-out in-domain perplexity by up to 27%, with the effect persisting even at 70B parameters. The benefit is largely captured by just one position. Unlike other adapters like LoRA or soft prompts, which add many parameters or sequence positions, this offset is a hot-switchable tool ideal for short in-domain scoring and calibration, improving tasks like retrieval reranking and domain classification when early tokens are crucial.