Prior-Guided Tuning Enhances LLM Performance with Low-Resource Data

Large Language Models (LLMs) often face significant challenges when trained with limited or ambiguous data, leading to suboptimal performance. While natural-language priors can offer valuable guidance in such scenarios, existing methods typically treat these priors as mere input context rather than integrating them directly into the training process as learning signals.This research introduces Prior-Guided Tuning (PGT), a novel training paradigm that leverages task-level natural-language priors as auxiliary learning signals for LLMs, especially in low-resource settings. Within this perspective, the authors propose Contrastive Prior Steering (CPS). CPS maintains the original supervised objective while adding both positive and negative prior-conditioned auxiliary losses. This encourages the model to learn task-consistent patterns and actively discourages plausible but ultimately misleading alternatives.Experiments conducted on diverse benchmarks like AmbiMath, Jigsaw, and MNLI/HANS demonstrate the consistent superiority of CPS over standard fine-tuning and prompt fine-tuning. For instance, CPS achieved 97.6% accuracy on AmbiMath and improved Macro F1 by 9.5 percentage points on Jigsaw, even with only one-tenth of the training data. On HANS, it boosted non-entailment accuracy by over 5 percentage points for LLaMA 3.1 8B and Qwen 2.5 7B, while preserving in-domain MNLI accuracy. These results strongly support the efficacy of using natural-language priors as integral learning signals for low-resource LLM training.