NL-PAC Quantifies Ambiguity in LLM-Mediated Supervision

Large language models are increasingly used to provide labels, evaluations, and feedback based on natural language specifications. A critical challenge arises when these specifications are ambiguous, allowing for multiple interpretations. In such cases, simply adding more labels does not resolve the underlying ambiguity, leading to an identification problem that limits the effectiveness of supervision. Researchers have introduced Natural Language PAC (NL-PAC), a framework designed to quantify this specification ambiguity and establish certified minimax risk floors for learners. NL-PAC uses a fixed model's thresholded decoding law to define admissible labels and candidate targets. The probability that multiple labels are admissible directly corresponds to the "diameter" of the pointwise-admissible target class. Under target-blind supervision, any learner will incur a worst-case risk of at least half this diameter, regardless of sample size. The framework provides finite-sample confidence bounds, allowing these quantities to be certifiable from held-out unlabeled inputs. Audits using a frozen Qwen 2.5-3B model demonstrated that specific prompts could yield positive certificates of ambiguity, while paraphrases or exact-rule controls did not. This research underscores that the guarantee is specific to the audited model, prompt, threshold, and input distribution, requiring external validation for human interpretations.