Detecting LLM Deception: Lie Typology Impacts Performance

Detecting deceptive outputs from large language models (LLMs) remains a complex challenge, with previous research indicating that detection probes often fail when encountering out-of-domain scenarios. This means a probe trained on one type of lie may not effectively identify other forms of deception. This new work conducts a systematic analysis of various factors affecting detection performance. The study examines representation depth, probe expressivity, sparse feature representations, and crucially, the typology of lies within the training data. By augmenting standard benchmarks with a dataset containing diverse deception types—including fabrication, omission, and exaggeration—the researchers found that the choice of training data and the specific lie typology profoundly influence detectability. Optimal representation depth is dataset-dependent, more expressive probes offer only selective advantages, and sparse autoencoder features perform similarly to dense hidden states. Ultimately, the research underscores that effective deception detection is fundamentally tied to the specific data representations used.