SemHash-LLM Boosts Document Deduplication with Multi-Granularity Hashing

Large-scale document deduplication is crucial for managing vast text corpora, but it requires methods that are both semantically accurate and computationally efficient. A new framework, SemHash-LLM, addresses this challenge by employing a multi-granularity approach. It integrates several techniques, including semantic projection hashing, which learns compact binary codes from distilled LLM embeddings, and attention-weighted MinHash, designed to prioritize informative content while suppressing boilerplate. The system also incorporates contrastive boundary learning and selective LLM-based adjudication. By combining signals from character, token, and document levels through a gated fusion mechanism, SemHash-LLM applies a cascaded filtering pipeline to reduce candidate sets efficiently. This robust method is designed to handle various complexities like template pollution, short text perturbations, and viral fragments, significantly improving duplicate detection quality with minimal neural verification costs.