HN-CLIP Improves Dense-Caption Retrieval with Adaptive Similarity Margins.

Current dense-caption retrieval systems often struggle with the InfoNCE objective, which can saturate quickly due to numerous near-duplicate captions in datasets. This saturation leads to inefficient training and limits performance gains, as the loss function becomes ineffective at distinguishing highly similar negative examples. Researchers have introduced HN-CLIP, a novel approach that tackles this by implementing per-negative adaptive similarity margins. HN-CLIP leverages the text encoder's inherent understanding of text-to-text similarity to dynamically adjust margins for negative samples. By adding a detached caption-similarity matrix to negative logits, the method assigns larger margins to more similar captions without requiring complex negative mining or data synthesis. This streamlined approach avoids auxiliary data, additional parameters, or offline preprocessing. Extensive evaluations on four dense-caption retrieval benchmarks demonstrate HN-CLIP's effectiveness, showing improvements of +2.4 to +4.3 R@1 over leading competitors. Furthermore, it achieves significantly faster training times, being 2.4 times quicker than GOAL and 5.4 times faster than StructXLIP. The objective also enhances all six tested fine-tuning frameworks and can match top full-data baselines with only 20% of the training data.