New Divergence Head Improves Asymmetric Representation Learning

Many representation learning tasks involve directed relationships, such as hierarchical structures in ontologies or entailment in language. Traditional symmetric distance metrics (Euclidean, cosine) fail to capture this directionality, while generic neural scorers lack geometric structure. This paper introduces a novel "role-aware neural convex divergence head" designed specifically for asymmetric representation learning. This new head applies distinct source- and target-role projections before calculating an input-convex neural Bregman divergence. This process yields a non-negative, structured score within the role-projected space, effectively modeling directionality. The research characterizes its properties, including projected-space identity and directional-gap decomposition. Experiments across various benchmarks, including lexical, sentence, and ontology tasks, show that the role-aware projections consistently enhance directional accuracy compared to plain ICNN-Bregman heads, while preserving a zero observed negative divergence rate. However, for large fixed-feature citation prediction, specialized symmetric or hyperbolic baselines still show stronger ranking accuracy. Overall, the proposed head serves as an interpretable, structured distance module for tasks where directional relations are paramount.