D3-Omni Benchmarks Multimodal Judges, Exposing Biases

Multimodal understanding models, often called "OmniJudges," are increasingly used to evaluate and annotate text-to-image, text-to-video, and text-to-speech generation. However, their reliability is questionable because current benchmarks and training data often overemphasize positive examples and conflate different failure modes, masking true capability gaps. To address this, the D3-Omni benchmark was developed. It offers a balanced and decoupled approach to diagnose fine-grained multimodal understanding across 53 distinct binary dimensions and over 10,000 samples. Instead of regenerating outputs, D3-Omni uses verified positive seeds and creates negative examples through controlled prompt rewriting and atomic perturbations, ensuring each error is attributable to a single capability. This "Dual-balanced, Decoupled, and Dynamic" design ensures near 1:1 per-dimension parity and a uniform distribution of total scores. Under this rigorous evaluation, even advanced OmniJudges show weaknesses in modality-related dimensions, are more adept at confirming successes than identifying failures, and tend to treat distinct attributes as a single decision. This suggests that high aggregate accuracy can hide significant, systematic biases that D3-Omni helps expose.