AffectOmni Improves Verifiable Affective Reasoning in MLLMs

This research introduces AffectOmni, a GRPO-trained framework designed to enhance the verifiable affective reasoning capabilities of multimodal large language models (MLLMs). While MLLMs excel at general scene understanding, their affective reasoning often suffers from "shortcut behavior," where models predict correct answers without genuinely attending to crucial people-centric cues like micro-expressions or body language. This lack of focus hinders traceability and external verification. AffectOmni addresses this by incorporating specific reinforcement learning rewards: "People Focus" encourages attention to human evidence, and "Temporal Order" promotes structured, time-sensitive reasoning. To overcome the issue of reward discriminability in LLM-as-a-Judge scoring, the framework employs within-group comparative scoring. For verification, a "Thinking Summarizer" converts free-form rationales into executable evidence instructions, which are then grounded to pixel-level regions using SAM3, providing an auditable interface outside the training loop. Experiments on datasets like IntentBench, Daily Omni, and WorldSense demonstrate consistent performance improvements over open-source 7B scale baselines. Notable gains include a 4.66% increase in emotion recognition and a 14.29% improvement on temporally sensitive tasks, showcasing AffectOmni's effectiveness in fostering more robust and verifiable affective understanding.