New Framework Enables Self-Correcting Concurrent Image Understanding and Generation

Human cognition seamlessly integrates understanding and generation across modalities, a capability largely absent in artificial systems. This research proposes Self-Correcting Coupled Markov Jump Processes (SC-CMJP) to bridge this gap, enabling AI to simultaneously process and create information across text and image. Unlike previous approaches that update modalities independently, SC-CMJP allows one modality's transitions to be influenced by the other's confidence scores, weighted by cross-modal attention. A key innovation is the "remasking jump," which allows the system to retract commitments when cross-modal evidence contradicts them, effectively repairing inconsistencies. In conjunction with SC-CMJP, the authors introduce CO2Jump, a novel training-free, single-pass sampler for joint multimodal generation. To facilitate training and evaluation, three new large-scale multimodal corpora—JEdit-1M, JMaze-200K, and JNono-200K—have been created and will be released. Experiments demonstrate that CO2Jump achieves superior joint performance in tasks like image understanding, editing, and visual reasoning (e.g., maze and nonogram solving). The sampler's performance scales positively with the number of denoising steps, indicating that the benefits of cross-modal coupling compound over time.