CoIn Enhances 2D-3D Inpainting with Gaussian Splatting

Reconstructing corrupted or occluded areas in 3D scenes, known as 3D inpainting, is a critical task. While recent methods using Gaussian Splatting (GS) offer efficient 3D editing, they are often limited to object removal and require precise segmentation masks. Researchers have introduced CoIn, a new framework that overcomes these limitations by creating a bidirectional information flow between 2D inpainting models and 3DGS. CoIn begins by using a diffusion model to generate initial inpainted images, allowing for flexible mask shapes and diverse tasks like object insertion, not just removal. The framework then reconstructs a coarse 3D scene using Reference Adaptive GS with Feature Attention, which prioritizes a reference 2D view. This 3D representation subsequently guides the 2D diffusion process via GS-based Reference Feature Warping, ensuring consistency across multiple views. Finally, a Texture-Enhancing Discriminator refines the 3D scene for high photometric realism. Experiments show CoIn achieves state-of-the-art performance in both object removal and insertion.