Optimizing 3D Generative Diffusion Models on NVIDIA GPUs

3D generative diffusion models, particularly for high-fidelity MRI synthesis, are crucial but demand substantial GPU resources. This is due to hundreds of U-Net evaluations per sample and highly varied kernel behavior. This research conducts an in-depth performance analysis of Med-DDPM, a state-of-the-art medical diffusion model, across three generations of NVIDIA GPU architectures. The analysis delves into kernel-level runtime breakdowns, instruction-mix characteristics, memory system utilization, and warp-level activities. It reveals that training is predominantly driven by cuDNN convolution and implicit-GEMM kernels, with inefficiencies stemming from memory-access patterns, tensor-layout conversions, and underutilization of Tensor Cores. Leveraging these insights, the study evaluates two architecture-aware optimizations: TF32 Tensor Core activation and a 3D channels-last layout. These optimizations are shown to dramatically reduce Streaming Multiprocessor (SM) cycles by up to 100x and dynamic instructions by 100x. Furthermore, Tensor Core utilization increased significantly (from 1.45x to 9.98x), and Instructions Per Cycle (IPC) improved by 7% on A100 GPUs, all without compromising the quality of the synthesized medical images.