UniF-MoE Optimizes Mixture-of-Experts with Adaptive Computation

Mixture-of-Experts (MoE) models are becoming increasingly sophisticated, moving beyond simply routing fixed numbers of experts. Recent advancements include shared-expert designs, fine-grained computation within experts, and dynamic routers that adapt the number of active experts. However, these innovations are often developed in isolation, overlooking the fundamental interplay between shared computation and the remaining expert capacity needed. Researchers propose a unified principle: "share first, then route what remains." This approach is based on decomposing sparsely upcycled feed-forward experts into key-value channels. Observations show that co-activated experts align at certain value positions, and removing these shared positions alters expert preferences, with greater shared coverage leading to lower residual expert demand. This principle is instantiated in UniF-MoE, a framework where each expert is partitioned into aligned blocks. A shared-demand score determines the shared block count and pathway weight, while key prototypes select shared content. The complementary demand then dictates the residual expert count through cumulative routing mass. A Gram regularizer further promotes diverse routing and sparse expert overlap. Experiments on DomainBed and GLUE demonstrate that UniF-MoE improves predictive performance over static and dynamic MoEs while significantly reducing activated computation, inference latency, and memory usage.