Dynamic Parameterization Does Not Equal Dynamic Inference or Savings.

A new research paper challenges the common assumption that models with input-dependent controller coefficients, often termed "dynamic parameterization," automatically achieve dynamic inference or computational efficiencies. The authors argue that these concepts are frequently conflated, distinguishing between mere coefficient variation, the functional dependence of a frozen model on how coefficients are assigned to inputs, and genuine conditional execution. They propose a method called Frozen-Controller Auditing (FCA) to rigorously evaluate these properties. FCA involves caching the complete coefficient tensor along an unperturbed trajectory, then disabling the controller and replaying the frozen model with various perturbations like cross-input reassignment, token shuffling, or static profiles. This approach allows researchers to measure assignment dependence without the confounding factor of recomputing the controller. The study applied FCA to several models, including 76M FeatureGate Transformers and 504M models, finding that static layerwise profiles retained a high percentage of performance, and layer identity explained a significant portion of coefficient variance. Crucially, the research found that models like FeatureGate and MUDDPythia-1.4B, despite exhibiting dynamic parameterization, still execute every Transformer block. This results in them being slower than dense models, with FeatureGate showing a 30.8% slowdown. The findings underscore that dynamic parameterization alone does not guarantee dynamic inference or computational savings, urging a more precise evaluation of claims about dynamic models by separately reporting coefficient variation, functional dependence, and actual execution.