HERO Library Benchmarks Federated Continual Learning for Diverse Data.

Federated continual learning (FCL) involves distributed clients learning from evolving data streams while retaining past knowledge. A major challenge in this field has been the lack of standardized evaluation, making it difficult to compare different FCL methods due to simultaneous variations in datasets, task splits, client data distributions, and task orders. To address this, a new benchmark library called HERO has been developed. HERO allows researchers to systematically control and isolate three critical factors: task split, client data split, and client task sequence. This enables more rigorous and comparable evaluations of FCL algorithms. Initial evaluations using HERO on datasets like CIFAR-100 and TinyImageNet reveal that method performance can vary significantly across different settings, and average accuracy metrics can mask poor performance among individual clients. The benchmark also highlights how task-order mismatch influences strategy effectiveness and provides a framework for exploring domain-shift challenges beyond image data.