Lightweight Transformers Benchmarked for On-Device Fault Detection.

Researchers conducted a benchmark study comparing lightweight transformer architectures with traditional machine learning methods for on-device fault detection. The goal was to assess their suitability for deployment on resource-constrained hardware, considering trade-offs between accuracy, latency, and model size. The evaluation covered models like DistilBERT, TinyBERT, MobileBERT, and traditional algorithms such as Random Forest and XGBoost, across three public datasets: NASA C-MAPSS, SECOM, and UCI AI4I 2020. Results showed that on well-separated sensor data (C-MAPSS), lightweight transformers could match traditional ML in F1-score (87.8%) but at a cost of 100x larger model size and 9000x higher latency. TinyBERT-4L emerged as the most deployment-friendly transformer, balancing size (55 MB) and CPU latency (18 ms). INT8 dynamic quantization further reduced model size by 25% while largely preserving accuracy. The study also proposed a two-stage adaptive inference pipeline, which routed 97.9% of predictions through a quantized triage model and only 2.1% to a larger expert model. This achieved comparable F1-score (87.6%) with an average latency of 19.5 ms. However, both traditional and transformer methods struggled significantly on severely imbalanced datasets (SECOM, UCI-PM), highlighting a fundamental limitation for extreme class imbalance in fault detection.