AI Detects HDFS Log Anomalies in Real-Time

The increasing complexity of distributed file systems like HDFS, coupled with the vast amounts of unstructured log data they generate, makes manual anomaly detection a challenging and time-consuming task for system operators. This research addresses this by proposing an automated solution for identifying HDFS block anomalies. The paper introduces a streaming HDFS log block anomaly workflow that leverages parallel computing for historical log processing. At its core is an LLM-BiLSTM hybrid deep learning model designed to detect anomalies. Furthermore, it outlines the construction of a real-time streaming log pipeline based on Kafka, enabling prompt and accurate anomaly detection to ensure system availability and streamline maintenance operations.