Ensuring the health and reliability of HDDs is paramount for large-scale operations. We at ULINK have used ML models and developed a successful disk failure prediction tool called ULINK DA Drive Analyzer. We recently explored the recent model developed by Meta and now examine the ML system developed a couple of years ago by Google Cloud, in collaboration with its HDD OEM partner Seagate, to predict and preemptively address recurring HDD issues.
Google Cloud operates some of the largest data centers globally, where any lapses in HDD management can lead to severe outages across various services. Traditionally, identifying and addressing HDD failures involved time-consuming and costly processes. This ML system represents a significant leap forward in HDD management that can enhance reliability, reduce costs, and mitigate risks associated with data center operations.
Managing millions of HDDs generates vast amounts of telemetry data, including SMART data and host metadata. Manual monitoring of these parameters is unfeasible, and therefore makes a great case for the development and adoption of automated ML-driven solutions. Google Cloud’s AI Services team, alongside Accenture, assisted Seagate in building a proof of concept utilizing Terraform for infrastructure management, BigQuery for data processing, and AutoML Tables for ML model development.
The predictive maintenance system processes data from various sources, including SMART indicators, host notifications, HDD logs, and manufacturing data. To be successful, the data pipeline needed to be both scalable and reliable for both batch and streaming data processes. Through advanced analytics and feature engineering, the system extracts meaningful insights to forecast HDD health accurately.
Two ML approaches were explored: an AutoML Tables classifier and a custom Transformer-based model. Despite the complexity of HDD data, the AutoML model demonstrated superior performance, achieving a precision of 98% and a recall of 35%. This model’s deployment, coupled with robust MLOps practices, streamlined the entire lifecycle from data ingestion to model deployment.
Google Cloud’s MLOps environment, encompassing Terraform, GitLab, and Cloud Composer, facilitated seamless orchestration and automation, enabling efficient model deployment and retraining. The success of this project underscores the transformative potential of ML-driven predictive maintenance in enhancing HDD management.
Similar to the Google model, the ULINK model also focuses on achieving a high precision and low false positives. However, it is important to note that the unique feature of DA Drive Analyzer’s ML system compared to the Google model lies in its data collection process. Instead of solely relying on HDD data from a single manufacturer, we aggregate data from multiple HDD manufacturers, such as Seagate, WDC, Toshiba, and HGST. This approach has enabled us to develop a comprehensive drive failure prediction system that is highly relevant and applicable to today’s diverse consumer market.
QNAP Launches the AI-Powered DA Drive Analyzer 2.0 – Predicts NAS Drive Failure Within 24 Hours & Enhances Enterprise Privacy
Photo Credit: Vladimir_Timofeev
Recent Comments