System and method for survival forecasting of disk drives using semi-parametric transfer learning
Embodiments are directed to a method and system of forecasting a disk drive survival period in a data storage network, by obtaining operating system data and manufacturer data for the disk drive to create a dataset, screening the dataset to identify a number of features to be selected for model creation, wherein the data set includes censored data and non-censored data, and performing, in an analytics engine, semi-parametric survival analysis on the data set using transfer learning on the model to provide a time-based failure prediction of the disk drive. A graphical user interface provides to a user the failure prediction in one of text form or graphical form.
1. A computer-implemented method of forecasting a survival period in a data storage network executing a data backup process transferring data for storage in a plurality of disk drives, comprising:
defining a dataset to be transferred from a source to the disk drives through the data backup process, wherein the disk drives comprise a high capacity disk drive array;
creating a model that combines uncensored data and censored data;
obtaining operating system data and manufacturer data for the disk drives to create the dataset, the operating system data comprising machine-generated data from the data storage network and including operational log data of the data backup process and automatically populated into a database table;
screening the dataset to identify a number of features to be selected for the model creation, wherein the screening step identifies features whose corresponding coefficients are guaranteed to be zero based on a Karush-Kuhn-Tucker (KKT) condition to eliminate unimportant features from the dataset; and
performing, in an analytics engine executing one or more computer-implemented artificial intelligence processes, semi-parametric survival analysis on the data set using transfer learning on the model to provide a time-based failure prediction of the disk drives to ensure proper execution of data transfers of the data backup process.
2. The method of claim 1 further comprising providing to a user the failure prediction through a graphical user interface in one of text form or graphical form.
3. The method of claim 1 wherein the high capacity disk drive array comprises a RAID array on the order of several multi-terabyte capacity disks, and wherein network comprises at least part of a SCSI network, and wherein the disk drive comprises one of a set of hard disk drives (HDD) or solid state drives (SDD) coupled to a backup server over the network.
4. The method of claim 1 wherein the censored data comprises data points for disk drives which are removed from observation before an actual failure, and the uncensored data points are data points that are kept in observation until end-of-life of a disk drive.
5. The method of claim 4 wherein the operating system data comprises certain SCSI sense code providing operating system parameters provided to the analytics engine.
6. The method of claim 4 wherein the manufacturer data comprises SMART (Self-Monitoring, Analysis and Reporting Technology) data of the disk drives provided to the analytics engine.
7. The method of claim 1 further comprising obtaining the operating system data and manufacturer data by:
generating a training dataset in a backup source domain;
generating a training dataset in a backup target domain; and
generating a testing dataset in the backup target domain.
8. The method of claim 7 wherein the dataset comprises, for a plurality of time instances:
a survival time for each feature of the number of features selected for the model, wherein the survival time is the time until the disks were powered on; and
a censor indicator denoting a status of the disk drive with a first binary value indicating a failed state and a second binary value indicating a normal state.
9. The method of claim 8 wherein the semi-parametric survival analysis comprises using a Cox model to estimate a risk of failure of the disk drives based on the selected operating system and SMART parameters.
10. A system for forecasting a survival period in a data storage network, comprising:
a data backup server executing a data backup process defining a dataset to be transferred from a source to a plurality of disk drives provided as a high capacity disk drive array;
a model created by combining uncensored data and censored data;
an interface obtaining operating system data and manufacturer data for the disk drives to create the dataset, the operating system data comprising machine-generated data from the data storage network and including operational log data of the data backup process and automatically populated into a database table;
a screening component screening the dataset to identify a number of features to be selected for model creation, wherein the screening component identifies features whose corresponding coefficients are guaranteed to be zero based on a Karush-Kuhn-Tucker (KKT) condition to eliminate unimportant features from the dataset; and
an analytics engine executing one or more computer-implemented artificial intelligence processes, semi-parametric survival analysis on the data set using transfer learning on the model to provide a time-based failure prediction of the disk drives to ensure proper execution of data transfers of the data backup process.
11. The system of claim 10 further comprising a graphical user interface providing to a user the failure prediction in one of text form or graphical form.
12. The system of claim 10 wherein the high capacity disk drive array comprises a RAID array on the order of several multi-terabyte capacity disks, and wherein network comprises at least part of a SCSI network, and wherein the disk drive comprises one of a set of hard disk drives (HDD) or solid state drives (SDD) coupled to a backup server over the network.
13. The system of claim 10 wherein the censored data comprises data points for disk drives which are removed from observation before an actual failure, and the uncensored data points are data points that are kept in observation until end-of-life of a disk drive.
14. The system of claim 10 wherein the operating system data comprises certain SCSI sense code providing operating system parameters provided to the analytics engine.
15. The system of claim 14 wherein the manufacturer data comprises SMART (Self-Monitoring, Analysis and Reporting Technology) data of the disk drives provided to the analytics engine.
16. The system of claim 10 further comprising a data processing component obtaining the operating system data and manufacturer data by:
generating a training dataset in a backup source domain;
generating a training dataset in a backup target domain; and
generating a testing dataset in the backup target domain.
17. The system of claim 16 wherein the dataset comprises, for a plurality of time instances:
a survival time for each feature of the number of features selected for the model, wherein the survival time is the time until the disk drives were powered on; and
a censor indicator denoting a status of the disk drives with a first binary value indicating a failed state and a second binary value indicating a normal state.
18. The system of claim 17 wherein the semi-parametric survival analysis comprises using a Cox model to estimate a risk of failure of the disk drives based on the selected operating system and SMART parameters.
19. A computer program product, comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method of forecasting a survival period in a data storage network executing a data backup process transferring data for storage in a plurality of disk drives, by:
storage in a plurality of disk drives, comprising:
defining a dataset to be transferred from a source to the disk drives through the data backup process, wherein the disk drives comprise a high capacity disk drive array;
creating a model that combines uncensored data and censored data;
obtaining operating system data and manufacturer data for the disk drives to create the dataset, the operating system data comprising machine-generated data from the data storage network and including operational log data of the data backup process and automatically populated into a database table;
screening the dataset to identify a number of features to be selected for the model creation, wherein the screening step identifies features whose corresponding coefficients are guaranteed to be zero based on a Karush-Kuhn-Tucker (KKT) condition to eliminate unimportant features from the dataset; and
performing, in an analytics engine executing one or more computer-implemented artificial intelligence processes, semi-parametric survival analysis on the data set using transfer learning on the model to provide a time-based failure prediction of the disk drives to ensure proper execution of data transfers of the data backup process.
20. The computer program product of claim 19 further comprising instructions to provide to a user, the failure prediction through a graphical user interface in one of text form or graphical form.