Hybrid and hierarchical outlier detection system and method for large scale data protection

1. A method for reducing a number of false positives in anomaly detection for large scale data protection, comprising:

estimating common trends indicative of covariance and interactions among a set of variables of a multivariate time series related to data backup jobs performed on one or more storage devices by applying a dimension reduction technique to the multivariate time series;

applying seasonal-trend decomposition to detect anomalies for each variable of the set of variables based on the common trends; and

preventing a failure in data protection of the data backup jobs by:

for each variable of the multivariate time series, selecting, from different anomaly detection models with different performance costs, an anomaly detection model suitable for the variable based on one or more characteristics exhibited by corresponding univariate time series data for the variable and covariations and interactions between the variable and at least one other variable of the multivariate time series;

performing ensemble learning based on each anomaly detection model selected for each variable of the multivariate time series to determine whether the multivariate time series is anomalous at a particular time point, wherein a variable of the set of variables has at least one vote in the ensemble learning if at least one anomaly is detected for the variable utilizing an anomaly detection model selected for the variable;

generating a report indicative of whether the multivariate time series is anomalous at the particular time point; and

providing the report to a backup monitoring system monitored by a service administrator managing the one or more storage devices and the data backup jobs.

2. The method of claim 1 , wherein each variable of the set of variables exhibits seasonality.

3. The method of claim 1 , wherein applying seasonal-trend decomposition to detect anomalies for each variable of the set of variables based on the common trends comprises applying seasonal-trend decomposition procedure based on loess.

4. The method of claim 1 , wherein applying seasonal-trend decomposition to detect anomalies for each variable of the set of variables based on the common trends comprises applying a combination of seasonal-trend decomposition procedure based on loess and dynamic factor analysis.

5. A system for reducing a number of false positives in anomaly detection for large scale data protection, comprising:

at least one processor; and

a non-transitory processor-readable memory device storing instructions that when executed by the at least one processor causes the at least one processor to perform operations including:

estimating common trends indicative of covariance and interactions among a set of variables of a multivariate time series related to data backup jobs performed on one or more storage devices by applying a dimension reduction technique to the multivariate time series;

applying seasonal-trend decomposition to detect anomalies for each variable of the set of variables based on the common trends; and

preventing a failure in data protection of the data backup jobs by:

for each variable of the multivariate time series, selecting, from different anomaly detection models with different performance costs, an anomaly detection model suitable for the variable based on one or more characteristics exhibited by corresponding univariate time series data for the variable and covariations and interactions between the variable and at least one other variable of the multivariate time series;

performing ensemble learning based on each anomaly detection model selected for each variable of the multivariate time series to determine whether the multivariate time series is anomalous at a particular time point, wherein a variable of the set of variables has at least one vote in the ensemble learning if at least one anomaly is detected for the variable utilizing an anomaly detection model selected for the variable;

generating a report indicative of whether the multivariate time series is anomalous at the particular time point; and

providing the report to a backup monitoring system monitored by a service administrator managing the one or more storage devices and the data backup jobs.

6. The system of claim 5 , wherein each variable of the set of variables exhibits seasonality.

7. The system of claim 5 , wherein applying seasonal-trend decomposition to detect anomalies for each variable of the set of variables based on the common trends comprises applying seasonal-trend decomposition procedure based on loess.

8. The system of claim 5 , wherein applying seasonal-trend decomposition to detect anomalies for each variable of the set of variables based on the common trends comprises applying a combination of seasonal-trend decomposition procedure based on loess and dynamic factor analysis.

9. A computer program product comprising a computer-readable hardware storage medium having program code embodied therewith, the program code being executable by a computer to implement a method comprising:

estimating common trends indicative of covariance and interactions among a set of variables of a multivariate time series related to data backup jobs performed on one or more storage devices by applying a dimension reduction technique to the multivariate time series;

applying seasonal-trend decomposition to detect anomalies for each variable of the set of variables based on the common trends; and

preventing a failure in data protection of the data backup jobs by:

for each variable of the multivariate time series, selecting, from different anomaly detection models with different performance costs, an anomaly detection model suitable for the variable based on one or more characteristics exhibited by corresponding univariate time series data for the variable and covariations and interactions between the variable and at least one other variable of the multivariate time series;

performing ensemble learning based on each anomaly detection model selected for each variable of the multivariate time series to determine whether the multivariate time series is anomalous at a particular time point, wherein a variable of the set of variables has at least one vote in the ensemble learning if at least one anomaly is detected for the variable utilizing an anomaly detection model selected for the variable;

generating a report indicative of whether the multivariate time series is anomalous at the particular time point; and

providing the report to a backup monitoring system monitored by a service administrator managing the one or more storage devices and the data backup jobs.

10. The computer program product of claim 9 , wherein each variable of the set of variables exhibits seasonality.

11. The computer program product of claim 9 , wherein applying seasonal-trend decomposition to detect anomalies for each variable of the set of variables based on the common trends comprises applying seasonal-trend decomposition procedure based on loess.

12. The computer program product of claim 9 , wherein applying seasonal-trend decomposition to detect anomalies for each variable of the set of variables based on the common trends comprises applying a combination of seasonal-trend decomposition procedure based on loess and dynamic factor analysis.