Hybrid and hierarchical outlier detection system and method for large scale data protection

1. A method comprising:

receiving metadata associated with one or more data backup jobs performed on one of more storage devices, wherein the metadata comprises univariate time series data for each variable of a multivariate time series, and the multivariate time series comprises different variables that exhibit different characteristics over time; and

decreasing likelihood of a failure in data protection involving the one or more data backup jobs by:

for each variable of the multivariate time series:

selecting, from different anomaly detection models with different performance costs, an anomaly detection model suitable for the variable based on one or more characteristics exhibited by corresponding univariate time series data for the variable and covariations and interactions between the variable and at least one other variable of the multivariate time series; and

detecting an anomaly on the variable utilizing the anomaly detection model selected for the variable; and

based on each anomaly detection model selected for each variable of the multivariate time series,

determining whether the multivariate time series is anomalous at a particular time point, and generating data indicative of whether the multivariate time series is anomalous at the particular time point.

2. The method of claim 1 , further comprising:

for each variable of the multivariate time series:

determining whether corresponding univariate time series data for the variable exhibits seasonality;

in response to determining the corresponding univariate time series data for the variable exhibits seasonality, grouping the variable into a first variable set; and

in response to determining the corresponding univariate time series data for the variable does not exhibit seasonality, grouping the variable into a second variable set.

3. The method of claim 2 , further comprising:

for the first variable set:

applying dynamic factor analysis to identify one or more common trends among the first variable set; and

for each variable of the first variable set, detecting an anomaly on the variable using seasonal-trend decomposition and the common trends.

4. The method of claim 2 , further comprising:

for the second variable set:

applying vector autoregression (VAR) modeling to all the variables;

for each variable of the second variable set, determining a goodness-of-fit value for the variable based on the VAR modeling; and

determining whether VAR fits the multivariate time series based on an average of each goodness-of-fit value determined for each variable of the second variable set.

5. The method of claim 4 , further comprising:

in response to determining VAR fits the multivariate time series, detecting an anomaly on each variable of the second variable set using VAR.

6. The method of claim 4 , further comprising:

in response to determining VAR does not fit the multivariate time series, detecting an anomaly on variables of the second variable set using local outlier factor.

7. A system comprising:

at least one processor; and

a non-transitory processor-readable memory device storing instructions that when executed by the at least one processor causes the at least one processor to perform operations including:

receiving metadata associated with one or more data backup jobs performed on one of more storage devices, wherein the metadata comprises univariate time series data for each variable of a multivariate time series, and the multivariate time series comprises different variables that exhibit different characteristics over time; and

decreasing likelihood of a failure in data protection involving the one or more data backup jobs by:

for each variable of the multivariate time series:

selecting, from different anomaly detection models with different performance costs, an anomaly detection model suitable for the variable based on one or more characteristics exhibited by corresponding univariate time series data for the variable and covariations and interactions between the variable and at least one other variable of the multivariate time series; and

detecting an anomaly on the variable utilizing the anomaly detection model selected for the variable; and

based on each anomaly detection model selected for each variable of the multivariate time series, determining whether the multivariate time series is anomalous at a particular time point, and generating data indicative of whether the multivariate time series is anomalous at the particular time point.

8. The system of claim 7 , the operations further comprising:

for each variable of the multivariate time series:

determining whether corresponding univariate time series data for the variable exhibits seasonality;

in response to determining the corresponding univariate time series data for the variable exhibits seasonality, grouping the variable into a first variable set; and

in response to determining the corresponding univariate time series data for the variable does not exhibit seasonality, grouping the variable into a second variable set.

9. The system of claim 8 , the operations further comprising:

for the first variable set:

applying dynamic factor analysis to identify one or more common trends among the first variable set; and

for each variable of the first variable set, detecting an anomaly on the variable using seasonal-trend decomposition and the common trends.

10. The system of claim 8 , the operations further comprising:

for the second variable set:

applying vector autoregression (VAR) modeling to all the variables;

for each variable of the second variable set, determining a goodness-of-fit value for the variable based on the VAR modeling; and

determining whether VAR fits the multivariate time series based on an average of each goodness-of-fit value determined for each variable of the second variable set.

11. The system of claim 10 , the operations further comprising:

in response to determining VAR fits the multivariate time series, detecting an anomaly on each variable of the second variable set using VAR.

12. The system of claim 10 , the operations further comprising:

in response to determining VAR does not fit the multivariate time series, detecting an anomaly on variables of the second variable set using local outlier factor.

13. A computer program product comprising a computer-readable hardware storage medium having program code embodied therewith, the program code being executable by a computer to implement a method comprising:

receiving metadata associated with one or more data backup jobs performed on one of more storage devices, wherein the metadata comprises univariate time series data for each variable of a multivariate time series, and the multivariate time series comprises different variables that exhibit different characteristics over time; and

decreasing likelihood of a failure in data protection involving the one or more data backup jobs by:

for each variable of the multivariate time series:

selecting, from different anomaly detection models with different performance costs, an anomaly detection model suitable for the variable based on one or more characteristics exhibited by corresponding univariate time series data for the variable and covariations and interactions between the variable and at least one other variable of the multivariate time series; and

detecting an anomaly on the variable utilizing the anomaly detection model selected for the variable; and

based on each anomaly detection model selected for each variable of the multivariate time series, determining whether the multivariate time series is anomalous at a particular time point, and generating data indicative of whether the multivariate time series is anomalous at the particular time point.

14. The computer program product of claim 13 , the operations further comprising:

for each variable of the multivariate time series:

determining whether corresponding univariate time series data for the variable exhibits seasonality;

in response to determining the corresponding univariate time series data for the variable exhibits seasonality, grouping the variable into a first variable set; and

in response to determining the corresponding univariate time series data for the variable does not exhibit seasonality, grouping the variable into a second variable set.

15. The computer program product of claim 14 , the operations further comprising:

for the first variable set:

applying dynamic factor analysis to identify one or more common trends among the first variable set; and

for each variable of the first variable set, detecting an anomaly on the variable using seasonal-trend decomposition and the common trends.

16. The computer program product of claim 14 , the operations further comprising:

for the second variable set:

applying vector autoregression (VAR) modeling to all the variables;

for each variable of the second variable set, determining a goodness-of-fit value for the variable based on the VAR modeling; and

determining whether VAR fits the multivariate time series based on an average of each goodness-of-fit value determined for each variable of the second variable set.

17. The computer program product of claim 16 , the operations further comprising:

in response to determining VAR fits the multivariate time series, detecting an anomaly on each variable of the second variable set using VAR.

18. The computer program product of claim 16 , the operations further comprising:

in response to determining VAR does not fit the multivariate time series, detecting an anomaly on variables of the second variable set using local outlier factor.