Scalable predictive early warning system for data backup event log
Techniques to detect backup-related anomalies are disclosed. In various embodiments, a processor is used to generate based at least in part on backup log data associated with a training period a predictive model. The predictive model is to detect, using the processor, anomalies in corresponding backup log data associated with a detection period.
1. A method of detecting backup related anomalies, comprising:
filtering backup log data associated with one or more backup clients during a training period into one or more sets of backup log data based on one or more attributes;
extracting from a set of the one or more sets of backup log data a prescribed set of features;
generating, using a processor, based at least in part on the extracted prescribed set of features, a predictive model;
using, by the processor, the predictive model to detect anomalies in backup log data associated with the one or more backup clients during a detection period, wherein the anomalies at least includes data being erroneously deleted;
computing a score for a detected anomaly; and
performing one or more responsive actions based at least in part on a comparison between the computed score and a detection threshold, wherein a responsive action includes a processor using backup data stored at a prior time to restore the erroneously deleted data.
2. The method of claim 1 , further comprising receiving via a network communication interface said backup log data associated with the training period.
3. The method of claim 2 , further comprising receiving via the network communication interface said backup log data associated with the one or more backup clients during the detection period.
4. The method of claim 1 , wherein said one or more attributes include one or more of the following: backup type; backup schedule; backup size; number of objects backed up; source system type; and other source system attribute.
5. The method of claim 1 , wherein the prescribed set of features include one or more of the following: backup size; number of objects backed up; and amount of change in backup data size.
6. The method of claim 1 , wherein the predictive model is associated with one or more of the following model types Gaussian hypothesis testing; KS test; and Kernel Density Estimation.
7. The method of claim 1 , wherein the predictive model is configured to be used to predict for a given set of extracted features associated with a backup performed during the detection period a corresponding statistical probability of occurrence of said given set of features.
8. The method of claim 1 , further comprising ranking detected anomalies based at least in part on their respective scores.
9. The method of claim 1 , wherein the one or more responsive actions are performed based at least in part on a determination that the computed score exceeds a detection threshold.
10. A system to detect backup related anomalies, comprising:
a communication interface; and
a processor coupled to the communication interface and configured to:
receive via the communication interface backup log data associated with one or more backup clients during a training period;
filter the backup log data associated with the one or more backup clients during the training period into one or more sets of backup log data based on one or more attributes;
extracting from a set of the one or more sets of backup log data a prescribed set of features;
generate a predictive model based at least in part on the extracted prescribed set of features;
use the predictive model to detect anomalies in backup log data associated with the one or more clients during a detection period, wherein the anomalies at least includes data being erroneously deleted;
compute a score for a detected anomaly; and
perform one or more responsive actions based at least in part on a comparison between the computed score and a detection threshold, wherein a responsive action includes a processor using backup data stored at a prior time to restore the erroneously deleted data.
11. The system of claim 10 , wherein said one or more attributes include one or more of the following: backup type; backup schedule; backup size; number of objects backed up; source system type; and other source system attribute.
12. The system of claim 10 , wherein the prescribed set of features include one or more of the following: backup size; number of objects backed up; and amount of change in backup data size.
13. A computer program product to detect backup related anomalies, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
filtering backup log data associated with one or more backup clients during a training period into one or more sets of backup log data based on one or more attributes;
extracting from a set of the one or more sets of backup log data a prescribed set of features;
generating, using a processor, based at least in part on the extracted prescribed set of features, a predictive model;
using, by the processor, the predictive model to detect anomalies in backup log data associated with the one or more backup clients during a detection period,
wherein the anomalies at least includes data being erroneously deleted;
computing a score for a detected anomaly; and
performing one or more responsive actions based at least in part on a comparison between the computed score and a detection threshold, wherein a responsive action includes a processor using backup data stored at a prior time to restore the erroneously deleted data.