Storage anomaly detection
The technology described in this document is, among other things, capable of efficiently monitoring storage device signal data for anomalies. In an example method, signal data for a plurality of non-transitory storage devices is collected. The method determines a hyper feature representation from the collected signal data and computes, using the hyper feature representation, scores for statistics associated with the non-transitory storage devices. The method further determines a reduced hyper feature representation aggregating the scores for each of the statistics associated with each of the non-transitory storage devices; generates, using the reduced hyper feature representation, storage device scores for the non-transitory storage devices of the plurality, respectively; and identifies one or more non-transitory storage devices from among the plurality of non-transitory storage devices exhibiting anomalous storage device behavior using the storage device scores.
1. A computer-implemented method comprising:
collecting, at one or more computing devices, signal data for a plurality of non-transitory storage devices, the signal data including time series for statistics associated with each of the non-transitory storage devices;
determining, using the one or more computing devices, a hyper feature representation from the collected signal data;
computing, using the one or more computing devices and the hyper feature representation, scores for each of the statistics associated with each of the non-transitory storage devices;
determining, using the one or more computing devices, a reduced hyper feature representation aggregating the scores for each of the statistics associated with each of the non-transitory storage devices;
generating, using the one or more computing devices and the reduced hyper feature representation, storage device scores for the non-transitory storage devices of the plurality, respectively; and
identifying, using the one or more computing devices, one or more non-transitory storage devices from among the plurality of non-transitory storage devices exhibiting anomalous storage device behavior using the storage device scores.
2. A computer-implemented method comprising:
determining, using one or more computing devices, a hyper feature representation aggregating scores for statistics associated with a plurality of non-transitory storage devices, wherein the hyper feature representation is a reduced hyper feature representation based on an initial hyper feature representation;
generating, using the one or more computing devices and the hyper feature representation, storage device scores for the non-transitory storage devices of the plurality of non-transitory storage devices, respectively; and
identifying, using the one or more computing devices, one or more non-transitory storage devices from among the plurality of non-transitory storage devices exhibiting anomalous storage device behavior using the storage device scores.
3. A system comprising:
one or more processors;
one or more memories; and
an analyzer configured to perform operations including:
determining a hyper feature representation aggregating scores for statistics associated with a plurality of non-transitory storage devices, wherein the hyper feature representation is a reduced hyper feature representation based on an initial hyper feature representation;
generating, using the hyper feature representation, storage device scores for the non-transitory storage devices of the plurality of non-transitory storage devices, respectively; and
identifying one or more non-transitory storage devices from among the plurality of non-transitory storage devices exhibiting anomalous storage device behavior using the storage device scores.
4. The computer-implemented method of claim 2 , further comprising:
ranking the non-transitory storage devices using the storage device scores.
5. The computer-implemented method of claim 2 , further comprising:
generating analytics identifying regions of interest for each of the non-transitory storage devices; and
providing the analytics for presentation by a client device of a stakeholder.
6. The computer-implemented method of claim 2 , further comprising:
computing, using the one or more computing devices and the initial hyper feature representation, the scores for the statistics associated with the plurality of non-transitory storage devices, wherein:
the initial hyper feature representation is a two-dimensional matrix including a time series for the statistics associated with the non-transitory storage devices.
7. The computer-implemented method of claim 2 , wherein the storage device scores are feature vectors having dimensions equal to a number of statistics used to determine the scores.
8. The computer-implemented method of claim 7 , wherein each feature vector of the feature vectors includes a ranking of the statistics corresponding to that feature vector.
9. The computer-implemented method of claim 2 , further comprising:
computing the scores for the statistics associated with the plurality of non-transitory storage devices, wherein computing the scores for the statistics includes performing hierarchical clustering to determine statistical significance of statistical values of the statistics.
10. The computer-implemented method of claim 2 , further comprising:
computing the scores for the statistics associated with the plurality of non-transitory storage devices, wherein computing the scores for the statistics includes fitting a multivariate t distribution to the scores.
11. The computer-implemented method of claim 2 , further comprising:
computing, using the one or more computing devices and the initial hyper feature representation, the scores for the statistics associated with the plurality of non-transitory storage devices, wherein the initial hyper feature representation is determined using principal component analysis while preserving a time series of the statistics associated the plurality of non-transitory storage devices.
12. The computer-implemented method of claim 2 , wherein the hyper feature representation is determined using principal component analysis.
13. A system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
collecting signal data for a plurality of non-transitory storage devices, the signal data including time series for statistics associated with each of the non-transitory storage devices;
determining a hyper feature representation from the collected signal data;
computing, using the hyper feature representation, scores for each of the statistics associated with each of the non-transitory storage devices;
determining a reduced hyper feature representation aggregating the scores for each of the statistics associated with each of the non-transitory storage devices;
generating, using the reduced hyper feature representation, storage device scores for the non-transitory storage devices of the plurality, respectively; and
identifying one or more non-transitory storage devices from among the plurality of non-transitory storage devices exhibiting anomalous storage device behavior using the storage device scores.
14. The system of claim 3 , wherein the analyzer is further configured to perform operations comprising:
ranking the non-transitory storage devices using the storage device scores.
15. The system of claim 3 , wherein the analyzer is further configured to perform operations comprising:
generating analytics identifying regions of interest for each of the non-transitory storage devices; and
providing the analytics for presentation by a client device of a stakeholder.
16. The system of claim 3 , wherein the analyzer is further configured to perform operations comprising:
computing, using the initial hyper feature representation, the scores for the statistics associated with the plurality of non-transitory storage devices, wherein the initial hyper feature representation is a two-dimensional matrix including a time series for the statistics associated with the non-transitory storage devices.
17. The system of claim 3 , wherein the storage device scores are feature vectors having dimensions equal to a number of statistics used to determine the scores.
18. The system of claim 17 , wherein each feature vector of the feature vectors includes a ranking of the statistics corresponding to that feature vector.
19. The system of claim 3 , wherein the analyzer is further configured to perform operations comprising:
computing the scores for the statistics associated with the plurality of non-transitory storage devices, wherein computing the scores for the statistics includes performing hierarchical clustering to determine statistical significance of statistical values of the statistics.
20. The system of claim 3 , wherein the analyzer is further configured to perform operations comprising:
computing the scores for the statistics associated with the plurality of non-transitory storage devices, wherein computing the scores for the statistics includes fitting a multivariate t distribution to the scores.
21. The system of claim 3 , wherein the analyzer is further configured to perform operations comprising:
computing, using the initial hyper feature representation, the scores for the statistics associated with the plurality of non-transitory storage devices, wherein the initial hyper feature representation is determined using principal component analysis while preserving a time series of the statistics associated the plurality of non-transitory storage devices.
22. The system of claim 3 , wherein the hyper feature representation is determined using principal component analysis.
23. A system comprising:
one or more processors;
one or more memories;
means for collecting signal data for a plurality of non-transitory storage devices, the signal data including time series for statistics associated with each of the non-transitory storage devices;
means for determining a hyper feature representation from the collected signal data;
means for computing, using the hyper feature representation, scores for each of the statistics associated with each of the non-transitory storage devices;
means for determining a reduced hyper feature representation aggregating the scores for each of the statistics associated with each of the non-transitory storage devices;
means for generating, using the reduced hyper feature representation, storage device scores for the non-transitory storage devices of the plurality, respectively; and
means for identifying one or more non-transitory storage devices from among the plurality of non-transitory storage devices exhibiting anomalous storage device behavior using the storage device scores.