Methods and systems for mining datacenter telemetry data
This disclosure is directed to systems and methods for mining streams of telemetry data in order to identify virtual machines (“VMs”), discover relationships between groups of VMs, and evaluate VM performance problems. The systems and methods transform streams of raw telemetry data consisting of resource usage and VM-related metrics into information that may be used to identify each VM, determine which VMs are similar based on their telemetry data patterns, and determine which VMs are similar based on their patterns of resource consumption. The similarity patterns can be used to group VMs that run the same applications and diagnose and debug VM performance.
1. A data-processing system comprising:
one or more processors;
one or more computer-readable media; and
a routine stored in the computer-readable media that when executed on the one or more processors,
collects streams of raw telemetry data, each stream composed of metric data associated with a virtual machine of a set of virtual machines;
generates data matrices from the streams of telemetry data, each data matrix generated from the raw telemetry data collected in an epoch of time and composed of rows of feature vectors with each feature vector having order statistics that represent one or more features of one of the virtual machines;
identify clusters of virtual machines having corresponding order statistics into clusters based on the data matrices; and
generates a fingerprint for each virtual machine in each cluster based on the feature vector associated with each virtual machine, the fingerprint identifies performance characteristics of each virtual machine.
2. The system of claim 1 , wherein generates the data matrices from the streams of raw telemetry data further comprises:
receives the raw telemetry data for each virtual machine from a virtual machine monitor;
for each epoch of time,
collects samples the raw telemetry data within each epoch;
removes constant valued metrics for each virtual machine;
computes an order statistic for each metric in the raw telemetry data; and
forms a data matrix composed of the order statistics for each virtual machine.
3. The system of claim 2 , wherein the order statistic is one of a percentile, minimum, maximum, sample average, and sample median over the epoch.
4. The system of claim 2 , wherein each row of the data matrix is composed of order statistics associated with one of the virtual machines.
5. The system of claim 2 , further comprise compacts the data matrix to a lower dimensional data matrix.
6. The system of claim 5 , wherein compacts each data matrix to a lower dimensional data matrix further comprises:
computes a mean for each of the order statistics in the data matrix;
computes a mean centered vector for each row of the data matrix based on the mean of the order statistics;
computes a covariance matrix from the means centered vector;
computes eigenvalues and eigenvectors from the covariance matrix; and
computes the lower dimensional data matrix from a product of the data matrix with a subset of eigenvectors that correspond to a user selected number of largest eigenvalues.
7. The system of claim 1 , wherein identifies clusters of virtual machines further comprises applies K-means clustering to each data matrix in order to identify clusters of similar virtual machines.
8. The system of claim 1 , wherein generates the fingerprint for each virtual machine further comprises applies one-versus-all logistic regression to each feature vector in a data matrix to generate a vector of coefficients based on a sigmoid classifier function, each vector of coefficients is a fingerprint of one of the virtual machines and identifies performance characteristics of the virtual machine.
9. A method carried out within a computer system having one or more processors and an electronic memory that mines datacenter telemetry data stored in one or more computer-readable media, the method comprising:
collecting streams of raw telemetry data, each stream composed of metric data associated with a virtual machine of a set of virtual machines;
generating data matrices from the streams of raw telemetry data, each data matrix generated from the raw telemetry data collected in an epoch of time and composed of rows of feature vectors with each feature vector having order statistics that represent features of one of the virtual machines;
identifying clusters of virtual machines having corresponding metric data based on the feature vectors of the data matrices; and
generating a fingerprint for each virtual machine in each cluster based on the feature vector associated with each virtual machine, the fingerprint identifies performance characteristics of each virtual machine.
10. The system of claim 9 , wherein generating the data matrices from the streams of raw telemetry data further comprises:
receiving the raw telemetry data for each virtual machine from a virtual machine monitor;
for each epoch of time,
collecting samples the raw telemetry data within each epoch;
removing constant valued metrics for each virtual machine;
computing an order statistic for each metric in the raw telemetry data; and
forming a data matrix composed of the order statistics for each virtual machine.
11. The method of claim 10 , wherein the order statistic is one of a percentile, minimum, maximum, sample average, and sample median over the epoch.
12. The method of claim 10 , wherein each row of the data matrix is composed of order statistics associated with one of the virtual machines.
13. The method of claim 10 , further comprise compacting the data matrix to a lower dimensional data matrix.
14. The method of claim 13 , wherein compacting each data matrix to a lower dimensional data matrix further comprises:
computing a mean for each of the order statistics in the data matrix;
computing a mean centered vector for each row of the data matrix based on the mean of the order statistics;
computing a covariance matrix from the means centered vector;
computing eigenvalues and eigenvectors from the covariance matrix; and
computing the lower dimensional data matrix from a product of the data matrix with a subset of eigenvectors that correspond to a user selected number of largest eigenvalues.
15. The method of claim 9 , wherein identifying clusters of virtual machines further comprises applying K-means clustering to each data matrix in order to identify clusters of similar virtual machines.
16. The method of claim 9 , wherein generating the fingerprint for each virtual machine further comprises applying one-versus-all logistic regression to each feature vector in a data matrix to generate a vector of coefficients based on a sigmoid classifier function, each vector of coefficients is a fingerprint of one of the virtual machines and identifies performance characteristics of the virtual machine.
17. A non-transitory computer-readable medium encoded with machine-readable instructions that implement a method carried out by one or more processors of a computer system to perform the operations of
collecting streams of raw telemetry data, each stream composed of metric data associated with a virtual machine of a set of virtual machines;
generating data matrices from the streams of raw telemetry data, each data matrix generated from the raw telemetry data collected in an epoch of time and composed of rows of feature vectors with each feature vector having order statistics that represent features of one of the virtual machines;
identifying clusters of virtual machines having corresponding metric data based on the feature vectors of the data matrices; and
generating a fingerprint for each virtual machine in each cluster based on the feature vector associated with each virtual machine, the fingerprint identifies performance characteristics of each virtual machine.
18. The medium of claim 17 , wherein generating the data matrices from the streams of raw telemetry data further comprises:
receiving the raw telemetry data for each virtual machine from a virtual machine monitor;
for each epoch of time,
collecting samples the raw telemetry data within each epoch;
removing constant valued metrics for each virtual machine;
computing an order statistic for each metric in the raw telemetry data; and
forming a data matrix composed of the order statistics for each virtual machine.
19. The medium of claim 18 , wherein the order statistic is one of a percentile, minimum, maximum, sample average, and sample median over the epoch.
20. The medium of claim 18 , wherein each row of the data matrix is composed of order statistics associated with one of the virtual machines.
21. The medium of claim 18 , further comprise compacting the data matrix to a lower dimensional data matrix.
22. The medium of claim 21 , wherein compacting each data matrix to a lower dimensional data matrix further comprises:
computing a mean for each of the order statistics in the data matrix;
computing a mean centered vector for each row of the data matrix based on the mean of the order statistics;
computing a covariance matrix from the means centered vector;
computing eigenvalues and eigenvectors from the covariance matrix; and
computing the lower dimensional data matrix from a product of the data matrix with a subset of eigenvectors that correspond to a user selected number of largest eigenvalues.
23. The medium of claim 17 , wherein identifying clusters of virtual machines further comprises applying K-means clustering to each data matrix in order to identify clusters of similar virtual machines.
24. The medium of claim 17 , wherein generating the fingerprint for each virtual machine further comprises applying one-versus-all logistic regression to each feature vector in a data matrix to generate a vector of coefficients based on a sigmoid classifier function, each vector of coefficients is a fingerprint of one of the virtual machines and identifies performance characteristics of the virtual machine.