Correlating data center resources in a multi-tenant execution environment using machine learning techniques
Methods, apparatus, and processor-readable storage media for correlating data center resources in a multi-tenant execution environment using machine learning techniques are provided herein. An example computer-implemented method includes obtaining multiple data streams pertaining to one or more data center resources in at least one multi-tenant executing environment; correlating one or more portions of the multiple data streams by processing at least a portion of the multiple data streams using at least one multi-tenant-capable search engine; determining one or more anomalies within the multiple data streams by processing the one or more correlated portions of the multiple data streams using a machine learning-based anomaly detection engine; and performing at least one automated action based at least in part on the one or more determined anomalies.
1. A computer-implemented method comprising:
obtaining multiple data streams pertaining to one or more data center resources in at least one multi-tenant executing environment;
correlating one or more portions of the multiple data streams by processing at least a portion of the multiple data streams using at least one multi-tenant-capable search engine comprising at least one database capable of processing at least one of one or more cross-table queries and one or more cross-index queries;
determining one or more anomalies within the multiple data streams by processing the one or more correlated portions of the multiple data streams using a machine learning-based anomaly detection engine; and
performing at least one automated action based at least in part on the one or more determined anomalies, wherein performing at least one automated action comprises automatically training at least a portion of the machine learning-based anomaly detection engine using at least a portion of the one or more anomalies;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.
2. The computer-implemented method of claim 1 , wherein processing the one or more correlated portions of the multiple data streams using the machine learning-based anomaly detection engine comprises processing the one or more correlated portions of the multiple data streams using an artificial neural network.
3. The computer-implemented method of claim 2 , wherein the artificial neural network comprises an autoencoder comprising an input layer containing one or more nodes, an output layer containing one or more nodes, and at least one hidden layer configured to learn at least one feature in the one or more correlated portions of the multiple data streams.
4. The computer-implemented method of claim 3 , wherein using the autoencoder further comprises using a mean squared error as a loss function.
5. The computer-implemented method of claim 3 , wherein using the autoencoder further comprises using an Adam optimization algorithm for stochastic optimization.
6. The computer-implemented method of claim 1 , wherein processing the one or more correlated portions of the multiple data streams using the machine learning-based anomaly detection engine comprises using principal component analysis and at least one direction-sensitive distance classifier algorithm.
7. The computer-implemented method of claim 1 , wherein correlating the one or more portions of the multiple data streams comprises classifying workloads associated with the one or more portions of the multiple data streams based at least in part on tenant activity.
8. The computer-implemented method of claim 7 , wherein correlating further comprises profiling performance of the workloads based at least in part on said classifying.
9. The computer-implemented method of claim 1 , wherein performing the at least one automated action comprises mitigating at least a portion of the one or more determined anomalies.
10. The computer-implemented method of claim 1 , wherein performing the at least one automated action comprises outputting one or more alerts to at least one operations entity.
11. The computer-implemented method of claim 1 , wherein the multiple data streams comprise at least one of one or more data streams pertaining to performance data, one or more data streams pertaining to job allocation data, one or more data streams pertaining to trace log data, and one or more data streams pertaining to workload data.
12. A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device:
to obtain multiple data streams pertaining to one or more data center resources in at least one multi-tenant executing environment;
to correlate one or more portions of the multiple data streams by processing at least a portion of the multiple data streams using at least one multi-tenant-capable search engine comprising at least one database capable of processing at least one of one or more cross-table queries and one or more cross-index queries;
to determine one or more anomalies within the multiple data streams by processing the one or more correlated portions of the multiple data streams using a machine learning-based anomaly detection engine; and
to perform at least one automated action based at least in part on the one or more determined anomalies, wherein performing at least one automated action comprises automatically training at least a portion of the machine learning-based anomaly detection engine using at least a portion of the one or more anomalies.
13. The non-transitory processor-readable storage medium of claim 12 , wherein processing the one or more correlated portions of the multiple data streams using the machine learning-based anomaly detection engine comprises processing the one or more correlated portions of the multiple data streams using an artificial neural network.
14. The non-transitory processor-readable storage medium of claim 13 , wherein the artificial neural network comprises an autoencoder comprising an input layer containing one or more nodes, an output layer containing one or more nodes, and at least one hidden layer configured to learn at least one feature in the one or more correlated portions of the multiple data streams.
15. The non-transitory processor-readable storage medium of claim 12 , wherein correlating the one or more portions of the multiple data streams comprises classifying workloads associated with the one or more portions of the multiple data streams based at least in part on tenant activity.
16. The non-transitory processor-readable storage medium of claim 15 , wherein correlating further comprises profiling performance of the workloads based at least in part on said classifying.
17. An apparatus comprising:
at least one processing device comprising a processor coupled to a memory;
the at least one processing device being configured:
to obtain multiple data streams pertaining to one or more data center resources in at least one multi-tenant executing environment;
to correlate one or more portions of the multiple data streams by processing at least a portion of the multiple data streams using at least one multi-tenant-capable search engine comprising at least one database capable of processing at least one of one or more cross-table queries and one or more cross-index queries;
to determine one or more anomalies within the multiple data streams by processing the one or more correlated portions of the multiple data streams using a machine learning-based anomaly detection engine; and
to perform at least one automated action based at least in part on the one or more determined anomalies, wherein performing at least one automated action comprises automatically training at least a portion of the machine learning-based anomaly detection engine using at least a portion of the one or more anomalies.
18. The apparatus of claim 17 , wherein processing the one or more correlated portions of the multiple data streams using the machine learning-based anomaly detection engine comprises processing the one or more correlated portions of the multiple data streams using an artificial neural network.
19. The apparatus of claim 18 , wherein the artificial neural network comprises an autoencoder comprising an input layer containing one or more nodes, an output layer containing one or more nodes, and at least one hidden layer configured to learn at least one feature in the one or more correlated portions of the multiple data streams.
20. The apparatus of claim 17 , wherein correlating the one or more portions of the multiple data streams comprises classifying workloads associated with the one or more portions of the multiple data streams based at least in part on tenant activity.