Hierarchical service oriented application topology generation for a network
The technology disclosed relates to understanding traffic patterns in a network with a multitude of processes running on numerous hosts. In particular, it relates to using at least one of rule based classifiers and machine learning based classifiers for clustering processes running on numerous hosts into local services and clustering the local services running on multiple hosts into service clusters, using the service clusters to aggregate communications among the processes running on the hosts and generating a graphic of communication patterns among the service clusters with available drill-down into details of communication links. It also relates to using predetermined command parameters to create service rules and machine learning based classifiers that identify host-specific services. In one implementation, user feedback is used to create new service rules or classifiers and/or modify existing service rules or classifiers so as to improve accuracy of the identification of the host-specific services.
1. A system for generating hierarchical service oriented application topology of a network with a multitude of processes running on numerous hosts, the system comprising:
a machine learning-based classifier trained to cluster the hosts into service profiles by:
evaluating command parameters of respective processes running on the hosts by applying logistic regression to string vectors of the command parameters to calculate a probability of classifying a host into a particular service profile, and
based on the evaluation, classifying hosts that run similar processes as having a same service profile; and
a graphic generator that generates a graphic of the topology of the network based on the service profiles produced by the machine learning-based classifier.
2. The system of claim 1 , wherein the service profiles cluster hosts that share common functionality.
3. The system of claim 1 , wherein the machine learning-based classifier evaluates string vectors of the command parameters against process-specific rules stored in a rule database.
4. The system of claim 1 , wherein the string vectors are chosen based on their respective term frequency-inverse document frequencies (TF-IDFs) prior to being evaluated by the machine learning-based classifier.
5. The system of claim 1 , wherein the machine learning-based classifier applies a logistic regression algorithm that uses cross-validation to classify hosts into service profiles.
6. The system of claim 1 , wherein the machine learning-based classifier is trained using manually labelled training data.
7. The system of claim 1 , wherein the machine learning-based classifier includes a neural network.
8. The system of claim 1 , wherein the machine learning-based classifier applies a naïve bayes algorithm to command parameters.
9. A method of generating hierarchical service oriented application topology of a network with a multitude of processes running on numerous hosts, the method including:
using a trained machine learning-based classifier to cluster the hosts into service profiles by:
evaluating command parameters of respective processes running on the hosts by applying logistic regression to string vectors of the command parameters to calculate a probability of classifying a host into a particular service profile, and
based on the evaluation, classifying hosts that run similar processes as having a same service profile; and
generating a graphic of the topology of the network based on the service profiles produced by the trained machine learning-based classifier.
10. The method of claim 9 , wherein the service profiles cluster hosts that share common functionality.
11. The method of claim 9 , wherein the trained machine learning-based classifier evaluates string vectors of the command parameters against process-specific rules stored in a rule database.
12. The method of claim 9 , wherein the string vectors are chosen based on their respective term frequency-inverse document frequencies (TF-IDFs) prior to being evaluated by the trained machine learning-based classifier.
13. The method of claim 9 , wherein the trained machine learning-based classifier applies a logistic regression algorithm that uses cross-validation to classify hosts into service profiles.
14. The method of claim 9 , wherein the trained machine learning-based classifier is trained using manually labelled training data.
15. The method of claim 9 , wherein the trained machine learning-based classifier includes a neural network.
16. The method of claim 9 , wherein the trained machine learning-based classifier applies a naïve bayes algorithm to command parameters.
17. One or more non-transitory computer readable media having instructions stored thereon for performing a method of generating hierarchical service oriented application topology of a network with a multitude of processes running on numerous hosts, the method including:
using a trained machine learning-based classifier to cluster the hosts into service profiles by:
evaluating command parameters of respective processes running on the hosts by applying logistic regression to string vectors of the command parameters to calculate a probability of classifying a host into a particular service profile, and
based on the evaluation, classifying hosts that run similar processes as having a same service profile; and
generating a graphic of the topology of the network based on the service profiles produced by the trained machine learning-based classifier.
18. The one or more non-transitory computer readable media of claim 17 , wherein the trained machine learning-based classifier includes a neural network.
19. The one or more non-transitory computer readable media of claim 17 , wherein the trained machine learning-based classifier evaluates string vectors of the command parameters against process-specific rules stored in a rule database.
20. The one or more non-transitory computer readable media of claim 17 , wherein the trained machine learning-based classifier applies a logistic regression algorithm that uses cross-validation to classify hosts into service profiles.