Hierarchical service oriented application topology generation for a network
The technology disclosed relates to understanding traffic patterns in a network with a multitude of processes running on numerous hosts. In particular, it relates to using at least one of rule based classifiers and machine learning based classifiers for clustering processes running on numerous hosts into local services and clustering the local services running on multiple hosts into service clusters, using the service clusters to aggregate communications among the processes running on the hosts and generating a graphic of communication patterns among the service clusters with available drill-down into details of communication links. It also relates to using predetermined command parameters to create service rules and machine learning based classifiers that identify host-specific services. In one implementation, user feedback is used to create new service rules or classifiers and/or modify existing service rules or classifiers so as to improve accuracy of the identification of the host-specific services.
1. A method of understanding traffic patterns in a network with a multitude of processes running on numerous hosts, the method including:
clustering processes running on numerous hosts into local services and clustering the local services running on multiple hosts into service clusters;
using the service clusters to aggregate communications among the processes running on the hosts; and
generating a graphic of communication patterns among the service clusters with available drill-down into process constituents of local services in the communication pattern graphic.
2. The method of claim 1 , wherein the clustering processes further includes parsing of application startup data for the numerous hosts.
3. The method of claim 2 , wherein the parsing includes generating tokenized shingles of the application startup data.
4. The method of claim 3 , further including identifying common functionality between the numerous hosts based on similarities between the tokenized shingles.
5. The method of claim 3 , further including using predetermined command parameters to create process rules that identify host-specific processes.
6. The method of claim 5 , wherein the predetermined command parameters include at least one of:
interpreter;
script name;
program name;
service port; and
options.
7. The method of claim 2 , further including applying curated rules to the parsing to ignore one or more processes that are not user-service oriented.
8. The method of claim 1 , wherein the clustering produces subsets of processes grouped into local services, further including determining which groups of processes are co-located on hosts.
9. The method of claim 1 , wherein the clustering produces subsets of processes grouped into local services, further including receiving human feedback on the clustering and updating the grouped processes based on the human feedback.
10. The method of claim 8 further including using a combination of the determined co-located groups and the updated grouped processes for further clustering.
11. The method of claim 3 , further including:
identifying common functionality between the numerous hosts based on grouping of tokenized shingles into one or more clusters;
receiving human feedback on the features for the numerous hosts, wherein the human feedback labels the clusters as belonging to at least one service;
feeding the labeled tokenized shingles from respective clusters as features to create machine-learning based classifiers for one or more service-specific models; and
using the service-specific models for subsequent classification of command parameters into one or more local services.
12. A system of understanding traffic patterns in a network with a multitude of processes running on numerous hosts, the system including:
a processor and a computer readable storage medium storing computer instructions configured to cause the processor to:
cluster processes running on numerous hosts into local services and clustering the local services running on multiple hosts into service clusters;
use the service clusters to aggregate communications among the processes running on the hosts; and
generate a graphic of communication patterns among the service clusters with available drill-down into process constituents of local services in the communication pattern graphic.
13. The system of claim 12 , wherein the clustering processes further includes parsing of application startup data for the numerous hosts.
14. The system of claim 13 , wherein the parsing includes generating tokenized shingles of the application startup data.
15. The system of claim 13 , further configured to identify common functionality between the numerous hosts based on similarities between the tokenized shingles.
16. The system of claim 13 , further configured to use predetermined command parameters to create process rules that identify host-specific processes.
17. The system of claim 16 , wherein the predetermined command parameters include at least one of:
interpreter;
script name;
program name;
service port; and
options.
18. The system of claim 13 , further configured to apply curated rules to the parsing to ignore one or more processes that are not user-service oriented.
19. The system of claim 12 , wherein the clustering produces subsets of processes grouped into local services, further including determining which groups of processes are co-located on hosts.
20. One or more non-transitory computer readable media having instructions stored thereon for performing a method of understanding traffic patterns in a network with a multitude of processes running on numerous hosts, the method including:
clustering processes running on numerous hosts into local services and clustering the local services running on multiple hosts into service clusters;
using the service clusters to aggregate communications among the processes running on the hosts; and
generating a graphic of communication patterns among the service clusters with available drill-down into process constituents of local services in the communication pattern graphic.