MDL-based clustering for dependency mapping
Systems and methods are provided for automatically discovering applications/clusters in a network and mapping dependencies between the applications/clusters. A network monitoring system can capture network flow data using sensors executing on physical and/or virtual servers of the network and sensors executing on networking devices connected to the servers. The system can determine a graph including nodes, representing at least the servers, and edges, between pairs of the nodes of the graph indicating the network flow data includes one or more observed flows between pairs of the servers represented by the pairs of the nodes. The system can determine a dependency map, including representations of clusters of the servers and representations of dependencies between the clusters, based on the graph. The system can display a first representation of a first cluster of the dependency map and information indicating a confidence level of identifying the first cluster.
1. A method comprising:
capturing, at a collector associated with a network monitoring system, network flow data relating to a plurality of endpoints, wherein each endpoint is a virtual machine, wherein each endpoint of the plurality of endpoints sends and/or receives packets via a virtual network interface dedicated to the endpoint; and
analyzing the network flow data via a machine-learning based analytics engine, the analyzing comprising the steps of:
creating an application dependency graph, wherein nodes in the application dependency graph correspond to a first set of endpoints that function together as a first distributed application, and wherein edges the application dependency graph correspond to network flows described in the network flow data; and
creating at least one cluster of two or more nodes of the first distributed application based upon a similarity of their connections in the application dependency graph,
wherein the collector receives network flow data relating to at least one additional endpoint, wherein the at least one additional endpoint is selected from a group containing a container, a hypervisor, a server, a network device, an external network, or a service, and wherein the network flow data from the at least one additional endpoint is based upon packets being sent and/or received via a network interface dedicated to the at least one additional endpoint.
2. The method of claim 1 , further comprising observing network packets being sent and/or received at each endpoint and sending network flow data describing the observed network flows to the collector.
3. The method of claim 1 , wherein the collector receives network flow information relating to at least one additional endpoint located on a different network, segment, or tenant space.
4. The method of claim 1 , wherein at least one cluster corresponds to a tier in the first distributed application.
5. The method of claim 1 , wherein application dependency graph descriptions of the endpoints within each cluster of the at least one cluster have a similar minimum description length (MDL).
6. The method of claim 1 , further comprising creating a second application dependency graph, wherein nodes in the second application dependency graph correspond to a second set of endpoints that function together as a second distributed application, and wherein edges in the second application dependency graph correspond to network flows described in the network flow data.
7. The method of claim 1 , further comprising presenting a visual representation of the application dependency graph via a user interface.
8. The method of claim 1 , wherein the network flow data is received and processed in substantially real time to map endpoints to applications.
9. The method of claim 8 , wherein changes in the first distributed application are reflected in changes in the application dependency graph.
10. A network traffic monitoring system comprising:
a collector and a machine-learning based analytics engine, wherein the collector receives network flow data relating to a plurality of endpoints, wherein each endpoint of the plurality of endpoints is a virtual machine, wherein each endpoint of the plurality of endpoints sends and/or receives packets via a virtual network interface dedicated to the endpoint, and wherein the machine-learning based analytics engine is configured to:
analyze the network flow data to identify a first set of endpoints functioning together as a first distributed application, and create an application dependency graph, wherein nodes in the application dependency graph correspond to a first set of endpoints associated with the first distributed application and edges in the application dependency graph correspond to network flows described in the network flow data; and
create at least one cluster of two or more nodes of the first distributed application based upon a similarity of their connections in the application dependency graph,
wherein the collector receives network flow data relating to at least one additional endpoint, wherein the at least one additional endpoint is selected from a group containing a container, a hypervisor, a server, a network device, an external network, or a service, and wherein the network flow data from the at least one additional endpoint is based upon packets being sent and/or received via a network interface dedicated to the at least one additional endpoint.
11. The system of claim 10 , wherein the system further comprises the plurality of endpoints.
12. The system of claim 10 , wherein the collector receives network flow information relating to at least one additional endpoint located on a different network, segment, or tenant space.
13. The system of claim 10 , wherein at least one cluster corresponds to a tier in the first distributed application.
14. The system of claim 10 , wherein application dependency graph descriptions of the endpoints within a cluster of the at least one cluster have a similar minimum description length (MDL).
15. The system of claim 10 , wherein a second set of endpoints correspond to a second distributed application.
16. The system of claim 10 , further comprising a presentation module, the presentation module includes a user interface, and wherein the presentation module creates a visual representation of the application dependency graph via the user interface.
17. The system of claim 10 , wherein the traffic data is captured and processed in substantially real time to map endpoints to applications.
18. The system of claim 17 , wherein changes in the first distributed application are reflected in changes in the application dependency graph.