Technologies for annotating process and user information for network flows
Systems, methods, and computer-readable media for annotating process and user information for network flows. In some embodiments, a capturing agent, executing on a first device in a network, can monitor a network flow associated with the first device. The first device can be, for example, a virtual machine, a hypervisor, a server, or a network device. Next, the capturing agent can generate a control flow based on the network flow. The control flow may include metadata that describes the network flow. The capturing agent can then determine which process executing on the first device is associated with the network flow and label the control flow with this information. Finally, the capturing agent can transmit the labeled control flow to a second device, such as a collector, in the network.
1. A non-transitory computer-readable media encoding a set of non-transitory computer-readable instructions, which when executed on one or more processors on devices connected to a network, cause one or more devices to:
at a first device, receive a stream of network flow data via an attached communications network;
evaluate the stream of network flow data to derive a directed control flow graph corresponding to a distributed application, the control flow graph including a plurality of nodes and a plurality of edges between various nodes, wherein:
the nodes of the graph correspond to network-addressable application components connected to the communications network, each application component sending and receiving network traffic including one or more packets at a network interface;
one or more of the application components includes a workload creating and/or processing a data stream as part of the distributed application; and
one or more first edges of the plurality of edges between the nodes of the graph correspond to data streams between source nodes and destination nodes;
annotate one or more flows associated with one or more nodes plurality of nodes and/or one or more second edges plurality of edges in the control flow graph with one or more tags, the tags relating to a functioning of the distributed application;
identify patterns of normal behavior of the distributed application; and
display a representation of the distributed application.
2. The non-transitory computer-readable media of claim 1 , further comprising computer-readable instructions that cause a second device to generate network flow data based upon the packets being sent and/or received via a network interface local to the second device, and send generated network flow data to the first device.
3. The non-transitory computer-readable media of claim 2 , further comprising computer-readable instructions that, when executed, cause the one or more devices to enable a user to search and query information associated with the directed control flow graph.
4. The non-transitory computer-readable media of claim 2 , wherein the one or more first and/or the one or more second edges in the directed control flow graph correspond to dependencies between services in the distributed application, and wherein the computer-readable media further comprise instructions that display to the user the dependencies between services.
5. The non-transitory computer-readable media of claim 2 , further comprising computer-readable instructions that, when executed, cause the one or more devices to evaluate newly received information from the stream of network flow data against the patterns and identify a change in conditions, the change in conditions including at least one of network traffic that varies from the normal behavior expected by the patterns and a change in the control flow graph; and respond to the change in conditions, wherein responding to the change in conditions includes updating the displayed representation of the distributed application.
6. The non-transitory computer-readable media of claim 2 , wherein the patterns are identified by a machine learning model.
7. The non-transitory computer-readable media of claim 6 , further comprising computer-readable instructions that, when executed, cause the one or more devices to use machine learning techniques to identify patterns that are desirable or unwanted.
8. The non-transitory computer-readable media of claim 7 , further comprising computer-readable instructions that, when executed, cause the one or more devices to enforce and/or modify a network traffic policy that mitigates an unwanted network traffic pattern.
9. The non-transitory computer-readable media of claim 5 , wherein responding to the change in conditions includes sending an alert.
10. The non-transitory computer-readable media of claim 2 , further comprising computer-readable instructions that, when executed, cause the one or more devices to persist at least a portion of information from the stream of network flow data via a storage facility accessible to the first device.
11. The non-transitory computer-readable media of claim 2 , further comprising computer-readable instructions that, when executed, cause the one or more devices to present a web-based user interface to the user.
12. The non-transitory computer-readable media of claim 1 , further comprising computer-readable instructions that, when executed, cause the one or more devices to enable a user to search and query information associated with the control flow graph.
13. The non-transitory computer-readable media of claim 1 , wherein the one or more first and/or the one or more second edges in the directed control flow graph correspond to dependencies between services in the distributed application, and wherein the computer-readable media further comprise instructions that display to the user the dependencies between services.
14. The non-transitory computer-readable media of claim 1 , further comprising computer-readable instructions that, when executed, cause the one or more devices to evaluate newly received information from the stream of network flow data against the patterns and identify a change in conditions, the change in conditions including at least one of network traffic that varies from the normal behavior expected by the patterns and a change in the control flow graph; and respond to the change in conditions, wherein responding to the change in conditions includes updating the displayed representation of the distributed application.
15. The non-transitory computer-readable media of claim 14 , wherein responding to the change in conditions includes sending an alert.
16. The non-transitory computer-readable media of claim 15 , further comprising computer-readable instructions that, when executed, cause the one or more devices to use machine learning techniques to identify patterns that are desirable or unwanted.
17. The non-transitory computer-readable media of claim 16 , further comprising computer-readable instructions that, when executed, cause the one or more devices to enforce and/or modify a network traffic policy that mitigates an unwanted network traffic pattern.
18. The non-transitory computer-readable media of claim 1 , wherein the patterns are identified by a machine learning model.
19. The non-transitory computer-readable media of claim 1 , further comprising computer-readable instructions that, when executed, cause the one or more devices to persist at least a portion of information from the stream of network flow data via a storage facility accessible to the first device.
20. The non-transitory computer-readable media of claim 1 , further comprising computer-readable instructions that, when executed, cause the one or more devices to present a web-based user interface to the user.
21. A method of monitoring network traffic, the method comprising:
at a first device, receiving a stream of network flow data via an attached communications network;
evaluating the stream of network flow data to derive a directed control flow graph corresponding to a distributed application, the control flow graph including a plurality of nodes and a plurality of edges between various nodes, wherein:
the nodes of the graph correspond to network-addressable application components connected to the communications network, each application component sending and receiving network traffic including one or more packets at a network interface;
one or more of the application components includes a container-hosted workload creating and/or processing a data stream as part of the distributed application; and
one or more first edges of the plurality of edges between the nodes of the graph correspond to data streams between source nodes and destination nodes;
annotating one or more flows associated with one or more nodes of the plurality of nodes and/or one or more second edges of the plurality of edges in the control flow graph with one or more tags, the tags relating to a functioning of the distributed application;
identifying patterns of normal behavior of the distributed application; and
displaying a representation of the distributed application.
22. The method of claim 21 , further comprising generating, at a second device, network flow data based upon the packets being sent and/or received via a network interface local to the second device; and sending generated network flow data from the second device to the first device.
23. The method of claim 22 , further comprising receiving a user-provided query and searching information associated with the control flow graph based on the query.
24. The method of claim 22 , wherein the patterns are identified by a machine learning model.
25. The method of claim 24 , further comprising using machine learning techniques to identify traffic patterns that represent normal operation of the distributed application, and traffic patterns which may indicate anomalies in or attacks against the distributed application.
26. The method of claim 21 , further comprising enforcing and/or modifying a network traffic policy to mitigate an anomalous or attacking network traffic pattern.
27. The method of claim 21 , further comprising receiving a user-provided query and searching information associated with the control flow graph based on the query.
28. The method of claim 21 , wherein the patterns are identified by a machine learning model.
29. The method of claim 28 , further comprising using machine learning techniques to identify traffic patterns that represent normal operation of the distributed application, and traffic patterns which may indicate anomalies in or attacks against the distributed application.
30. A system comprising:
one or more processors; and
non-transitory computer-readable media encoding instructions which, when executed by the one or more processors, cause one or more processors to:
receive, at a first device, a stream of network flow data via an attached communications network;
evaluate the stream of network flow data to derive a directed control flow graph corresponding to a distributed application, the control flow graph including a plurality of nodes and a plurality of edges between various nodes, wherein:
the nodes of the graph correspond to network-addressable application components connected to the communications network, each application component sending and receiving network traffic including one or more packets at a network interface;
one or more of the application components includes a workload creating and/or processing a data stream as part of the distributed application; and
one or more first edges of the plurality of edges between the nodes of the graph correspond to data streams between source nodes and destination nodes;
annotate one or more flows associated with one or more nodes of the plurality of nodes and/or one or more second edges of the plurality of edges in the control flow graph with one or more tags, the tags relating to a functioning of the distributed application;
identify patterns of normal behavior of the distributed application; and
display a representation of the distributed application.