Technologies for annotating process and user information for network flows
Systems, methods, and computer-readable media for annotating process and user information for network flows. In some embodiments, a capturing agent, executing on a first device in a network, can monitor a network flow associated with the first device. The first device can be, for example, a virtual machine, a hypervisor, a server, or a network device. Next, the capturing agent can generate a control flow based on the network flow. The control flow may include metadata that describes the network flow. The capturing agent can then determine which process executing on the first device is associated with the network flow and label the control flow with this information. Finally, the capturing agent can transmit the labeled control flow to a second device, such as a collector, in the network.
1. A network traffic monitoring system comprising:
a plurality of distributed sensors, each sensor associated with a particular device of a plurality of physical or virtual devices, wherein:
each sensor generates network flow data based upon packets sent and/or received via a network interface local to the particular device associated with that sensor;
a first device of the plurality of physical or virtual devices is associated with at least one first sensor of the plurality of distributed sensors and comprises a virtual machine;
a second device of the plurality of physical or virtual devices is associated with at least one second sensor and comprises a container; and
a third device of the plurality of physical or virtual devices is associated with at least one third sensor comprises a network switch; and
a backend comprising a collector, an analytics module, and a presentation module, wherein the collector includes a storage, and wherein the presentation module includes one or more application programming interface (API) segments;
wherein the collector is communicably attached to a communications network and receives a plurality of network flow data from the plurality of distributed sensors via the attached communications network,
wherein the analytics module evaluates the plurality of network flow data to establish patterns of a particular behavior of the plurality of physical or virtual devices, and uses a machine learning model to evaluate received information from the plurality of network flow data, and
wherein upon identifying received information that varies from the machine learning model of the particular behavior of the plurality of physical or virtual devices, the system provides, via the presentation module, a report of anomalous flow data.
2. The network traffic monitoring system of claim 1 , wherein the analytics module further evaluates the plurality of network flow data to generate a directed control flow graph corresponding to components of a distributed application.
3. The network traffic monitoring system of claim 2 , wherein the presentation module includes a user interface, and wherein the presentation module creates a visual representation of the directed control flow graph via the user interface.
4. The network traffic monitoring system of claim 2 , wherein the analytics module annotates a node and/or an edge with one of a process ID, a process name, a username, a location, or an environment variable.
5. The network traffic monitoring system of claim 4 , wherein the presentation module includes a user interface, and wherein information about nodes, network flows, the directed control flow graph, and tags are visually represented via the user interface.
6. The network traffic monitoring system of claim 1 , wherein identifying received information that varies from the machine learning model of the particular behavior includes analyzing whether a particular flow from the plurality of network flow information complies with a security policy.
7. The network traffic monitoring system of claim 1 , wherein at least one API segment allows access via a database protocol.
8. The network traffic monitoring system of claim 1 , wherein at least one API segment allows access via a web server.
9. The network traffic monitoring system of claim 1 , wherein at least one API segment allows access for ad-hoc queries.
10. A method of monitoring network traffic comprising:
at a plurality of distributed sensors, each sensor associated with a particular device of a plurality of physical or virtual devices:
generating network flow data based upon packets being sent and/or received via a network interface local to the particular device; and
sending the network flow data via a communications network, wherein:
a first device of the plurality of physical or virtual devices is associated with at least one first sensor of the plurality of distributed sensors and comprises a virtual machine;
a second device of the plurality of physical or virtual devices is associated with at least one second sensor and comprises a container; and
a third device of the plurality of physical or virtual devices is associated with at least one third sensor comprises a network switch;
receiving, at a collector communicably attached to the communications network, a plurality of network flow data, each network flow data coming from a sensor of the plurality of distributed sensors;
storing at least a portion of the plurality of network flow data received by the collector; and
analyzing the plurality of network flow data received by the collector via an associated analytics module, wherein analyzing the plurality of network flow data includes:
establishing patterns of a particular behavior of the plurality of physical or virtual devices;
using a machine learning model to evaluate received information from the plurality of network flow data; and
providing, via a presentation module, a result of the analyzing.
11. The method of claim 10 , wherein analyzing the plurality of network flow data further comprises identifying received anomalous information that varies from the machine learning model of particular behavior, and wherein providing the result further comprises including analysis of the received anomalous information.
12. The method of claim 10 , wherein providing the result of the analyzing includes making the result available via an application programming interface (API).
13. The method of claim 12 , wherein the API is a database access API.
14. The method of claim 12 , wherein making the result available via an API includes allowing ad-hoc queries.
15. The method of claim 12 , wherein the API is provided via a web server.
16. The method of claim 10 , wherein providing the result of the analyzing includes making the result available via a graphical user interface.
17. Non-transitory computer-readable media encoding a set of computer-readable instructions which, when executed on one or more processors on devices connected to a network, cause one or more devices to:
at a plurality of distributed sensors, each sensor associated with a particular device of a plurality of physical or virtual devices:
generate network flow data based upon packets sent and/or received via a network interface local to the particular device; and
send the network flow data via a communications network, wherein:
a first device of the plurality of physical or virtual devices is associated with at least one first sensor of the plurality of distributed sensors and comprises a virtual machine;
a second device of the plurality of physical or virtual devices is associated with at least one second sensor and comprises a container; and
a third device of the plurality of physical or virtual devices is associated with at least one third sensor comprises a network switch;
receive, at a collector communicably attached to the communications network, a plurality of network flow data, each network flow data coming from a sensor of the plurality of distributed sensors;
store at least a portion of the plurality of network flow data received by the collector; and
analyze, via an analytics module, the plurality of network flow data received by the collector, wherein the computer-readable instructions, when executed on the one or more processors, cause the one or more devices to:
establish patterns of a particular behavior of the plurality of physical or virtual devices;
use a machine learning model to evaluate received information from the plurality of network flow data; and
provide, via a presentation module, a result of the analyzing.
18. The non-transitory computer-readable media of claim 17 , further including instructions that, when executed by the one or more processors, cause the one or more devices to make the result available via an application programming interface (API).
19. The non-transitory computer-readable media of claim 17 , further including instructions that, when executed on the one or more processors, cause the one or more devices to make the result available via a graphical user interface.
20. The non-transitory computer-readable media of claim 17 , further including instructions that, when executed on the one or more processors, cause the one or more devices to identify received anomalous information that varies from the machine learning model of the particular behavior, wherein the result of the analyzing includes analysis of the anomalous information.