Systems and methods for network data flow aggregation
Embodiments of the present disclosure can aggregate network flow data over various periods of time, and present a graphical representation of the network flow information based on the aggregated data instead of (or in addition to) the raw network flow data. Among other things, embodiments of the present disclosure are able to present historical network flow data for relatively long periods of time without having to store large volumes of raw network flow data.
1. A computer-implemented method comprising:
collecting, by a computer system, network flow data from a plurality of different types of sources on a network;
identifying, by the computer system based on the collected network flow data, a plurality of network assets and connections between the plurality of network assets;
generating, by the computer system, a first aggregation of the collected network flow data that includes a subset of the collected network flow data over a first predetermined period of time;
generating, by the computer system, a second aggregation of the collected network flow data that includes a subset of the collected network flow data over a second predetermined period of time;
storing, by the computer system, the aggregations in a first database during at least one of the first predetermined period of time or the second predetermined period of time;
loading, by the computer system, the aggregations from the first database to a second database after the first and second predetermined periods of time; and
presenting, by the computer system based on the aggregations of the collected network flow data, a graphical representation that includes the plurality of network assets and the connections between the plurality of network assets,
wherein the graphical representation is presented via a display of a user interface in communication with the computer system,
wherein the plurality of network assets includes a host network asset and a client network asset,
wherein presenting the graphical representation includes presenting a flow information graph that depicts the host network asset, the client network asset, connections between the host network asset and the client network asset, and services that are permitted to be accessed between the host network asset and the client network asset,
wherein the flow information graph visually indicates an attribute of the host network asset, an attribute of the client network asset, and attributes of a connection between the host network asset and the client network asset, and
wherein the attributes of the connection between the host network asset and the client network asset include sizes of data communicated, times when the data is communicated, and rates at which the data is communicated, detected vulnerable service communications, and detected attack signatures;
analyzing the aggregations of the collected network flow data based on a security policy, wherein the security policy is a collection of security technical control configuration policies that specify a specific operation of a security technical control;
dynamically assigning the security policy to the host network asset or the client network asset based on a trust zone to which the host network asset or the client network asset is assigned; and
generating an alert in response to the analysis of the aggregations of the collected network flow data.
2. The method of claim 1 , wherein each of the aggregations includes a unique identifier associated with a network asset from the plurality of network assets.
3. The method of claim 2 , wherein each of the aggregations includes a unique identifier associated with a source asset from the plurality of network assets from which network traffic originates and a unique identifier associated with a destination asset from the plurality of network assets to which network traffic is directed.
4. The method of claim 1 , further comprising:
receiving a selection, via the user interface, of a flow policy associated with a particular type of network flow, wherein the flow policy is one of:
a policy to track network flow based on at least one of the aggregations; and
a policy to track network flow based on both the at least one of the aggregations and the collected network flow data in its entirety; and
generating the aggregations of the collected network flow data based on the received flow policy.
5. The method of claim 4 , further comprising receiving a selection, via the user interface, to configure at least one of the predetermined time periods.
6. The method of claim 5 , wherein the second period of time is longer than the first period of time and the second aggregation of collected network flow data includes a subset of the first aggregation of collected network flow data.
7. The method of claim 6 , wherein generating the second aggregation of collected network flow data includes updating one or more of the following in the second aggregation of network flows based on the first aggregation of collected network flow data: a count of network flows between assets on the network, and a number of bytes transmitted between assets on the network.
8. The method of claim 6 , wherein generating the second aggregation of collected network flow data includes a subset of collected network flow data from each of a plurality of aggregations of collected network flow data for a respective plurality of time periods, and wherein each of the respective plurality of time periods is shorter than the second period of time.
9. The method of claim 8 , wherein each of the plurality of time periods is of equal length.
10. The method of claim 8 , wherein at least two of the plurality of time periods are of different length.
11. The method of claim 1 , wherein:
the flow lines indicate a type of the source of the collected network flow data used to identify the connection, and/or the flow information graph depicts connections between the host network asset and the client network asset using selectable directional flow lines.
12. The method of claim 1 , wherein presenting the flow information graph includes:
identifying a security vulnerability associated with one or more of the host network asset and the client network asset; and
visually indicating the identified security vulnerability in the flow information graph.
13. The method of claim 1 , wherein the collected network flow data is obtained from a source selected from the group consisting of: a network tap, a router, a switch, a firewall, an intrusion detection system, an intrusion protection system, and combinations thereof.
14. The method of claim 1 , further comprising:
analyzing the aggregations of the collected network flow data by comparing the aggregations of the collected network flow data to one or more of: the security policy, a standards compliance policy, and a performance standard; and
generating an alert in response to the analysis of the aggregations of the collected network flow data.
15. A tangible, non-transitory computer-readable medium storing instructions that, when executed, cause a computer system to:
collect network flow data from a plurality of different types of sources on a network;
identify, based on the collected network flow data, a plurality of network assets and connections between the plurality of network assets;
generate a first aggregation of the collected network flow data that includes a subset of the collected network flow data over a first predetermined period of time;
generate a second aggregation of the collected network flow data that includes a subset of the collected network flow data over a second predetermined period of time;
store the aggregations in a first database during at least one of the first predetermined period of time or the second predetermined period of time;
load the aggregations from the first database to a second database after the first and second predetermined periods of time; and
present, based on the aggregations of the collected network flow data, a graphical representation that includes the plurality of network assets and the connections between the plurality of network assets, wherein the graphical representation is presented via a display of a user interface in communication with the computer system;
wherein the plurality of network assets includes a host network asset and a client network asset,
wherein presenting the graphical representation includes presenting a flow information graph that depicts the host network asset, the client network asset, connections between the host network asset and the client network asset, and services that are permitted to be accessed between the host network asset and the client network asset,
wherein the flow information graph visually indicates an attribute of the host network asset, an attribute of the client network asset, and attributes of a connection between the host network asset and the client network asset, and
wherein the attributes of the connection between the host network asset and the client network asset include sizes of data communicated, times when the data is communicated, and rates at which the data is communicated, detected vulnerable service communications, and detected attack signatures;
analyze the aggregations of the collected network flow data based on a security policy wherein the security policy is a collection of security technical control configuration policies that specify a specific operation of a security technical control;
dynamically assigning the security policy to the host network asset or the client network asset based on a trust zone to which the host network asset or the client network asset is assigned; and
generate an alert in response to the analysis of the aggregations of the collected network flow data.
16. A computer system comprising:
a processor; and
memory in communication with the processor and storing instructions that, when executed by the processor, cause the computer system to:
collect network flow data from a plurality of different types of sources on a network;
identify, based on the collected network flow data, a plurality of network assets and connections between the plurality of network assets;
generate a first aggregation of the collected network flow data that includes a subset of the collected network flow data over a first predetermined period of time;
generate a second aggregation of the collected network flow data that includes a subset of the collected network flow data over a second predetermined period of time;
store the aggregations in a first database during at least one of the first predetermined period of time or the second predetermined period of time;
load the aggregations from the first database to a second database after the first and second predetermined periods of time;
present, based on the aggregations of the collected network flow data, a graphical representation that includes the plurality of network assets and the connections between the plurality of network assets, wherein the graphical representation is presented via a display of a user interface in communication with the computer system;
wherein the plurality of network assets includes a host network asset and a client network asset,
wherein presenting the graphical representation includes presenting a flow information graph that depicts the host network asset, the client network asset, connections between the host network asset and the client network asset, and services that are permitted to be accessed between the host network asset and the client network asset,
wherein the flow information graph visually indicates an attribute of the host network asset, an attribute of the client network asset, and attributes of a connection between the host network asset and the client network asset, and
wherein the attributes of the connection between the host network asset and the client network asset sizes of data communicated, times when the data is communicated, and rates at which the data is communicated, detected vulnerable service communications, and detected attack signatures;
analyze the aggregations of the collected network flow data based on a security policy wherein the security policy is a collection of security technical control configuration policies that specify a specific operation of a security technical control;
dynamically assigning the security policy to the host network asset or the client network asset based on a trust zone to which the host network asset or the client network asset is assigned; and
generate an alert in response to the analysis of the aggregations of the collected network flow data.
17. The method of claim 1 , wherein the loading of the aggregations from the first database to the second database includes:
loading the aggregations to a new version of the second database, wherein the new version of the second database is associated with at least one of the first and second predetermined periods of time; and
replacing a previous version of the second database entirely.
18. The method of claim 17 , wherein the loading of the aggregations from the first database to the second database includes creating a new database partition for the new version of the second database.
19. The method of claim 1 , wherein the first database is a non-relational database and the second database is a relational database.
20. The method of claim 1 , wherein dynamically assigning the security policy to the host network asset or the client network asset based on a trust zone to which the host network asset or the client network asset is assigned comprises:
updating the security policy based on a movement of the host network asset or the client network asset from a first trust zone to a second trust zone.