Method for summarizing flow information of network devices
A method for organizing network flow information within a relational database relates to minimizing contention for the network flow information. In particular, data is generally aggregated at certain time intervals and separately organized. In this way, contention is decreased as analysis can occur on the separated existing flow records, which are optionally aggregated, while new raw flow records are stored. In another embodiment, the aggregated data can be re-aggregated at second, larger time intervals.
1. A system for aggregating network flow information, the system comprising:
a flow generating device configured to generate flow records from network traffic and to export the flow records to a data collection server comprising a non-transitory storage device,
the storage device comprising
a raw data table comprising new flow records for a current time period; and
a first aggregated data table comprising first aggregated flow records for a first prior time period and a second aggregated data table comprising second aggregated flow records for a second prior time period,
wherein, when the new flow records age beyond a predefined threshold, the new flow records are aggregated in the same raw data table and the raw data table becomes an aggregated data table and a new raw data table is created in the storage device,
wherein said current time period, said first prior time period, and said second prior time period do not overlap,
wherein the new flow records are sampled prior to being aggregated, and wherein the sampled flow records comprise estimations of flow volume.
2. The system of claim 1 , wherein said first prior time period and said second prior time period are of equal duration.
3. The system of claim 1 , wherein said first prior time period has a first duration and said second prior time period has a second duration, and wherein the first prior time period is longer than the second prior time period.
4. The system of claim 1 , further comprising a data analysis device, wherein said flow generating device is configured to concurrently provide the new flow records to the raw data table as the data analysis device accesses said first and second aggregated data tables.
5. The system of claim 1 , further comprising an archival storage system that is configured to store raw flow record data for the first prior time period.
6. A method for aggregating network data flows, the method comprising:
generating flow records from network traffic and exporting the flow records to a data collection server comprising a non-transitory storage device;
storing new flow records for a current period of time in a raw data table of the storage device;
storing, by the storage device, first flow records in a first table during a first prior period of time;
aggregating said first flow records in said first table;
after the first prior period of time and during a second prior period of time, creating a second table;
storing, by the storage device, second flow records in the second table during the second prior period of time;
aggregating said second flow records in said second table;
sampling the new flow records, wherein the sampling comprises estimating flow volume from the new flow records; and
when the new flow records age beyond a predefined threshold, aggregating the new flow records in the same raw data table and the raw data table becomes an aggregated data table and creating a new raw data table in the storage device;
wherein said sampling further comprises sampling said first flow records prior to the aggregating of said first flow records in said first table;
wherein said current period of time, said first prior period of time, and said second prior period of time do not overlap.
7. The method of claim 6 , wherein said first prior period of time and said second prior period of time are of equal duration.
8. The method of claim 6 , wherein said first prior period of time is longer than said second prior period of time.
9. The method of claim 6 , wherein said storing of the second flow records in the second table and said aggregating said first flow records in said first table occur concurrently.
10. The method of claim 6 , further comprising: archiving said first flow records and said second flow records; and compressing said archived first and second flow records.