Method for partitioning network flows based on their time information
View Patent ↗To organize flows records into buckets and optimize their indexed access to/from storage medium, a bucket identifier is associated with each flow record. The bucket identifier is derived from some other characteristic of the network flow, such as a timestamp or source identification associated with the network flow. In this way, contention is decreased as analysis can occur on the aggregated flow records, while new flow records are stored.
1. A system for organizing network flow information, the system comprising:
a data storage system comprising a raw data table, wherein the raw data table comprises flow records comprising data categories and a plurality of cluster identifiers; and
a flow generating device configured to provide the flow records to the raw data table; and
wherein one of said cluster identifiers is appended to each of said flow records, wherein each of said flow records comprises a time stamp, and
wherein said cluster identifiers are produced according to timestamps such that each of the cluster identifiers identify different time periods;
wherein the data storage system further comprises sub-tables configured to receive data from said raw data table, wherein each of said sub-tables is uniquely associated with one of the cluster identifiers, and
wherein the raw data table is configured to be partitioned by moving each of the flow records into one said sub-tables associated with the cluster identifier appended to that flow record;
a data analysis device configured to analyze a aggregated data tables,
wherein said system is configured such that the flow generating device and the data analysis device operate concurrently.
2. The system of claim 1 , wherein the data storage system further comprises aggregated data tables, wherein each of said aggregated data tables is associated with one of said sub-tables and wherein each of said aggregated data tables is configured to receive and aggregate flow record data contained in said associated sub-table.
3. The system of claim 1 , wherein said time periods do not overlap.
4. The system of claim 1 , wherein said time periods are of equal duration.
5. The system of claim 1 , wherein each of said flow records comprises a network device identifier and where said cluster identifiers are produced according to said network device identifiers.
6. The system of claim 5 , wherein the data storage system further comprises:
a first aggregated data table containing first aggregated flow records for a first network device in a first prior time period, and
a second aggregated data table containing second aggregated flow records for a second network device in the first prior time period.
7. The system of claim 5 , wherein the data storage system further comprises:
a first aggregated data table containing first aggregated flow records for a first network device in a first time period, and
a second aggregated data table containing second aggregated flow records for the first network device in a second time period.
8. The system of claim 1 further comprising an archival storage system that is configured to store the raw flow record data for the first network device.
9. The system of claim 1 , wherein said cluster identifiers are further produced using at least one of said data categories.
10. A method for organizing network data flows, the method comprising:
during a first period of time, storing first flow records in a first table, said first flow records comprising data categories; and
appending a cluster identifier to each of the first flow records, wherein the cluster identifier is produced using at least one of said data categories;
after the first period of time and during a second period of time, creating a second table;
storing second flow records in the second table; and
moving each of said second flow records to sub-tables configured to receive data from said second table according to cluster identifiers, wherein each of said sub-tables is uniquely associated with one of the cluster identifiers;
wherein said storing second flow records in the second table and aggregating said identified first flow records in said first table occur concurrently.
11. The method of claim 10 further comprising aggregating the moved second flow records in each of the subtables.
12. The method of claim 10 , wherein said first period and said second period do not overlap.
13. The method of claim 10 , wherein said first period and said second period are of equal duration.
14. The method of claim 10 further comprising:
archiving said first flow records and said second flow records; and
compressing said archived first and second flow records.
15. A system for organizing network flow information, the system comprising:
a flow generating device configured to access a storage system to provide flow records;
a storage system configured to receive and store flow records and to append a cluster identifier to each of the flow records; and
a data analysis device configured to access the storage system,
wherein said flow generating device and said data analysis device are configured to access said storage system concurrently,
wherein each of said flow records comprises a time stamp and wherein said cluster identifier is produced according to said timestamps such that each cluster identifier identifies a different time period;
wherein the storage system further comprises aggregated data tables, wherein each of said aggregated data tables is associated with one of sub-tables and wherein each of said aggregated data tables is configured to receive and aggregate flow record data contained in said associated sub-table.
16. The system of claim 15 wherein the storage system comprises sub-tables, wherein each of said sub-tables is uniquely associated with one of the cluster identifiers.
17. The system of claim 15 , wherein each of said flow records comprises a network device identifier and where said cluster identifiers are produced according to said network device identifiers.
18. The system of claim 15 further comprising an archival storage system that is configured to store the raw flow record data.