Distributed network traffic data collection and storage
View Patent ↗Network traffic information from multiple sources, at multiple time scales, and at multiple levels of detail are integrated so that users may more easily identify relevant network information. The network monitoring system stores and manipulates low-level and higher-level network traffic data separately to enable efficient data collection and storage. Packet traffic data is collected, stored, and analyzed at multiple locations. The network monitoring locations communicate summary and aggregate data to central modules, which combine this data to provide an end-to-end description of network traffic at coarser time scales. The network monitoring system enables users to zoom in on high-level, coarse time scale network performance data to one or more lower levels of network performance data at finer time scales. When high-level network performance data of interest is selected, corresponding low-level network performance data is retrieved from the appropriate distributed network monitoring locations to provide additional detailed information.
1. A method of collecting and aggregating distributed traffic network data, the method comprising:
collecting first and second packet traffic data from first and second network locations, respectively;
storing first and second packet traffic data in first and second data storage devices at the first and second network locations, respectively;
generating first and second summary network traffic data at the first and second network locations, respectively, wherein the first and second summary network traffic data is at a first time scale coarser than a time scale associated with the first and second packet traffic data;
communicating first and second summary network traffic data from the first and second network locations to a third network location; and
storing the first and second summary network traffic data at the third network location;
wherein the first summary network traffic data includes first references adapted to retrieve at least corresponding portions of the first packet traffic data from the first network location and the second summary network traffic data includes second references adapted to retrieve at least corresponding portions of the second packet traffic data from the second network location.
2. The method of claim 1 , comprising:
generating third summary network traffic data from the first and second summary network traffic data, wherein the third summary network traffic data includes third references adapted to retrieve at least corresponding portions of the first and second summary network traffic data.
3. The method of claim 2 , wherein the first and second summary network traffic data include duplicated flow traffic data and the third summary network traffic data includes connected flow traffic data.
4. The method of claim 3 , wherein the third summary network traffic data includes related network traffic flows having one of the same source or destination as at least a portion of the connected flow traffic data.
5. The method of claim 2 , wherein the third summary network traffic data includes network traffic data at a second time scale coarser than the first time scale.
6. The method of claim 5 , wherein the network traffic data at the second time scale includes macroflow traffic data.
7. The method of claim 2 , comprising:
storing the third summary network traffic data at the third network location.
8. The method of claim 1 , wherein the first references included in the first summary network traffic data are associated with third summary network traffic data stored at the first network location and having a second time scale finer than the first time scale and coarser than the time scale associated with the first packet traffic data.
9. The method of claim 8 , wherein the third summary network traffic data includes microflow traffic data.
10. The method of claim 1 , wherein the first summary network traffic data includes an attribute common to at least a portion of the first packet traffic data.
11. The method of claim 1 , wherein the first summary network traffic data includes an aggregate attribute of at least a portion of the first packet traffic data.
12. The method of claim 1 , wherein the first summary network traffic data includes a statistical attribute determined from at least a portion of the first packet traffic data.
13. The method of claim 1 , wherein the first summary network traffic data includes a user-configured attribute of at least a portion of the first packet traffic data.
14. The method of claim 13 , wherein the user-configured attribute is determined by executing user-provided code.
15. The method of claim 1 , wherein storing first packet traffic data in the first data storage device at the first network location comprises:
identifying a portion of the first packet traffic data collected within a first time period;
identifying a packet data block reference in a data storage device file system associated with the first time period;
storing the portion of the first packet traffic data in at least one data block of the data storage device file system; and
adding a reference to the at least one data block storing the portion of the first packet traffic data to the packet data block reference.
16. The method of claim 1 , wherein collecting and storing the first packet traffic data is performed at least partially within a virtual machine application.
17. The method of claim 1 , wherein collecting and storing the first packet traffic data is performed at least partially within a network monitoring appliance.
18. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method of collecting and aggregating distributed traffic network data, the method comprising:
collecting first and second packet traffic data from first and second network locations, respectively;
storing first and second packet traffic data in first and second data storage devices at the first and second network locations, respectively;
generating first and second summary network traffic data at the first and second network locations, respectively, wherein the first and second summary network traffic data is at a first time scale coarser than a time scale associated with the first and second packet traffic data;
communicating first and second summary network traffic data from the first and second network locations to a third network location; and
storing the first and second summary network traffic data at the third network location;
wherein the first summary network traffic data includes first references adapted to retrieve at least corresponding portions of the first packet traffic data from the first network location and the second summary network traffic data includes second references adapted to retrieve at least corresponding portions of the second packet traffic data from the second network location.
19. An apparatus, comprising:
one or more processors; and
a non-transitory computer-readable storage medium storing instructions that, when executed by the one or more processors, cause the apparatus to perform a method of collecting and aggregating distributed traffic network data, the method comprising:
collecting first and second packet traffic data from first and second network locations, respectively;
storing first and second packet traffic data in first and second data storage devices at the first and second network locations, respectively;
generating first and second summary network traffic data at the first and second network locations, respectively, wherein the first and second summary network traffic data is at a first time scale coarser than a time scale associated with the first and second packet traffic data;
communicating first and second summary network traffic data from the first and second network locations to a third network location; and
storing the first and second summary network traffic data at the third network location;
wherein the first summary network traffic data includes first references adapted to retrieve at least corresponding portions of the first packet traffic data from the first network location and the second summary network traffic data includes second references adapted to retrieve at least corresponding portions of the second packet traffic data from the second network location.