Adaptive, flow-based network traffic measurement and monitoring system
View Patent ↗Methods, apparatuses and systems directed to an adaptive network traffic monitoring and measurement system. In one implementation, the present invention provides a measurement engine that monitors data flows on the packet processing path of a network device and executes measurement policy that control the granularity with which measurement data is recorded. In one implementation, the present invention provides a filtering and aggregation mechanism that can seamlessly adapt to changes in measurement policy configuration.
1. A method for recording network traffic data, comprising:
maintaining, in a memory of a network device, a measurement data structure including at least one entry and a corresponding measurement key, wherein the at least one entry comprises at least one measurement attribute;
receiving, at the network device, a packet having a plurality of packet attributes;
associating the packet to a data flow by hashing a plurality of packet attributes of the packet to create a first flow key and matching the first flow key to a second flow key associated with a data flow object of a plurality of data flow objects, wherein each data flow object includes a plurality of data flow attributes, and adding a new data flow object that includes the first flow key to the plurality of data flow objects, if the packet does not match any data flow objects of the plurality of data flow objects;
selecting, based on one or more data flow attributes of the data flow object corresponding to the data flow, a measurement engine policy from a plurality of measurement engine policies;
creating a second measurement key for the packet based on the selected measurement engine policy and at least one of the plurality of data flow attributes, wherein the plurality of packet attributes used to create the first flow key and the at least one of the data flow attributes used to create the second measurement key differ by at least attribute; and
matching the second measurement key to an entry in a measurement data structure; and
if a matching entry is found, updating a measurement value of the entry based on the packet;
if no matching entry is found, creating a new entry in the measurement data structure.
2. The method of claim 1 wherein a first measurement engine policy of the plurality of measurement engine policies is applied by default if no other measurement engine policy is specifically identified.
3. The method of claim 1 wherein the measurement policy identifies at least one measurement axis.
4. The method of claim 3 , wherein the at least one measurement axis is one selected from a group consisting of a service type identifier, an initiator network address, an acceptor network address, a flow control policy identifier, a measurement tag, an ingress interface, an egress interface, and a port identifier.
5. The method of claim 2 , wherein the measurement policy identifies at least one measurement axis, wherein the selecting step comprises comparing the at least one measurement axis to corresponding attributes of the data flow object.
6. The method of claim 1 wherein the creating step comprises concatenating, based on the measurement policy, a plurality of attributes in the data flow object; and
hashing the concatenated attributes to create the second measurement key.
7. The method of claim 3 wherein the creating step comprises
concatenating, based on the at least one measurement axis identified in the measurement policy, a plurality of attributes in the data flow object; and
hashing the concatenated attributes to create the second measurement key.
8. The method of claim 1 further comprising
periodically writing the data in the measurement data structure to a persistent data store; and
clearing the measurement data structure.
9. The method of claim 1 wherein the measurement data structure comprises a table.
10. The method of claim 9 wherein the second measurement key is an index to the entries in the table.
11. An apparatus for recording network traffic data, comprising:
a processor;
at least one network interface,
a memory, the memory comprising a measurement data structure, and at least one data flow object;
a software application, physically stored in the memory, comprising instructions operable to cause the processor and the apparatus to:
receive a packet having a plurality of packet attributes;
associate the packet to a data flow by hashing a plurality of packet attributes of the packet to create a first flow key and matching the first flow key to a second flow key associated with a data flow object in the memory, wherein the data flow object includes a plurality of data flow attributes, and adding a new data flow object that includes the first flow key in the memory, if the packet does not match any data flow objects existing in the memory;
select, based on one or more data flow attributes of the data flow object corresponding to the data flow, a measurement engine policy from a plurality of measurement engine policies;
create a measurement key for the packet based on the selected measurement engine policy and at least one of the plurality of data flow attributes, wherein the plurality of packet attributes used to create the first flow key and the at least one of the data flow attributes used to create the measurement key differ by at least attribute; and
match the measurement key to an entry in a measurement data structure; and
if a matching entry is found, update a measurement value in the entry based on the packet;
if no matching entry is found, create a new entry in the measurement data structure.
12. The apparatus of claim 11 wherein the software application further comprises instructions operative to cause the processor to
select a first measurement engine policy of the plurality of measurement engine policies by default if no other measurement engine policy is specifically identified.
13. The apparatus of claim 11 wherein the measurement policy identifies at least one measurement axis.
14. The method of claim 13 , wherein the at least one measurement axis is one selected from a group consisting of a service type identifier, an initiator network address, an acceptor network address, a flow control policy identifier, a measurement tag, an ingress interface, an egress interface, and a port identifier.
15. The apparatus of claim 12 , wherein the measurement policy identifies at least one measurement axis, wherein to select the measurement policy, the software application further comprises instructions operative to cause the processor to compare the at least one measurement axis to corresponding attributes of the data flow object.
16. The apparatus of claim 11 wherein to create the measurement key, the software application further comprises instructions operative to cause the processor to
concatenate, based on the measurement policy, a plurality of attributes in the data flow object; and
hash the concatenated attributes to create measurement key.
17. The apparatus of claim 13 wherein to create the measurement key, the software application further comprises instructions operative to cause the processor to
concatenate, based on the at least one measurement axis identified in the measurement policy, a plurality of attributes in the data flow object; and
hash the concatenated attributes to create measurement key.
18. The apparatus of claim 11 further comprising a persistent data store, and wherein the software application further comprises instructions operative to cause the processor to
periodically write the data in the measurement data structure to a persistent data store; and
clear the measurement data structure.
19. The apparatus of claim 11 wherein the measurement data structure comprises a table.
20. The apparatus of claim 19 wherein the measurement key is an index to the entries in the table.