Enhanched flow processing
A network monitoring device responds to a network status data (whether “pushed” from the network device or “pulled” from the network device), maintaining a buffer of saved status data. The status data is reordered, manipulated, and presented to users in order. The monitoring device can thus report an accurate momentary report of the status of the network environment. When status data is delayed too long, the monitoring device can discard it, or reduce its weighted consideration. The monitoring device adjusts its wait for status data, either as an average or individually per device, attempting to balance accuracy and latency. The monitoring device also records of how much status data it is required to process, in response to the amount it can process reliably, and maintains a sampling rate for status data, somewhere between evaluating all of the status data, and evaluating only a small portion of the status data, when capable, attempting to balance the degree of sampling, against both error and latency.
1. A method, including steps of
coupling a network monitoring device to a dynamic network environment, the network monitoring device including:
a buffer of saved data entries responsive to status of the network environment, the data in those data entries having been received from the network, the data entries having been reordered so that their order in the buffer differs from the order they were received,
a record of delay associated with the data entries, the record of delay being sufficient to allow the network monitoring device to determine an amount of weight to give to the data to balance accuracy and latency of reports on status of the network environment,
a record of amount of data the network monitoring device is able to process reliably, the record of an amount being sufficient to allow the network monitoring device to maintain a sampling rate to balance projected accuracy and latency of reports on status of the network environment;
determining a projected accuracy of reports on status of the network environment in response to a rate of receiving information from the network environment, the rate of receiving information being in response to a statistical measure of experience with data from the network environment;
processing the record of delay and the record of amount of data, wherein accuracy and latency of reports on status of the network environment are balanced in accordance with one or more parameters set by an operator of the network monitoring device;
adjusting one or more of: a size of the buffer, a rate of receiving data from the dynamic network environment, a rate of requesting data from the dynamic network environment, to provide an amount of data to be considered, wherein the amount of data to be considered is dynamically adjusted in response to changes in the network environment; and
dynamically adjusting one or more of: the amount of weight to give to the data, the sampling rate, the amount of data to be considered, adjusted in response to changes in the dynamic network environment, wherein accuracy and latency of reports on a status of the network environment are balanced in accordance with one or more parameters set by an operator of the network monitoring device;
wherein the network monitoring device gives more or less weight to more recent data from the network environment, the network environment provides more or less data to the network monitoring device, in response to activity of the network environment.
2. The method as in claim 1 , including steps of
receiving the data entries in a first format; and
converting the data entries from the first format to a second format.
3. The method as in claim 1 , wherein
the network monitoring device receives data from the network environment in message packets.
4. The method as in claim 1 , wherein
at least some of the data entries received from the network indicate data that is unclear or unknown.
5. The method as in claim 1 , wherein
at least some of the data received from the network is fuzzy data or probabilistic data.
6. The method as in claim 1 , wherein
the buffer includes at least one of: a tree structure, a directed graph structure, a lattice structure, another hierarchical data structure.
7. The method as in claim 1 , wherein
the data entries can indicate one or more of: processing capacity, memory capacity, storage capacity, network activity, or other activity or capability of a selected network device.
8. The method as in claim 1 , wherein
the data received from the network includes status data maintained in a shared memory accessible both by the network monitoring device and by one or more devices in the dynamic network environment.
9. The method as in claim 1 , wherein
the data received from the network includes status data maintained with respect to one or more of:
a selected subset of devices in the dynamic network environment, a selected subset of endpoints of traffic in the dynamic network environment, a selected subset of virtual machine applications in the dynamic network environment, a selected subset of virtual machines in the dynamic network environment, or another selected subset of activity in the dynamic network environment.
10. The method as in claim 1 , wherein
the data received from the network includes:
status data send from first devices in the dynamic network environment is sent to the network monitoring device at the initiative of those first devices; and
status data send from second devices in the dynamic network environment is sent to the network monitoring device in response to a request by the network monitoring device to those second devices.
11. The method as in claim 10 , wherein
one or more requests from the network monitoring device to those second devices is sent to those second devices in a round-robin fashion.
12. The method as in claim 1 , wherein
the network monitoring device receives data from the network environment in message packets.
13. A system including
a network monitoring device disposed to be coupled to a dynamic network environment, the network monitoring device including:
a buffer of saved data entries responsive to status of the network environment, the data in those data entries having been received from the network, the data entries having been reordered so that their order in the buffer differs from the order they were received,
a record of delay associated with the data entries, the record of delay being sufficient to allow the network monitoring device to determine an amount of weight to give to the data to balance accuracy and latency of reports on status of the network environment,
a record of amount of data the network monitoring device is able to process reliably, the record of an amount being sufficient to allow the network monitoring device to maintain a sampling rate to balance projected accuracy and latency of reports on status of the network environment,
instructions directing a computing device to perform one or more steps of:
determining a projected accuracy of reports on status of the network environment in response to a rate of receiving information from the network environment, the rate of receiving information being in response to a statistical measure of experience with data from the network environment;
processing the record of delay and the record of amount of data, wherein accuracy and latency of reports on status of the network environment are balanced in accordance with one or more parameters set by an operator of the network monitoring device;
adjusting one or more of: a size of the buffer, a rate of receiving data from the dynamic network environment, a rate of requesting data from the dynamic network environment, to provide an amount of data to be considered, wherein the amount of data to be considered is dynamically adjusted in response to changes in the network environment; and
dynamically adjusting one or more of: the amount of weight to give to the data, the sampling rate, the amount of data to be considered, adjusted in response to changes in the dynamic network environment, wherein accuracy and latency of reports on a status of the network environment are balanced in accordance with one or more parameters set by an operator of the network monitoring device;
wherein the network monitoring device gives more or less weight to more recent data from the network environment, the network environment provides more or less data to the network monitoring device, in response to activity of the network environment.