Methods and apparatus for providing adaptive private network centralized management system timestamp correlation processes
System and techniques are described which provide a method for time correlated monitoring of nodes in a network. A statistics request message is sent from a centralized control point to a node in a network having a plurality of nodes, wherein the statistics request message contains a centralized control point time stamp t n of when the message was sent. The statistics request message is received at the node which records a node timestamp t a when the message was received in the node. Node statistics are converted to system time at the centralized control point according to a first delta between t n and t a to generate adjusted node statistics. A second delta is determined at the centralized control point which is applied to the adjusted node statistics to provide consistent system time information for feedback or presentation through a network attached user interface.
1. A method to provide a consistent system timeline of node communications in an adaptive private network (APN) having a plurality of nodes, the method comprising:
providing a first APN virtual machine that maintains an APN time used for forwarding packets through conduits in the APN;
providing an APN appliance that forwards packets through the conduits in the APN and that maintains time local to the APN appliance;
repeatedly sending statistics request messages from the first APN virtual machine to the APN appliance, wherein the first APN virtual machine comprises monitor, manager, and network statistics processing functions which are configured to provide the consistent system timeline and each of the statistics request messages contains a timestamp t n of when the statistics request message was sent in the APN time maintained by the first APN virtual machine;
receiving each of the statistics request messages at the APN appliance and recording a timestamp t a in the local time of the APN appliance when each of the statistics request messages was received at the APN appliance;
in response to each received statistics request message, returning requested node statistics from the APN appliance to the first APN virtual machine, wherein the requested node statistics each include a timestamp recorded in the local time of the APN appliance and state information regarding the APN appliance;
determining, by the first APN virtual machine and using the timestamps associated with successive statistics received from the APN appliance, a time jump in the local time of the APN appliance that does not correspond to a timestamping interval between the successive statistics; and
adjusting, by the first APN virtual machine and using the time jump, the timestamps of the statistics received from the APN appliance to form adjusted node statistics which are included in the consistent system timeline of node communications in the APN at the first APN virtual machine, wherein the consistent system timeline is provided for feedback through a network attached user interface.
2. The method of claim 1 , wherein time-stamped counter values are received from nodes in the APN and converted to the APN time maintained by the first APN virtual machine without requiring time synchronization between any nodes in the APN.
3. The method of claim 1
wherein determining the time jump includes subtracting a timestamp value from a first statistic from a timestamp value associated with a second statistic recorded after the first statistic.
4. The method of claim 1 wherein adjusting the timestamps includes adjusting the timestamps to correspond to the APN time maintained by the first APN virtual machine.
5. The method of claim 1 , wherein the timestamping interval is guaranteed at each node in the APN to be of a fixed time period and each requested set of statistics data is time-stamped in consecutive time periods.
6. The method of claim 1 , wherein the timestamping interval is guaranteed to be of a fixed time period allowing the received node statistics to be adjusted using the time jump and the timestamping interval.
7. The method of claim 1 , wherein a local clock in each node in the APN is synchronized with a network time in a second APN virtual machine separate from the first APN virtual machine based on an evaluation of a function of a ratio of an average send time from the second APN virtual machine versus average arrival time for a plurality of timing messages received at each node, current time in each node, and a round trip time between the second APN virtual machine and each node in the APN, wherein timestamps of data gathered from the second APN virtual machine and each node in the APN are correlated to a master time specified by the second APN virtual machine and the synchronized clocks in each node in the APN are calculated separately from providing the consistent system timeline.
8. The method of claim 1 , wherein communication objects are counted by counters and sampled for inclusion in the node statistics in responding to the statistics request message.
9. The method of claim 1 , wherein the consistent system timeline accounts for different gaps in time that occur in different nodes in the APN.
10. A method for adapting a system timeline at an adaptive private network (APN) virtual machine based on correlated monitoring of appliances in an APN having time jumps that occur at the appliances, the method comprising:
providing an APN virtual machine that maintains an APN time used for forwarding packets through conduits in the APN;
providing an APN appliance that forwards packets through the conduits in the APN and that maintains time local to the APN;
receiving, at the first APN virtual machine, statistics from the APN appliance, each of the statistics including a timestamp recorded in the local time of the APN and state information regarding the APN appliance, wherein the first APN virtual machine comprises monitor, manager, and network statistics processing functions which are configured to provide a consistent system timeline;
determining, by the first APN virtual machine and using the timestamps associated with successive statistics received from the APN appliance, a time jump in the local time of the APN appliance that does not correspond to a timestamping interval between the successive statistics;
and
adjusting, by the first APN virtual machine and using the time jump, the timestamps of the statistics received from the APN appliance to correlate in time the statistics received from the APN appliance with statistics received from other nodes in the APN on the consistent system timeline and the time correlated received node statistics provided at a program accessible storage location.
11. The method of claim 10 , wherein an aggregate bandwidth limit is specified for total statistics of request and reply messaging across the appliances in the APN and actual used bandwidth across the APN may be modified by adjusting polling parameters.
12. The method of claim 10 further comprising:
sending a request for statistics to the APN appliance for one polling period and repeatedly sending the request according to a periodic polling process.
13. The method of claim 12 , wherein the periodic polling process is based on a fixed prespecified period between sending requests for statistics.
14. The method of claim 10 further comprising:
sending, to the APN appliance, a statistics request message indicating that statistics are to be gathered for a number of fixed prespecified periods since a last statistics request message was received.
15. The method of claim 14 , wherein sending of a statistics request message to gather statistics over a number of fixed prespecified periods is a catch up mode that is entered if no response has been received for prior requests sent to the APN appliance.
16. The method of claim 10 , wherein time-stamped counter values are received from the nodes in the APN and converted to the APN time maintained by the first APN virtual machine without requiring time synchronization between any nodes in the APN.
17. The method of claim 16 , wherein local time changes on remote nodes in the APN are converted to the APN time maintained by the first APN virtual machine according to timing information included in statistic information received from each of the remote nodes.
18. The method of claim 16 , wherein the first APN virtual machine connects to a management server that provides a user interface for presenting the time correlated received node statistics on APN maps and in APN reports.
19. The method of claim 10 , wherein each node in the APN has a local clock synchronized to a master clock in a second APN virtual machine separate from the first APN virtual machine and each node's local synchronized clock is calculated separately from providing the consistent system timeline.
20. A computer readable non-transitory medium storing a computer program which causes a computer system to perform a method to provide a consistent system timeline of node communications in an adaptive private network (APN) having a plurality of nodes, the method comprising:
providing a first APN virtual machine that maintains an APN time used for forwarding packets through conduits in the APN;
providing an APN appliance that forwards packets through the conduits in the APN and that maintains time local to the APN appliance;
repeatedly sending statistics request messages from the first APN virtual machine to the APN appliance, wherein the first APN virtual machine comprises monitor, manager, and network statistics processing functions which are configured to provide the consistent system timeline and each of the statistics request messages contains a timestamp t n of when the message was sent in the APN time maintained by the first APN virtual machine;
receiving each of the statistics request messages at the APN appliance and recording a timestamp t a in the local time of the APN appliance when each of the statistics request messages was received at the APN appliance;
in response to each received statistics request message, returning requested node statistics to the first from the APN appliance to the first APN virtual machine, wherein the requested node statistics each include a timestamp recorded in the local time of the APN appliance and state information regarding the APN appliance;
determining, by the first APN virtual machine and using the timestamps associated with successive statistics received from the APN appliance, a time jump in the local time of the APN appliance that does not correspond to a timestamping interval between the successive statistics; and
adjusting, by the first APN virtual machine and using the time jump, the timestamps of the statistics received from the APN appliance to form adjusted node statistics which are included in the consistent system timeline of node communications in the network at the first APN virtual machine, wherein the consistent system timeline is provided for presentation through a network attached user interface.
21. The computer readable non-transitory medium method of claim 20 , wherein timestamped counter values are received from the nodes in the APN and converted to the APN time maintained by the first APN virtual machine without requiring time synchronization between any nodes in the APN.
22. The computer readable non-transitory medium method of claim 20
wherein determining the time jump includes subtracting a timestamp value associated with a first statistic from a timestamp value associated with a second statistic recorded by the APN appliance after the first statistic.
23. The computer readable non-transitory medium method of claim 20
wherein adjusting the timestamps includes adjusting the timestamps to correspond to the APN time maintained by the first APN virtual machine.
24. The computer readable non-transitory medium method of claim 20 further comprising:
sending a second statistics request message to the APN appliance that statistics are to be gathered for a number of fixed prespecified periods since the last request was received, wherein the sending a request to gather statistics over the number of fixed prespecified periods is a catch up mode that is entered if no response has been received for prior requests sent to the APN appliance.