Systems and methods to collect logs from multiple nodes in a cluster of load balancers
The systems and methods of the present solution are directed to collecting log information from multiple nodes in a multi-nodal cluster. Generally, a logging process runs to collect log information from multiple nodes in a multi-nodal cluster, e.g., a cluster of appliances. The logging process collects the log information and merges the collected log information to create a coherent unified log. The logging process may run on a node designated for the purpose. The designated node may be internal or external to the cluster. The logging process determines a topology for the cluster, establishes a communication channel with each active intermediary device identified in the topology, collects log entries from each active intermediary device, each log entry comprising information on network traffic traversing the respective intermediary device, and merges the collected log entries into a unified cluster log comprising information on network traffic traversing the cluster.
1. A method for collecting logs from multiple nodes in a cluster of intermediary devices, the method comprising:
receiving, by a logger of a first device in communication via a network with a cluster coordinator of a cluster of intermediary devices intermediary to a plurality of clients and a plurality of servers, information from the cluster coordinator identifying a first topology for the cluster, the first topology comprising identification of each active intermediary device of the cluster and a network address for each active intermediary device of the cluster;
establishing, by the logger of the first device responsive to the first topology received from the cluster coordinator, a communication channel via a transport layer connection between the first device and the network address for each active intermediary device identified in the first topology;
collecting, by the logger of the first device, via each communication channel, log entries from each active intermediary device identified in the topology, each log entry comprising information on network traffic traversing the respective intermediary device; and
merging, by the logger of the first device, the collected log entries into a unified cluster log comprising information on network traffic between the plurality of clients and the plurality of servers traversing each of the intermediary devices in the cluster ordered according to an ordering scheme.
2. The method of claim 1 , further comprising connecting, by the logger of the first device that is not an intermediary device of the cluster, to the cluster coordinator by the communication channel to a predetermined network address of the logger mapped to the cluster coordinator.
3. The method of claim 1 , wherein the cluster coordinator is at least one of a first intermediary device of the cluster acting as a view leader or a second intermediary device of the cluster acting as the view leader after the first intermediary device has failed.
4. The method of claim 1 , further comprising collecting, by each active intermediary device comprising multiple cores, log records from the multiple cores, and communicating, via each communication channel, the collected log records from the multiple cores to the first device.
5. The method of claim 1 , merging, by the first device, the collected log entries comprising sequencing information into the unified cluster log by sorting the collected log entries according to the sequencing information.
6. The method of claim 5 , further comprising sorting, by the first device, the collected log entries using a k-way heap merge.
7. The method of claim 5 , further comprising maintaining a happens-before ordering.
8. The method of claim 5 , further comprising maintaining, by the cluster, clock synchronicity between intermediary devices of the cluster and wherein the sequencing information is a timestamp.
9. The method of claim 1 , comprising maintaining, by the first device, a queue of transaction entries for each communication channel with each active intermediary device identified in the first topology.
10. The method of claim 9 , comprising maintaining, by the first device, a global heap;
and periodically extracting a transaction entry from the global heap and writing the extracted transaction entry to a log file.
11. The method of claim 1 , further comprising receiving, by the first device, from the cluster coordinator, an indication of a change in the cluster resulting in a second topology; and
establishing, by the first device, the communication channel with an intermediary device absent from the first topology and identified as active in the second topology.
12. A system for collecting logs from multiple nodes in a cluster of intermediary devices, the system comprising:
a cluster of intermediary devices intermediary to a plurality of clients and a plurality of servers; and
a logger of a first device in communication via a network with a cluster coordinator of the cluster, the logger of the first device configured to:
receive, information from the cluster coordinator identifying a first topology of the cluster, the first topology comprising identification of each active intermediary device of the cluster and a network address for each active intermediary device of the cluster;
establish, responsive to the first topology received from the cluster coordinator, a communication channel via a transport layer connection between the first device and the network address for each active intermediary device identified in the first topology;
collect, via each communication channel, log entries from each active intermediary device identified in the topology, each log entry comprising information on network traffic traversing the respective intermediary device; and
merge the collected log entries into a unified cluster log comprising information on network traffic between the plurality of clients and the plurality of servers traversing each of the intermediary devices in the cluster ordered according to an ordering scheme.
13. The system of claim 12 , wherein first device is not an intermediary device of the cluster, and the logger of the first device is configured to connect to the cluster coordinator by establishing the communication channel to a predetermined network address mapped to the cluster coordinator.
14. The system of claim 12 , wherein each active intermediary device comprising multiple cores is configured to collect log records from the multiple cores and communicate, via each communication channel, the collected log records from the multiple cores to the first device.
15. The system of claim 12 , wherein the first device is configured to merge the collected log entries comprising sequencing information into the unified cluster log by sorting the collected log entries according to the sequencing information.
16. The system of claim 15 , wherein the first device is configured to sort the log entries using a k-way heap merge.
17. The system of claim 15 , wherein the first device is configured to sort the log entries to maintain a happens-before ordering.
18. The system of claim 12 , wherein the first device is configured to maintain a queue of transaction entries for each communication channel with each active intermediary device identified in the first topology.
19. The system of claim 18 , wherein the first device is configured to maintain a global heap, periodically extract a transaction entry from the global heap, and write the extracted transaction entry to a log file.
20. The system of claim 12 , wherein the first device is configured to receive, from the cluster coordinator, an indication of a change in the cluster resulting in a second topology; and establish, in response to the indication of the change, the communication channel with an intermediary device absent from the first topology and identified as active in the second topology.