Systems and methods for capturing and consolidating packet tracing in a cluster system
View Patent ↗The present solution relates to systems and methods for capturing and consolidating packet tracing in a cluster system. A multi-nodal cluster processing network traffic contains multiple nodes each handling some of the processing. A node may initially receive a flow and transfer processing of the flow to another node for processing. A flow may therefore pass from one node to another, from two nodes to many nodes. In some instances, it is helpful to generate a trace of a flow. For example, in debugging a network communication flow, a trace of the flow through the cluster can be helpful. Each node has a packet engine (“PE”) which processes data packets and can, when trace is enabled, generate a trace file for the packets processed at the respective node. A trace aggregator merges these distinct trace files into an aggregate trace for the cluster.
1. A system for consolidating packet trace messages of a cluster of intermediary devices, the system comprising:
a cluster of intermediary devices interconnected by a communication back plane, each of the intermediary devices comprising one or more network interfaces for communications via a network,
a plurality of packet processors each configured to process network packets received at one or more of the network interfaces and write messages to a respective trace buffer, the messages comprising information to trace the network packets and source information identifying the respective packet processor and a sequence number, and
at least one trace aggregator configured to gather the messages into a device log and a first intermediary device of the cluster further configured to collect the respective device logs into an aggregated device log for the cluster.
2. The system of claim 1 , wherein each intermediary device comprises a plurality of cores, and each core of the plurality of cores is configured to execute one or more of the plurality of packet processors.
3. The system of claim 1 , wherein the respective trace buffers are stored in memory shared between the plurality of packet processors.
4. The system of claim 1 , wherein the at least one trace aggregator is further configured to read trace log entries from the trace buffers, apply a filtering constraint to the trace entries, and aggregate the trace entries satisfying the filtering constraint into the device log.
5. The system of claim 1 , wherein each packet processor of the plurality of packet processors is further configured to write packet trace messages to its respective trace buffer responsive to being placed in a trace mode by a control message.
6. The system of claim 5 , wherein each packet processor of the plurality of packet processors is configured to stop writing messages to its respective trace buffer responsive to being removed from the trace mode by a second control message, and wherein the first intermediary device in the cluster of intermediary devices is configured to collect the respective device logs after generating the second control message.
7. The system of claim 1 , wherein the first intermediary device of the cluster is further configured to sort the collected device logs in the aggregated device log for the cluster according to the sequence numbers.
8. The system of claim 1 , wherein the sequence numbers are synchronized time stamps.
9. The system of claim 1 , wherein each packet processor of the plurality of packet processors is configured to operate in between at least one of a logging mode or a non-logging mode, responsive to a control message, and each packet processor is configured to transmit its respective device log to the first intermediary device upon entering the non-logging mode.
10. The system of claim 1 , wherein each respective trace buffer is a ring buffer.
11. A method for consolidating packet trace messages of a cluster of intermediary devices, the method comprising:
writing, by each packet processor of a plurality of packet processors in each intermediary device of a cluster of intermediary devices interconnected by a communication back plane, messages to a trace buffer, the messages comprising information to trace the network packets processed by each respective packet processor and source information identifying the respective packet processor and a sequence number;
gathering, by a trace aggregator for each intermediary device, the messages from the trace buffers into a device log for the respective intermediary device; and
collecting, by a first intermediary device in the cluster of intermediary devices, the device logs from each of the intermediary devices for an aggregated device log for the cluster.
12. The method of claim 11 , wherein each intermediary device comprises a plurality of cores, and each core of the plurality of cores is configured to execute one or more of the plurality of packet processors.
13. The method of claim 11 , further comprising:
reading, by the trace aggregator, trace log entries from the trace buffers;
applying, by the trace aggregator, a filtering constraint to the trace entries; and
aggregating, by the trace aggregator, the trace entries satisfying the filtering constraint into the device log.
14. The method of claim 11 , further comprising writing, by each packet processor, packet trace messages to its respective trace buffer responsive to being placed in a trace mode by a control message.
15. The method of claim 14 , further comprising ceasing, by each packet processor, to write packet trace messages to its respective trace buffer responsive to being removed from the trace mode by a second control message, and collecting, by the first intermediary device, the respective device logs after generating the second control message.
16. The method of claim 11 , further comprising sorting, by the first intermediary device, the collected device logs in the aggregated device log for the cluster according to the sequence numbers.
17. The method of claim 11 , further comprising transmitting, by the first packet processor, its respective device log to the first intermediary device.
18. The method of claim 17 , further comprising receiving, by the first packet processor, a control message; terminating packet trace collection responsive to receiving the control message; and transmitting, by the first packet processor, its respective device log to the first intermediary device after terminating the packet trace collection.
19. A system comprising:
a cluster of intermediary devices interconnected by a communication back plane, each of the intermediary devices comprising one or more network interfaces for communications via a network,
a plurality of packet processors each configured to process network packets received at one or more of the network interfaces and write messages to a respective trace buffer, the messages comprising information to trace the network packets,
at least one trace aggregator configured to gather the messages into a device log and a first intermediary device of the cluster further configured to collect the respective device logs into an aggregated device log for the cluster,
wherein each packet processor of the plurality of packet processors is configured to write packet trace messages to its respective trace buffer responsive to being placed in a trace mode by a control message and to stop writing messages to its respective trace buffer responsive to being removed from the trace mode by a second control message, and
wherein the first intermediary device in the cluster of intermediary devices is configured to collect the respective device logs after generating the second control message.