Systems and methods for N-core tracing
Described herein are systems and methods for aggregating performance data or trace data for a multi-core, multi-packet-engine networked appliance. A plurality of packet engines operate on the appliance and manage network traffic between a plurality of clients and a plurality of servers. In various embodiments, the system comprises a plurality of packet engines and an aggregator communicating with the packet engines through shared memory allocations. Each packet engine can write performance data or trace data to the a shared memory allocation. The aggregator can obtain the performance data or trace data and consolidate the data to provide unified performance data or unified trace data representative of the appliance.
1. A method of aggregating a trace of network traffic for an appliance intermediary to a plurality of clients and a server, the appliance comprising a plurality of cores and a plurality of packet engines, one or more of the packet engines managing network traffic between the server and one or more of the clients, the method comprising:
a) capturing, by each packet engine executing on a respective core of the appliance comprising a plurality of cores, trace information for network traffic received or transmitted by each packet engine, each packet engine storing trace information to memory shared with an aggregator responsive to a command stored by the aggregator to a location in the shared memory;
b) storing, by the aggregator executing on a core of the plurality of cores, to a buffer the trace information obtained from each packet engine of the plurality of packet engines, the aggregator obtaining trace information stored by each packet engine in the shared memory responsive to a flag set by each packet engine;
c) consolidating, by the aggregator, the trace information from all packet engines to provide unified trace data of network traffic for the appliance;
d) receiving, by the aggregator, a request from an agent for network trace data of the appliance; and
e) transmitting, by the aggregator, the unified trace data of network traffic in response to the request.
2. The method of claim 1 , wherein step (a) further comprises
a-1) storing, by the aggregator, a first command in a first location of memory shared with a first core, the first command identifying a first request for trace information from a first packet engine of the first core; and
a-2) storing, by the aggregator, a second command in a second location of memory shared with a second core, the second command identifying a second request for trace information from a second packet engine of the second core.
3. The method of claim 2 , wherein step (b) further comprises
b-1) reading, by the aggregator, trace information stored by the first packet engine in memory shared between the aggregator and the first core, the reading responsive to detecting a change to data of a first flag by the first packet engine to indicate that the first packet engine has written data to memory; and
b-2) reading, by the aggregator, trace information stored by the second packet engine in memory shared between the aggregator and the second core, the reading responsive to detecting a change to data of a second flag by the second packet engine to indicate that the second packet engine has written data to memory.
4. The method of claim 1 , wherein the unified trace data does not identify the plurality of cores of the appliance.
5. The method of claim 1 further comprising converting the unified trace data to a format recognizable by the agent.
6. The method of claim 1 , wherein step (a) further comprises filtering, by each packet engine, trace information for network traffic and wherein step (c) further comprises consolidating, by the aggregator, the filtered trace information and providing a unified record of trace information for the appliance in accordance with the filtering.
7. The method of claim 1 , wherein step (a) further comprises capturing, by each packet engine, trace information for network traffic from an identified source IP address and wherein step (c) further comprises consolidating, by the aggregator, the captured trace information and providing a unified record of network traffic managed by the appliance from the identified source IP address.
8. The method of claim 1 , wherein step (a) further comprises capturing, by each packet engine, trace information for network traffic to an identified destination IP address and wherein step (c) further comprises consolidating, by the aggregator, the captured trace information and providing a unified record of network traffic managed by the appliance to the identified destination IP address.
9. The method of claim 1 , wherein step (c) further comprises concatenating trace data for a time interval from each packet engine.
10. The method of claim 1 , wherein step (c) further comprises combining trace data according to a source address and destination address and concatenating the combined trace data.
11. A system for aggregating a trace of network traffic for an appliance intermediary to a plurality of clients and a server, the appliance comprising a plurality of cores and a plurality of packet engines, one or more of the packet engines managing network traffic between the server and one or more of the clients, the system comprising:
a plurality of packet engines, each packet engine executing on a respective core of the appliance comprising a plurality of cores, each packet engine capturing trace information for network traffic received or transmitted by each packet engine, each packet engine storing trace information to memory shared with an aggregator responsive to a command stored by the aggregator to a location in the shared memory; and
the aggregator executing on a core of the plurality of cores, the aggregator storing to a buffer the trace information obtained from each packet engine of the plurality of packet engines, and consolidating the trace information from all packet engines to provide unified trace data of network traffic for the appliance, the aggregator obtaining trace information stored by each packet engine in the shared memory responsive to a flag set by each packet engine further wherein
the aggregator receives a request from an agent for network trace data of the appliance; and transmits the unified trace data of network traffic in response to the request.
12. The system of claim 11 , wherein the aggregator further stores a first command in a first location of memory shared with a first core, the first command identifying a first request for trace information from a first packet engine of the first core, and
stores a second command in a second location of memory shared with a second core, the second command identifying a second request for trace information from a second packet engine of the second core.
13. The system of claim 12 , wherein the aggregator further reads trace information stored by the first packet engine in memory shared between the aggregator and the first core responsive to detecting a change to a data of a first flag by the first packet engine to indicate that the first packet engine has written data to memory, and reads trace information stored by the second packet engine in memory shared between the aggregator and the second core responsive to detecting a change to a data of a second flag by the second packet engine to indicate that the second packet engine has written data to memory.
14. The system of claim 11 , wherein the unified trace data does not identify the plurality of cores of the appliance.
15. The system of claim 11 , wherein the aggregator converts the unified trace data to a format recognizable by the agent.
16. The system of claim 11 , wherein each packet engine filters trace information for network traffic and wherein the aggregator consolidates the filtered trace information and provides a unified record of trace information for the appliance in accordance with the filtering.
17. The system of claim 11 , wherein each packet engine captures trace information for network traffic from an identified source IP address and wherein the aggregator consolidates the captured trace information and provides a unified record of network traffic managed by the appliance from the identified source IP address.
18. The system of claim 11 , wherein each packet engine captures trace information for network traffic to an identified destination IP address and wherein the aggregator consolidates the captured trace information and provides a unified record of network traffic managed by the appliance to the identified destination IP address.
19. The system of claim 11 , wherein the aggregator concatenates trace data for a time interval from each packet engine.
20. The system of claim 11 , wherein the aggregator combines trace data according to a source address and destination address and concatenates the combined trace data.