Packet loss debug system and method
View Patent ↗A mechanism is provided for debugging of system-wide packet loss issues in network devices by automatically identifying packet loss conditions at runtime of the network device and by logging and analyzing relevant data to help diagnose the issues resulting in lost packets. A network programmer defines a path through the communications processor that identified packets should follow, and then hardware mechanisms within the modules of the communications processor are used to determine whether the packets are indeed following the defined path. If not, then the hardware mechanisms halt processing and logging being performed by all or part of the communications processor and provide logged information and packet information to an analysis tool for display. In this manner, debugging information can be provided in real time, along with a history of the packet's progress through the communication processor stages.
1. A communications processor configured to provide real-time packet debug, the communications processor comprising:
one or more packet processing cores coupled to one another by a communication bus;
one or more task accelerator modules, coupled to the one or more processing cores by the communication bus;
one or more packet watchpoint units, wherein
each packet processing core and each accelerator module has an associated packet watchpoint unit,
a set of the one or more packet processing cores and a set of the one or more task accelerator modules are configurable to form stages of a packet processing pipeline in which a previous pipeline stage passes data packet information to a subsequent pipeline stage from an initial pipeline stage to a final pipeline stage,
each packet watchpoint unit is configured to determine if a packet received by the associated packet processing core or task accelerator module was expected to be received, and
each packet processing core and task accelerator module is configured to log trace information regarding the packet,
each packet watchpoint unit is configured to halt logging the trace information in response to a determination that the packet received by the associated packet processing core or task accelerator module was not expected to be received; and
a memory, coupled to the one or more packet processing cores and the one or more task accelerator modules by the communication bus, and configured to store the log information regarding the packet.
2. The communications processor of claim 1 further comprising a hardware trace unit associated with each processing core and each accelerator module, wherein the hardware trace unit is configured to perform the logging of trace information.
3. The communications processor of claim 1 wherein the packet watchpoint unit performs said determining if the packet received was expected to be received by
reading a packet debug tag associated with the packet header; and
comparing information contained in the packet debug tag with expected packet data provided to the packet watchpoint unit.
4. The communications processor of claim 3 , wherein
the expected packet data is provided to the packet watchpoint unit during configuration of the packet watchpoint unit, and
the configuration is performed prior to the packet being received by the packet processing core or task accelerator module.
5. The communication processor of claim 3 , wherein, if the packet was expected to be received, the packet watchpoint unit is further configured to
update the packet debug tag with information regarding progress of the packet, and
submit the packet to the packet processing core or task accelerator module for stage processing.
6. The communication processor of claim 3 , wherein the packet debug tag comprises information indicative of the packet's path through the communication processor.
7. The communication processor of claim 1 wherein a packet watchpoint unit of the one or more packet watchpoint units is further configured to halt processing by the associated packet processing core or task accelerator module, in response to a determination that the packet received by the associated packet processing module was not expected to be received.
8. The communication processor of claim 7 wherein the packet watchpoint unit of the one or more packet watchpoint units is further configured to halt communication processor processing of packets in the pipeline.
9. The communication processor of claim 1 wherein the packet watchpoint unit is further configured to transfer information related to the packet to a debug analysis tool, in response to the determination that the packet received by the associated packet processing core or task accelerator module was not expected to be received.
10. A method for performing packet debug in a communication processor, the method comprising:
reading a packet debug tag, wherein
said reading is performed by a packet watchpoint unit associated with a stage of a packet processing pipeline in the communication processor, and
the packet debug tag is a portion of a header of a data packet being processed by the communication processor;
determining if the packet debug tag was expected by the stage;
if the packet debug tag was expected, then updating the packet debug tag with information related to the stage of the packet processing pipeline and logging packet information in a trace log; and
if the packet debug tag was not expected, then
logging the packet as lost in the trace log,
halting logging of the trace log, and
passing trace information and packet debug tag information to an analysis tool.
11. The method of claim 10 wherein said updating the packet debug tag further comprises:
processing packet information using an action function defined prior to processing the packet debug tag; and
revising the packet debug tag to include results of the action function.
12. The method of claim 11 wherein the action function uses information from the packet debug tag as received by the stage of the packet processing pipeline to generate the revised packet debug tag.
13. The method of claim 10 further comprising:
if the packet debug tag was not expected, then halting processing by the packet processing pipeline, and passing processor status information to the analysis tool.
14. The method of claim 10 further comprising:
if the packet debug tag was not expected, then halting processing by the stage of the packet processing pipeline, and passing stage status information to the analysis tool.
15. The method of claim 10 wherein the packet processing pipeline comprises one or more packet processing cores, each configured to provide one or more stages of the packet processing pipeline.
16. The method of claim 15 wherein the packet processing pipeline further comprises one or more task accelerator modules, each configured to provide one or more stages of the packet processing pipeline.
17. The method of claim 10 further comprising:
providing an expected set of stages in the packet processing pipeline;
determining a configuration of a packet watchpoint unit associated with each stage of the packet processing pipeline, wherein the configuration is determined in part by the expected set of stages and in part by a set of information expected from the packet watchpoint unit; and
configuring each packet watchpoint unit in accord with the determined configuration associated with each packet watchpoint unit.
18. The method of claim 17 wherein said determining if the packet debug tag was expected by the stage further comprises:
comparing information stored in the packet debug tag with information provided during said configuring each packet watchpoint unit.
19. The method of claim 10 further comprising, if the packet was not expected:
providing a system debug tool for analyzing register status of the communication processor and for analyzing the packet;
collecting trace log information;
analyzing the trace log information for entries associated with the unexpected packet; and
displaying processing history of the unexpected packet.