Recovering from failures without impact on data traffic in a shared bus architecture
View Patent ↗Methods of detecting and recovering from communication failures within an operating network switching device that is switching packets in a communication network, and associated structures. The communication failures addressed involve communications between the packet processors and a host CPU over a shared communications bus, e.g., PCI bus. The affected packet processor(s)—which may be all or a subset of the packet processors of the network switch—may be recovered without affecting hardware packet forwarding through the affected packet processors. This maximizes the up time of the network switching device. Other packet processor(s), if any, of the network switching device, which are not affected by the communication failure, may continue their normal packet forwarding, i.e., hardware forwarding that does not involve communications with the host CPU as well as forwarding or other operations that do involve communications with the host CPU.
1. A method in a network device, the method comprising:
storing, in a memory associated with a host processor of the network device, a set of data structures used for transferring data, on a shared bus, between the host processor and a plurality of packet processors of the network device, each packet processor in the plurality of packet processors configured to forward, from the network device, one or more packets received by the network device;
detecting, by the host processor, an error condition indicative of a communication error between the host processor and a first packet processor from the plurality of packet processors;
in response to detection of the error condition:
identifying, by the host processor, from the plurality of packet processors, the first packet processor affected by the error condition; and
performing, by the host processor, a set of recovery actions for recovering from the error condition, the set of recovery actions including disabling communication between the host processor and the first packet processor; and
while the set of recovery actions is being performed, communicating data on the shared bus, between the host processor and at least one packet processor from the plurality of packet processors other than the first packet processor and forwarding, by the first packet processor at least one packet received by the network device using forwarding information programmed prior to the host processor detecting the error condition.
2. The method of claim 1 wherein the shared bus is a shared PCI (peripheral component interconnect) bus.
3. The method of claim 1 wherein the detecting comprises detecting setting of one or more error bits due to a parity error detected on the shared bus.
4. The method of claim 1 wherein the detecting comprises generating an interrupt to the host processor.
5. The method of claim 1 wherein the detecting comprises polling a register.
6. The method of claim 5 wherein the polling is programmed to occur periodically.
7. The method of claim 5 wherein the polling is programmed to occur depending on occurrence of a hotswap.
8. The method of claim 1 wherein the error condition is a timeout.
9. The method of claim 1 wherein the forwarding, by the first packet processor, of the at least one packet includes forwarding the at least one packet to a second packet processor in the plurality of packet processors or forwarding the at least one packet to an output port of the network device.
10. The method of claim 1 wherein the set of recovery actions further comprises restoring, by the host processor, to an initial state a data structure from the set of data structures used for communication, on the shared bus, between the host processor and the first packet processor.
11. The method of claim 10 wherein restoring the data structure includes resetting data values of the data structure with initial data values.
12. The method of claim 1 wherein the set of recovery actions further comprises re-enabling communication between the host processor and the first packet processor.
13. A network device comprising:
a control processor with an associated memory; and
a plurality of packet processors, each packet processor in the plurality of packet processors configured to forward, from the network device, one or more packets received by the network device;
the control processor configured to:
store, in the associated memory, a set of data structures used for transferring data, on a shared bus, between the control processor and the plurality of packet processors;
in response to detection of an error condition relating to communication between the control processor and a first packet processor from the plurality of packet processors, identify from the plurality of packet processors, the first packet processor affected by the error condition and perform a set of recovery actions for recovering from the error condition, the set of recovery actions including disabling communication with the first packet processor;
the first packet processor from the plurality of packet processors configured to:
while the set of recovery actions is being performed by the control processor, forward at least one packet received by the network device using forwarding information programmed prior to the host processor detecting the error condition; and
a second packet processor from the plurality of packet processors configured to:
while the set of recovery actions is being performed by the control processor, continue to communicate on the shared bus with the control processor.
14. The device of claim 13 wherein the shared bus is a shared PCI (peripheral component interconnect) bus.
15. The device of claim 13 wherein the control processor is further configured to detect the error condition by detecting setting of one or more error bits due to a parity error detected on the shared bus.
16. The device of claim 13 wherein the control processor is further configured to detect the error condition by polling a register.
17. The device of claim 16 wherein the polling is programmed to occur periodically.
18. The device of claim 16 wherein the polling is programmed to occur depending on occurrence of a hotswap.
19. The device of claim 13 wherein the error condition is a timeout.
20. The device of claim 13 wherein the first packet processor is configured to forward the at least one data packet by forwarding the at least one data packet to another packet processor in the plurality of packet processors or forwarding the at least one data packet to an output port of the network device.
21. The device of claim 13 wherein:
the first packet processor includes at least one data register; and
the control processor is further configured to store initialization data to the at least one data register in performing the set of recovery actions.
22. The device of claim 13 wherein the control processor is further configured to re-enable communication with the first packet processor in performing the set of recovery actions.