Recovering from failures without impact on data traffic in a shared bus architecture
View Patent ↗Methods of detecting and recovering from communication failures within an operating network switching device that is switching packets in a communication network, and associated structures. The communication failures addressed involve communications between the packet processors and a host CPU over a shared communications bus, e.g., PCI bus. The affected packet processor(s)—which may be all or a subset of the packet processors of the network switch—may be recovered without affecting hardware packet forwarding through the affected packet processors. This maximizes the up time of the network switching device. Other packet processor(s), if any, of the network switching device, which are not affected by the communication failure, may continue their normal packet forwarding, i.e., hardware forwarding that does not involve communications with the host CPU as well as forwarding or other operations that do involve communications with the host CPU.
1. A method for error recovery in a network device, the method comprising:
storing, in a memory associated with a host processor of the network device, a set of data structures used for facilitating communication between the host processor and a plurality of packet processors of the network device;
forwarding, by a first packet processor from the plurality of packet processors, one or more packets received by the network device using forwarding information programmed by the host processor into a memory accessed by the first packet processor;
detecting, by the host processor, an error condition indicative of a communication error between the host processor and a packet processor from the plurality of packet processors;
in response to detection of the error condition:
identifying, by the host processor, from the plurality of packet processors, the first packet processor affected by the error condition; and
performing, by the host processor, a set of recovery actions for recovering from the error condition, the performing comprising
disabling communication between the host processor and the first packet processor, and
restoring, by the host processor, to an initial state a data structure from the set of data structures used for communication between the host processor and the first packet processor; and
while the set of actions is being performed, forwarding, by the first packet processor, at least one packet received by the network device using the forwarding information programmed into the memory accessed by the first packet processor prior to detecting the error condition.
2. The method of claim 1 further comprising
while the set of actions is being performed, communicating data between the host processor and at least one packet processor from the plurality of packet processors other than the first packet processor.
3. The method of claim 1 further comprising resetting a portion of the memory associated with the host processor with initialization data.
4. The method of claim 1 wherein the host processor and the plurality of packet processors communicate using a shared PCI (peripheral component interconnect) bus architecture.
5. The method of claim 1 wherein restoring the data structure includes resetting data values of said data structure with initial data values.
6. The method of claim 1 wherein the forwarding by the first packet processor of the at least one packet includes receiving the at least one packet at a port of the first packet processor and forwarding the received at least one packet to a second packet processor from the plurality of packet processors.
7. The method of claim 1 wherein the forwarding by the first packet processor of the at least one packet includes forwarding the at least one packet to an output port of the network device.
8. The method of claim 1 wherein the detecting comprises detecting setting of one or more error bits due to a parity error detected on a shared bus used for communicating between the host processor and the plurality of packet processors.
9. The method of claim 1 , wherein the detecting comprises generating an interrupt to the host processor.
10. The method of claim 1 , wherein the detecting comprises polling a register.
11. The method of claim 10 , wherein said polling is programmed to occur periodically.
12. The method of claim 10 , wherein said polling is programmed to occur depending on occurrence of a hotswap.
13. The method of claim 1 , wherein the error condition is a timeout.
14. A network switching device comprising:
a control processor with an associated memory; and
a plurality of packet processors, including a first packet processor configured to forward one or more data packets received by the network switching device using forwarding information programmed by the control processor into a memory accessed by the first packet processor;
said control processor configured to:
store, in the associated memory, a set of data structures used for facilitating communication between the control processor and the plurality of packet processors;
in response to detection of an error condition relating to communication between said control processor and one or more of said plurality of packet processors, identify the first packet processor affected by the error condition and perform a set of recovery actions for recovering from the error condition, the performing including disabling communication activity with the first packet processor, and restoring to an initial state a data structure from the set of data structures used for facilitating communication between the control processor and the first packet processor;
said first packet processor configured to:
while the set of actions is being performed by the control processor, forward at least one data packet received by the network switching device using the forwarding information programmed into the memory accessed by the first packet processor prior to detection of the error condition.
15. The device of claim 14 further comprising a shared bus configured to facilitate communications between the control processor and the plurality of packet processors.
16. The device of claim 15 wherein said shared bus is a PCI bus.
17. The device of claim 14 wherein the set of recovery actions comprises
re-enabling communication activity with the first packet processor.
18. The device of claim 17 wherein said identified first packet processor includes at least one data register, wherein said control processor is further configured to store initialization data to said at least one data register in performing the set of recovery actions.