Method and system for intelligent distributed health monitoring in switching system equipment
A method for distributed health monitoring and fault repairing in a switching system. The switching system having one or more supervisory cards, one or more line cards, and one or more switch fabric cards. The method includes transmitting a health status poll request message to the one or more line cards and the one or more switch fabric cards. Thereafter, the method includes receiving health status poll response messages from each of the one or more line cards and the one or more switch fabric cards. Each health status poll response message includes health status summary of the corresponding card. Further, the method involves detecting one or more faults in the switching system based on the health poll response messages. Finally, the method includes triggering at least one action on the detection of the faults in the switching system. These actions are triggered based on a set of predefined rules.
1. A method for distributed health monitoring and fault repairing in switching system equipment, the switching system equipment having a supervisory card, a line card, and a switch fabric card, wherein the supervisory card is interconnected with the line card and the switch fabric card through a backplane, each of the supervisory card, the line card, and the switch fabric card having corresponding control processors capable of running embedded operating systems, the method comprising:
transmitting a health status poll request message to the line card and the switch fabric card from the supervisory card, wherein the supervisory card is separate from the line card and the switch fabric card and communicates with each over the backplane for the transmitting;
receiving a health status poll response message by the supervisory card from each of the line card and the switch fabric card in response to the health status poll request message, wherein the health status poll response message comprises a health status summary report of hardware and software modules of one of the corresponding line card and switch fabric card;
discovering a relationship between all cards in the switching system based on the health status poll response message to maintain an up-to-date topology of the cards for the health monitoring;
detecting a fault in the switching system equipment by the supervisory card based on an analysis of the health poll response message; and
automatically triggering an action to repair the fault detected in the switching system equipment, wherein the action is triggered based on a set of predefined policies, wherein the action comprises any one or more of re-running a set of diagnostic tests, resetting the control processors of the cards, restarting the switching system equipment, triggering a self-repair through restart of the cards, and triggering a switchover of a first supervisory card to a second supervisory card already operating in the switching system equipment.
2. The method as recited in claim 1 , wherein the health status poll request message is transmitted periodically by a master health monitor module running on the supervisory card.
3. The method as recited in claim 2 , wherein the fault in the switching system equipment is detected by the master health monitor module based on an analysis of the health poll response message.
4. The method as recited in claim 1 , wherein the health status poll response message corresponding to the line card is generated by a slave health monitor module running on the line card.
5. The method as recited in claim 4 , wherein the fault in the switching system equipment is detected by the slave health monitor module.
6. The method as recited in claim 1 , wherein the health status poll response message corresponding to the switch fabric card is generated by a slave health monitor module running on the switch fabric card.
7. The method as recited in claim 6 , wherein the fault in the switching system equipment is detected by the slave health monitor module.
8. The method as recited in claim 1 , wherein the health status poll response message includes performance statistics information of the hardware and software modules on one of the line card and switch fabric card.
9. The method as recited in claim 8 , wherein the performance statistics information comprises a transmitted and received packet count, a transmitted and received byte count, a packet drop count of traffic through internal and external physical and logical ports.
10. A method for distributed health monitoring and fault repairing in switching system equipment, the switching system equipment having at least one supervisory card, one or more line cards, and one or more switch fabric cards, wherein the at least one supervisory card is interconnected with the one or more line cards and the one or more switch fabric cards through a backplane, each of the at least one supervisory card, the one or more line cards, and the one or more switch fabric cards having corresponding control processors capable of running embedded operating systems, the method comprising:
multicasting a master health status summary report message from the at least one supervisory card to the one or more line cards and the one or more switch fabric cards, wherein the at least one supervisory card is separate from the one or more line cards and the one or more switch fabric cards and communicates with each over the backplane for the multicasting;
receiving one or more slave health status summary report messages by the at least one supervisory card from each of the one or more line cards and the one or more switch fabric cards in response to the master health status summary report message, wherein each of the one or more slave health status summary report messages comprise a health status summary of hardware and software modules of the corresponding line card or switch fabric card;
discovering a relationship between all cards in the switching system based on the one or more slave health status summary report messages to maintain an up-to-date topology of the cards for the health monitoring;
detecting one or more faults in the switching system equipment by the at least one supervisory card based on an analysis of the master health status summary report message and the one or more slave health status summary report messages; and
automatically triggering at least one action to repair the one or more faults detected in the switching system equipment, wherein the at least one action is triggered based on a set of predefined policies, wherein the at least one action comprises any one or more of re-running a set of diagnostic tests, resetting the control processors of the cards, restarting the switching system equipment, triggering a self-repair through restart of the cards, and triggering a switchover of a first supervisory card to a second supervisory card already operating in the switching system equipment.
11. The method as recited in claim 10 , wherein the master health status summary report message is transmitted periodically by a master health monitor module running on the at least one supervisory card.
12. The method as recited in claim 11 , wherein the faults in the switching system equipment are detected by the master health monitor module based on the analysis of said slave health status summary report messages and said master health status summary report message.
13. The method as recited in claim 10 , wherein the slave health status summary report message corresponding to a line card of the one or more line cards is generated by a slave health monitor module running on the line card of the one or more line cards.
14. The method as recited in claim 13 , wherein the faults in the switching system equipment are detected by the slave health monitor module based on the analysis of said slave health status summary report messages and said master health status summary report message.
15. The method as recited in claim 10 , wherein the slave health status summary report message corresponding to a switch fabric card of the one or more switch fabric cards is generated by a slave health monitor module running on the switch fabric card of the one or more switch fabric cards.
16. The method as recited in claim 15 , wherein the faults in the switching system equipment are detected by the slave health monitor module based on the analysis of said slave health status summary report messages and said master health status summary report message.
17. The method as recited in claim 10 , wherein said slave health status summary report messages and said master health status summary report message include performance statistics information of the hardware and software modules of the corresponding card.
18. The method as recited in claim 10 , wherein the at least one action is selected from the group comprising re-running a set of diagnostic tests, resetting the control processors of the cards, restarting the switching system equipment, triggering a self-repair through restart of the cards, and triggering a switchover of a first supervisory card to a second supervisory card.
19. A supervisory card for distributed health monitoring and fault repairing in switching system equipment, the supervisory card comprising:
one or more control processors capable of running an embedded operating system;
connections to the switching system equipment comprising a line card and a switch fabric card, wherein the supervisory card is interconnected with the line card and the switch fabric card through a backplane, each of the line cards and the switch fabric card having corresponding control processors capable of running embedded operating systems,
wherein the one or more control processors are configured to
transmit a health status poll request message to the line card and the switch fabric card from the supervisory card, wherein the supervisory card is separate from the line card and the switch fabric card and communicates with each over the backplane for the transmission of the health status poll message;
receive a health status poll response message by the supervisory card from each of the line card and the switch fabric card in response to the health status poll request message, wherein the health status poll response message comprises a health status summary report of hardware and software modules of one of the corresponding line card and switch fabric card;
discover a relationship between all cards in the switching system based on the health status poll response message to maintain an up-to-date topology of the cards for the health monitoring;
detect a fault in the switching system equipment by the supervisory card based on analysis of the health poll response message; and
automatically trigger an action to repair the fault detected in the switching system equipment, wherein the action is triggered based on a set of predefined policies; wherein the action comprises any one or more of re-running a set of diagnostic tests, resetting the control processors of the cards, restarting the switching system equipment; triggering a self-repair through restart of the cards, and triggering a switchover of a first supervisory card to a second supervisory card already operating in the switching system equipment.