System and method for remote system recovery

1. A computer-implemented method for resetting a hung-up component of a server system, comprising:

receiving, by a hardware watch-dog-timer (HW WDT) circuit of the server system, heartbeat signals from a baseboard management controller (BMC) of the server system;

determining, by the HW WDT circuit, a health condition of the BMC based upon the heartbeat signals;

resetting the BMC if the health condition of the BMC fails to meet a predefined criterion;

collecting, by a first complex programmable logic device (CPLD), health information of components of the server system from the BMC;

determining that a hardware component of the server system other than the BMC is hung-up, based upon (i) the collected health information of the components of the server system, and (ii) the health condition of the BMC, the hardware component of the server system other than the BMC including (i) a processor, (ii) north bridge logic, (iii) south bridge logic, (iv) one or more peripheral component interconnect express (PCIe) slots, (v) one or more peripheral component interconnect (PCI) slots, (vi) one or more industry standard architecture (ISA) slots; (vii) a storage device, (viii) a physical layer integrated circuit (PHY IC), (ix) the HW WDT circuit, or (x) any combination of (i)-(ix);

sending, by the first CPLD, a reset signal to reset the hardware component of the server system that is determined to be hung-up;

monitoring, by the CPLD, the reset signal sent to the hardware component of the server system that is determined to be hung-up; and

in response to determining that the reset signal has failed to reset the hardware component of the server system that is determined to be hung-up, resending the reset signal.

2. The computer-implemented method of claim 1 , further comprising: upon receiving a management message from an administrator or a remote device, communicating, by a physical layer (PHY) integrated-circuit (IC), with the first CPLD via a power management event (PME) signal to cause the CPLD to collect the health information of the components of the server system from the BMC; and collecting, by the first CPLD, the health condition of the BMC from the HW WDT circuit.

3. The computer-implemented method of claim 2 , wherein the first CPLD is connected to the BMC via an inter-integrated circuit (I 2 C) channel.

4. The computer-implemented method of claim 2 , wherein the BMC is connected to the PHY IC via a communication channel that enables the BMC to control the PHY IC, the communication channel being one of media-independent interface (MII), reduced media-independent interface (RMII), gigabit media-independent interface (GMII), reduced gigabit media-independent interface (RGMII), 10-gigabit media-independent interface (XGMII) and serial gigabit media-independent interface (SGMII).

5. The computer-implemented method of claim 2 , wherein the server system further comprises a second CPLD, the second CPLD configured to server as a redundant backup of the first CPLD, and/or directly collect the health information of the components of the server system.

6. The computer-implemented method of claim 5 , wherein the second CPLD comprises a second time-out detection module.

7. The computer-implemented method of claim 1 , further comprising:

causing the BMC to scan the components of the server system, and collect health information of the components of the server system.

8. The computer-implemented method of claim 1 , wherein the BMC comprises a software WDT resettable by the heartbeat signals of the BMC, the software WDT configured to generate a timeout signal to reset the BMC in an event that the heartbeat signals of the BMC fail to reset the software WDT.

9. A system, comprising:

a processor;

a baseboard management controller (BMC)

a hardware watch-dog-timer (HW WDT) circuit;

a physical layer (PHY) integrated-circuit (IC);

a first complex programmable logic device (CPLD); and

a computer-readable medium storing instructions that, when executed by the processor, cause the system to perform operations comprising:

receiving, by the HW WDT circuit, heartbeat signals from the BMC;

determining, by the HW WDT circuit, a health condition of the BMC based upon the heartbeat signals;

resetting the BMC if the health condition of the BMC fails to meet a predefined criterion;

collecting, by the first CPLD, health information of components of the server system from the BMC;

determining that a hardware component of the server system other than the BMC is hung-up, based upon (i) the collected health information of the components of the server system, and (ii) the health condition of the BMC, the hardware component of the server system other than the BMC including (i) a processor, (ii) north bridge logic, (iii) south bridge logic, (iv) one or more peripheral component interconnect express (PCIe) slots, (v) one or more peripheral component interconnect (PCI) slots, (vi) one or more industry standard architecture (ISA) slots; (vii) a storage device, (viii) a physical layer integrated circuit (PHY IC), (ix) the HW WDT circuit, or (x) any combination of (i)-(ix);

sending, by the first CPLD, a reset signal to reset the hardware component of the server system that is determined to be hung-up;

monitoring, by the CPLD, the reset signal sent to the hardware component of the server system that is determined to be hung-up; and

in response to determining that the reset signal has failed to reset the hardware component of the server system that is determined to be hung-up, resending the reset signal.

10. The system of claim 9 , wherein the computer-readable medium stores instructions that, when executed by the processor, further cause the system to perform operations comprising:

upon receiving a management message from an administrator or a remote device, communicating, by the PHY IC, with the first CPLD via a power management event (PME) signal to cause the first CPLD to collect the health information of the components of the server system from the BMC; and

collecting, by the first CPLD, the health condition of the BMC from the HW WDT circuit.

11. The system of claim 10 , wherein the first CPLD is connected to the BMC via an inter-integrated circuit (I 2 C) channel.

12. The system of claim 10 , wherein the BMC is connected to the PHY IC via a communication channel that enables the BMC to control the PHY IC, the communication channel being one of media-independent interface (MII), reduced media-independent interface (RMII), gigabit media-independent interface (GMII), reduced gigabit media-independent interface (RGMII), 10-gigabit media-independent interface (XGMII) and serial gigabit media-independent interface (SGMII).

13. The system of claim 9 , wherein the server system further comprises a second CPLD, the second CPLD configured to server as a redundant backup of the first CPLD, and/or directly collect the health information of the components of the server system.

14. The system of claim 13 , wherein the second CPLD comprises a second time-out detection module.

15. The system of claim 9 , wherein the BMC comprises a software WDT resettable by the heartbeat signals of the BMC, the software WDT configured to generate a timeout signal to reset the BMC in an event that the heartbeat signals of the BMC fail to reset the software WDT.

16. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a server system, cause the server system to perform operations comprising:

receiving, by a hardware watch-dog-timer (HW WDT) circuit of the server system, heartbeat signals from a baseboard management controller (BMC) of the server system;

determining, by the HW WDT circuit, a health condition of the BMC based upon the heartbeat signals;

resetting the BMC if the health condition of the BMC fails to meet a predefined criterion;

upon receiving a management message from an administrator or a remote device, communicating, by a physical layer (PHY) integrated-circuit (IC), with a first complex programmable logic device (CPLD) via a power management event (PME) signal;

collecting, by the first CPLD, health information of components of the server system from the BMC;

collecting, by the first CPLD, the health condition of the BMC from the HW WDT circuit;

determining that a hardware component of the server system other than the BMC is hung-up, based upon (i) the collected health information of the components of the server system, and (ii) the health condition of the BMC, the hardware component of the server system other than the BMC including (i) a processor, (ii) north bridge logic, (iii) south bridge logic, (iv) one or more peripheral component interconnect express (PCIe) slots, (v) one or more peripheral component interconnect (PCI) slots, (vi) one or more industry standard architecture (ISA) slots; (vii) a storage device, (viii) a physical layer integrated circuit (PHY IC), (ix) the HW WDT circuit, or (x) any combination of (i)-(ix);

sending, by the first CPLD, a reset signal to reset the hardware component of the server system that is determined to be hung-up;

monitoring, by the CPLD, the reset signal sent to the hardware component of the server system that is determined to be hung-up; and

in response to determining that the reset signal has failed to reset the hardware component of the server system that is determined to be hung-up, resending the reset signal.