Method to detect boot failure of peer compute node by determining if watchdog timers exceeds threshold of normal boot operation
In part, in one aspect, the disclosure relates to a method of monitoring an active compute node in a fault tolerant system. The method may include monitoring a boot process of a first compute node by interrogating one or more watchdog timers running on the first compute node during the boot process; determining a boot process outcome for the first compute node, wherein the boot process outcome is a failure if the one or more of the watchdog timers exceeds a threshold time indicative of normal boot operation; performing a first action if the outcome is a failure; and performing a second action if the outcome is not a failure.
1 . A method of monitoring an active compute node in a fault tolerant system comprising:
monitoring a boot process of a first compute node, using a second compute node of the fault tolerant system, by interrogating one or more watchdog timers running on the first compute node during the boot process, wherein the first compute node is the active compute node;
determining a boot process outcome for the first compute node, wherein the boot process outcome is a failure if the one or more of the watchdog timers exceeds a threshold time indicative of normal boot operation;
performing a first action if the outcome is a failure; and
performing a second action if the outcome is not a failure, wherein the second action is initiating a migration process such that the second compute node can take over for the first compute node with the second compute node becoming a new active compute node.
2 . The method of claim 1 , wherein determining the boot process outcome is performed using the second compute node of the fault tolerant system.
3 . The method of claim 2 , wherein the first compute node and the second compute node are connected using a communication interface, wherein watchdog timer values are received at the second compute node from the first compute node using the communication interface.
4 . The method of claim 3 , wherein the communication interface is an Intelligent Platform Management Interface (IPMI), wherein the first compute node comprises a first BMC, wherein the second compute node comprises a second BMC, wherein the second BMC performs the interrogation of the one or more watchdog times.
5 . The method of claim 1 , wherein the first action is rebooting the first compute node.
6 . The method of claim 1 , wherein the first action is generating an alert or a log indicative of a problem with the first computer node.
7 . The method of claim 1 , wherein the second action is reprovisioning a plurality of peripheral devices from the first compute node to the second compute node.
8 . The method of claim 1 , wherein the boot process comprises a Unified Extensible Firmware Interface (UEFI) control phase and an operating system (OS) control phase.
9 . The method of claim 8 , wherein the one or more watchdog timers, comprises a first watchdog timer and a second watchdog timer that are started during the UEFI control phase of the boot process.
10 . The method of claim 9 , wherein the one or more watchdog timers further comprises a third watchdog timer that is started during the OS control phase of the boot process.
11 . The method of claim 10 , wherein in response to all of the first, second, and third watchdog timers not exceeding their respective threshold values, the second action is for the first compute node to remain the active node.
12 . The system of claim 1 , wherein the fault tolerant system comprises the first computer node and the second compute node.
13 . A fault tolerant computer system comprising:
a first compute node comprising a Fault Resilient Boot (FRB) hardware device, wherein the FRB hardware device is configured to generate a set of watchdog timers, wherein each watchdog timer in the set corresponds to a phase or an event in a boot process;
a communications interface;
a management processor; and
a second compute node, wherein the second compute node is in communication with the FRB hardware device using the communications interface,
wherein the second compute node comprises the management processor,
the management processor configured to read each watchdog timer in the set of watchdog timers,
the management processor configured to determine if each watchdog timer exceeds a threshold time for its respective phase of the boot process, wherein not exceeding the threshold time for each respective timer corresponds to the normal operation of the compute node during the corresponding phase of the boot process.
14 . The system of claim 13 , wherein the communications interface is an intelligent platform management bus or an IPMI.
15 . The system of claim 14 , wherein the management processor is a BMC.
16 . The system of claim 15 , wherein the second compute node is configured to take control of the first compute node in response to one or more of the threshold times for each respective watchdog timer being exceeded.
17 . The system of claim 13 wherein the second compute node is configured to reboot the first compute node in response to one or more of the threshold times for each respective watchdog timer being exceeded.
18 . The system of claim 13 wherein the second compute node is configured to initiate a transfer of processor state and memory information from the first compute node to the second compute node in response to one or more of the threshold times for each respective watchdog timer being exceeded.
19 . The system of claim 13 wherein the second compute node is configured to generate a report or an error log in response to one or more of the threshold times for each respective watchdog timer being exceeded.