System and method for recovering an operating system after an upgrade hang using a dual-flash device
One embodiment provides a computer implemented method for recovering an operating system (OS) after an upgrade hang using a dual-flash device. The method includes detecting a system hang during an OS upgrade; initiating a boot from the dual-flash device; checking for OS configuration data in the dual-flash device; creating a hook in OS boot scripts to recover the OS configuration data after OS boot; and applying the OS configuration data after OS boot.
1. A computer implemented method for recovering an operating system (OS) after an upgrade hang using a dual-flash device, the method comprising:
detecting the upgrade hang during performance of an OS upgrade by an OS upgrade module of the OS in connection with a boot drive and the dual-flash device, the OS upgrade comprising upgrading a recent version of the OS into an upgraded OS on the boot drive;
initiating a boot from the dual-flash device in response to the detected upgrade hang;
checking for OS configuration data in the dual-flash device including comparing recent OS files in the dual-flash device against a running OS version to determine whether is a mismatch, wherein when there is a mismatch between the recent OS files in the dual-flash device and the running OS version, the running OS version is copied to the dual-flash device and the recent OS files are deleted;
creating a hook in OS boot scripts to recover the OS configuration data in the dual-flash device after OS boot, wherein the OS configuration data comprise a recent version of the OS before performing the OS upgrade, the recent version of the OS different from a factory OS and the OS configuration data different from original factory settings; and
applying the OS configuration data after OS boot.
2. The computer implemented method of claim 1 , further comprising:
saving the OS configuration data into the dual-flash device before initiating the OS upgrade.
3. The computer implemented method of claim 1 , further comprising:
performing a subsequent OS upgrade after OS boot and after applying the OS configuration data.
4. The computer implemented method of claim 1 , wherein the system hang is detected during the OS upgrade by a watchdog timer within a baseboard management controller (BMC).
5. The computer implemented method of claim 4 , wherein the watchdog timer expires after failing to receive a heartbeat signal within one minute.
6. The computer implemented method of claim 5 , wherein a dual-flash monitor module is configured to relay the heartbeat signal from an OS upgrade module to the BMC.
7. A system for recovering an operating system (OS) after an upgrade hang using a dual-flash device, comprising:
a server;
a boot drive;
a baseboard management controller (BMC) configured to detect a system hang during an upgrade of an OS of the server by an OS upgrade module of the OS in connection with the boot drive and the dual-flash device, the upgrade of the OS comprising upgrading a recent version of the OS of the server into an upgraded OS on the boot drive;
the dual-flash device configured to initiate a boot upon detecting a system hang during the OS upgrade;
a dual-flash monitor module configured to
check for OS configuration data in the dual-flash device including comparing recent OS files in the dual-flash device against a running OS version to determine whether is a mismatch, wherein when there is a mismatch between the recent OS files in the dual-flash device and the running OS version, the running OS version is copied to the dual-flash device and the recent OS files are deleted, and
create a hook in OS boot scripts to recover the OS configuration data in the dual-flash device after OS boot, wherein the OS configuration data comprise the recent version of the OS before the upgrade of the OS, the recent version of the OS different from a factory OS and the OS configuration data different from original factory settings; and
an OS boot module configured to apply the OS configuration data after the boot from the dual-flash device.
8. The system of claim 7 , wherein the dual-flash monitor module is further configured to save the OS configuration data into the dual-flash device before the OS upgrade initiates.
9. The system of claim 7 , further comprising:
an OS upgrade module configured to perform a subsequent OS upgrade after OS boot and after the OS boot module applies the OS configuration data.
10. The system of claim 9 , wherein the BMC includes a watchdog timer configured to detect the system hang during the OS upgrade.
11. The system of claim 10 , wherein the watchdog timer is configured to expire after failing to receive a heartbeat signal within one minute.
12. The system of claim 11 , wherein a dual-flash monitor module is configured to relay the heartbeat signal from the OS upgrade module to the BMC.
13. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform an upgrade hang operating system (OS) recovery operation using a dual-flash device, the operation comprising:
detecting a system hang during an OS upgrade by an OS upgrade module of the OS in connection with a boot drive and the dual-flash device, the OS upgrade comprising upgrading a recent version of the OS into an upgraded OS on the boot drive;
initiating a boot from the dual-flash device in response to the detected system hang;
checking for OS configuration data in the dual-flash device including comparing recent OS files in the dual-flash device against a running OS version to determine whether is a mismatch, wherein when there is a mismatch between the recent OS files in the dual-flash device and the running OS version, the running OS version is copied to the dual-flash device and the recent OS files are deleted;
creating a hook in OS boot scripts to recover the OS configuration data in the dual-flash device after OS boot, wherein the OS configuration data comprise a recent version of the OS before performing the OS upgrade, the recent version of the OS different from a factory OS and the OS configuration data different from original factory settings; and
applying the OS configuration data after OS boot.
14. The non-transitory computer-readable medium of claim 13 , wherein the operation further comprises:
saving the OS configuration data into the dual-flash device before initiating the OS upgrade.
15. The non-transitory computer-readable medium of claim 13 , wherein the operation further comprises:
performing a subsequent OS upgrade after OS boot and after applying the OS configuration data.
16. The non-transitory computer-readable medium of claim 13 , wherein the system hang is detected during the OS upgrade by a watchdog timer within a baseboard management controller (BMC).
17. The non-transitory computer-readable medium of claim 16 , wherein the watchdog timer expires after failing to receive a heartbeat signal within one minute.
18. The non-transitory computer-readable medium of claim 17 , wherein a dual-flash monitor module is configured to relay the heartbeat signal from an OS upgrade module to the BMC.