System and method for runtime firmware verification, recovery, and repair in an information handling system
A management system for an information handling system includes a memory device and a management controller. The memory device includes a primary firmware image and a backup firmware image. The management controller boots the management system. The boot of the management system includes a read of a first block at an offset within the primary firmware image. The management controller further determines that the read of the first block had a first read error, and reboots the management system in response to determining that the read of the first block had the first read error. The reboot of the management system includes a read of a second block at the offset within the backup firmware image.
1. A management system for an information handling system, the management system comprising:
a memory device including a primary firmware image and a backup firmware image; and
a baseboard management controller configured to:
read a first block at an offset within the primary firmware image;
determine that the read of the first block had a first read error;
determine that the backup firmware image is a duplicate copy of the primary firmware image;
read a second block at the offset within the backup firmware image in response to determining that the backup firmware image is a duplicate copy of the primary firmware image;
write the second block at the offset within the primary firmware image; and
reboot the management system in response to writing the second block at the offset within the primary firmware image, wherein the reboot of the management system includes a read of the second block at the offset within the primary firmware image.
2. The management system of claim 1 , wherein the baseboard management controller is further configured to:
determine that the primary firmware image is a current firmware image and the backup firmware image is a previous firmware image;
reboot the management system in response to determining that the primary firmware image is a current firmware image and the backup firmware image is a previous firmware image, wherein the reboot of the management system includes a read of the second block at the offset within the backup firmware image; and
provide an indication to a management computer that the management system is utilizing the backup firmware image in response to rebooting the management system.
3. The management system of claim 1 , wherein, in determining that the read of the first block had the first read error, the baseboard management controller is further configured to verify contents of the first block based upon a hash associated with the first block.
4. The management system of claim 1 , wherein:
the baseboard management controller includes a setting to direct a boot of the management system from one of the primary firmware image and the backup firmware image; and
the baseboard management controller is further configured to change the setting from a first state to a second state in response to determining that the read of the first block had the first read error, wherein rebooting the management system is in further response to changing the setting.
5. The management controller of claim 1 , wherein:
the baseboard management controller includes a non-volatile memory component; and
the baseboard management controller is further configured to write information to the non-volatile memory component that identifies the first block in response to determining that the read of the first block had the first read error.
6. The management system of claim 5 , wherein the write information includes the offset.
7. The management system of claim 1 , wherein:
the first block includes an error-correcting code portion;
the baseboard management controller is further configured to determine that the first read error is an uncorrectable read error based upon the error-correcting portion, wherein rebooting is in further response to determining that the first read error is the uncorrectable read error.
8. The management system of claim 7 , wherein the baseboard management controller is further configured to:
determine that the first read error is a correctable read error based upon the error-correcting portion;
correct the first block based upon the error-correcting portion in response to determining that the first read error is the correctable read error; and
write the corrected first block at the offset within the primary firmware image.
9. The management controller of claim 1 , wherein, after a successful boot process of the management system, the baseboard management controller is further configured to:
periodically read successive blocks of the primary firmware image;
determine that a read of a particular block had a second read error; and
write information to a non-volatile memory component of the management controller, wherein the information identifies the particular block in response to determining that the read of the particular block had the second read error.
10. A method for managing firmware for a management system of an information handling system, the method comprising:
providing, in a memory device of the management system, a primary firmware image and a backup firmware image;
reading, by a management controller, a first block at an offset within the primary firmware image;
determining that the read of the first block had a first read error;
determining that the backup firmware image is a duplicate copy of the primary firmware image;
reading a second block at the offset within the backup firmware image in response to determining that the backup firmware image is a duplicate copy of the primary firmware image;
writing the second block at the offset within the primary firmware image; and
rebooting the management system in response to writing the second block at the offset within the primary firmware image, wherein the reboot of the management system includes the read of a second block at the offset within the primary firmware image.
11. The method of claim 10 , wherein the method further comprises:
determining that the primary firmware image is a current firmware image and the backup firmware image is a previous firmware image;
rebooting the management system in response to determining that the primary firmware image is a current firmware image and the backup firmware image is a previous firmware image, wherein the reboot of the management system includes a read of the second block at the offset within the backup firmware image; and
providing, by the management controller, an indication to a management computer that the management system is utilizing the backup firmware image in response to rebooting the management system.
12. The method of claim 10 , wherein, in determining that the read of the first block had the first read error, the method further comprises:
verifying, by the management controller, contents of the first block based upon a hash associated with the first block.
13. The method of claim 10 , wherein the management controller includes a setting to direct a boot of the management system from one of the primary firmware image and the backup firmware image, the method further comprising:
changing, by the management controller, the setting from a first state to a second state in response to determining that the read of the first block had the first read error, wherein rebooting the management system is in further response to changing the setting.
14. The method of claim 10 , wherein the management controller includes a non-volatile memory component, the method further comprising:
writing, by the management controller, information to the non-volatile memory component that identifies the first block in response to determining that the read of the first block had the first read error, wherein the write information includes the offset.
15. The method of claim 10 , wherein the first block includes an error-correcting code portion, the method further comprising:
determining, by the management controller, that the first read error is an uncorrectable read error based upon the error-correcting portion, wherein rebooting is in further response to determining that the first read error is the uncorrectable read error.
16. The method of claim 15 , further comprising:
determining, by the management controller, that the first read error is a correctable read error based upon the error-correcting portion;
correcting, by the management controller, the first block based upon the error-correcting portion in response to determining that the first read error is the correctable read error; and
writing, by the management controller, the corrected first block at the offset within the primary firmware image.
17. The method of claim 10 , wherein, after a successful boot process of the management system, the method further comprises:
periodically reading, by the management controller, successive blocks of the primary firmware image;
determining, by the management controller, that a read of a particular block had a second read error; and
writing, by the management controller, information to a non-volatile memory component of the management controller, wherein the information identifies the particular block in response to determining that the read of the particular block had the second read error.
18. An information handling system, comprising:
a host system having a processor to execute host firmware; and
a management system including:
a memory device including a primary firmware image and a backup firmware image; and
a baseboard management controller configured to:
boot the management system, wherein the boot of the management system includes a read of a first block at an offset within the primary firmware image, wherein the first block includes an error-correcting portion;
determine that the read of the first block had a first read error; and
write the corrected first block at the offset within the primary firmware image;
wherein, after a successful boot process of the management system, the baseboard management controller is further configured to:
periodically read successive blocks of the primary firmware image;
determine that a read of a particular block had a second read error; and
write information to a non-volatile memory component of the management controller, wherein the information identifies the particular block in response to determining that the read of the particular block had the second read error.