Managing unrecoverable errors for data processing systems using out of band communication channels
Methods and systems for managing unrecoverable errors for data processing systems are disclosed. Upon identifying that an unrecoverable error has occurred by hardware resources of a data processing system, a management controller for the data processing system may obtain a data package indicating at least an operating state of the hardware resources prior to the occurrence of the unrecoverable error. The management controller may utilize an out of band communication channel to provide the data package to a trusted management system. The management controller may obtain at least one action in response to the data package from the trusted management system, the at least one action being intended to initiate remediation of the unrecoverable error.
1. A method of managing data processing systems, the method comprising:
identifying, by a management controller of a data processing system of the data processing systems, an occurrence of an unrecoverable error by hardware resources of the data processing system;
based on the identifying:
obtaining, by the management controller and from the hardware resources, a data package indicating, at least, an operating state of the hardware resources when the occurrence of the unrecoverable error occurred; and
providing, by the management controller and via an out of band communication channel, at least a portion of the data package to a trusted management system to initiate remediation of the unrecoverable error.
2. The method of claim 1 , further comprising:
after providing the at least the portion of the data package:
obtaining, by the management controller and via the out of band communication channel, at least one action for remediation of the unrecoverable error; and
initiating, by the management controller, performance of the at least one action.
3. The method of claim 1 , wherein identifying the occurrence of the unrecoverable error comprises:
identifying, by the management controller, at least one unsuccessful attempt to restart the data processing system after the occurrence of the unrecoverable error.
4. The method of claim 1 , wherein identifying the occurrence of the unrecoverable error comprises:
obtaining, by the management controller, a message from an agent hosted by the hardware resources, the message indicating that a management entity of the hardware resources has suffered the unrecoverable error.
5. The method of claim 4 , wherein the management entity is an operating system running in a virtualized environment hosted by the hardware resources, and the agent is outside of the virtualized environment.
6. The method of claim 1 , wherein the data package comprises:
a snapshot of contents of memory of the hardware resources; and
contents of a processor of the hardware resources.
7. The method of claim 6 , wherein the data package further comprises:
at least one log of activity of the hardware resources prior to the occurrence of the unrecoverable error.
8. The method of claim 1 , wherein the data processing system comprises a network module adapted to separately advertise network endpoints for the management controller and the hardware resources, the network endpoints being usable by the trusted management system to address communications to the hardware resources and the management controller.
9. The method of claim 8 , wherein the management controller and the network module are on separate power domains from the hardware resources so that the management controller and the network module are operable while the hardware resources are inoperable.
10. The method of claim 8 , wherein the out of band communication channel runs through the network module, and an in band communication channel that services the hardware resources also runs through the network module.
11. The method of claim 8 , wherein the management controller hosts a transmission control protocol/internet protocol (TCP/IP) stack to facilitate network communications via the out of band communication channel.
12. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing data processing systems, the operations comprising:
identifying, by a management controller of a data processing system of the data processing systems, an occurrence of an unrecoverable error by hardware resources of the data processing system;
based on the identifying:
obtaining, by the management controller and from the hardware resources, a data package indicating, at least, an operating state of the hardware resources when the occurrence of the unrecoverable error occurred; and
providing, by the management controller and via an out of band communication channel, at least a portion of the data package to a trusted management system to initiate remediation of the unrecoverable error.
13. The non-transitory machine-readable medium of claim 12 , further comprising:
after providing the at least the portion of the data package:
obtaining, by the management controller and via the out of band communication channel, at least one action for remediation of the unrecoverable error; and
initiating, by the management controller, performance of the at least one action.
14. The non-transitory machine-readable medium of claim 12 , wherein identifying the occurrence of the unrecoverable error comprises:
identifying, by the management controller, at least one unsuccessful attempt to restart the data processing system after the occurrence of the unrecoverable error.
15. The non-transitory machine-readable medium of claim 12 , wherein identifying the occurrence of the unrecoverable error comprises:
obtaining, by the management controller, a message from an agent hosted by the hardware resources, the message indicating that a management entity of the hardware resources has suffered the unrecoverable error.
16. The non-transitory machine-readable medium of claim 15 , wherein the management entity is an operating system running in a virtualized environment hosted by the hardware resources, and the agent is outside of the virtualized environment.
17. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing data processing systems, the operations comprising:
identifying, by a management controller of a data processing system of the data processing systems, an occurrence of an unrecoverable error by hardware resources of the data processing system;
based on the identifying:
obtaining, by the management controller and from the hardware resources, a data package indicating, at least, an operating state of the hardware resources when the occurrence of the unrecoverable error occurred; and
providing, by the management controller and via an out of band communication channel, at least a portion of the data package to a trusted management system to initiate remediation of the unrecoverable error.
18. The data processing system of claim 17 , further comprising:
after providing the at least the portion of the data package:
obtaining, by the management controller and via the out of band communication channel, at least one action for remediation of the unrecoverable error; and
initiating, by the management controller, performance of the at least one action.
19. The data processing system of claim 17 , wherein identifying the occurrence of the unrecoverable error comprises:
identifying, by the management controller, at least one unsuccessful attempt to restart the data processing system after the occurrence of the unrecoverable error.
20. The data processing system of claim 17 , wherein identifying the occurrence of the unrecoverable error comprises:
obtaining, by the management controller, a message from an agent hosted by the hardware resources, the message indicating that a management entity of the hardware resources has suffered the unrecoverable error.