Failure detection and recovery techniques for converged network adaptors
Techniques discussed herein relate to monitoring and/or detecting destabilization events of a host machine of a cloud computing environment. At least some of these destabilization events can be associated with a converged network adaptor of the host machine. The converged network adaptor may include a combination of functionality of a host network interface card (NIC) and functionality associated with a smart adaptor (e.g., a smart NIC). The smart adaptor may be configured to use at the host machine. A converged infrastructure management service may monitor operational data associated with the host machine to identify destabilization events that may be associated with the converged network adaptor. One or more remedial actions may be identified based at least in part on identifying these destabilizing events.
1 . A computer-implemented method comprising:
configuring, by a converged infrastructure management service of a cloud computing environment, a smart adaptor of a converged network interface card for use at a host machine of the cloud computing environment, the converged network interface card being a hardware component comprising 1) a first storage partition with which first functionality associated with a host network interface card is provided and 2) a second storage partition associated with the smart adaptor and with which second functionality associated with providing network virtualization functions related to at least one virtual cloud network is provided, the second storage partition being hidden from a host operating system executing at the host machine, the converged network interface card being configured with an image that has been booted at a second host machine as part of an auto-qualification process for the image;
monitoring, by the converged infrastructure management service of the cloud computing environment, operational data corresponding to the host machine;
identifying, by the converged infrastructure management service of the cloud computing environment, a destabilization event that is associated with the converged network interface card based at least in part on the monitoring; and
performing, by the converged infrastructure management service of the cloud computing environment, one or more remedial actions based at least in part on identifying the destabilization event.
2 . The computer-implemented method of claim 1 , wherein the smart adaptor is configured with one or more features, and where configuring the smart adaptor comprises:
determining, based at least in part on an identifier corresponding to the host machine, a subset of permitted features selected from the one or more features of the smart adaptor, the subset of permitted features being features that are permitted to be utilized at the host machine; and
executing operations to cause the smart adaptor to 1) expose an interface associated with the subset of permitted features, or 2) restrict access to features other than the subset of permitted features.
3 . The computer-implemented method of claim 1 , further comprising:
initiating the auto-qualification process for pre-testing one or more images with the converged network interface card;
booting, at the second host machine, each of the one or more images, the second host machine being configured to operate into an isolated network and configured with an instance of the converge network interface card; and
associating each of the one or more images with a label that indicates a respective image is stable or unstable based at least in part on identifying whether a given image booted fully at the second host machine, wherein a corresponding image that is associated with an unstable label indicating the corresponding image is unstable is restricted from being deployed to infrastructure components, including the host machine, that are configured with the converged network interface card.
4 . The computer-implemented method of claim 1 , wherein identifying the destabilization event further comprises providing input to a machine learning model trained to identify destabilization events from input data, the machine learning model being previously trained using a supervised learning algorithm and training data comprising training data examples individually being associated with corresponding operational data, or data generated from the corresponding operational data, and a known destabilization event label.
5 . The computer-implemented method of claim 1 , wherein the operational data is obtained from at least one of: 1) an integrated lights out manager, 2) a baseboard management controller, 3) an agent operating at an operating system executing at the host machine, 4) a load balancer associated with the host machine, 5) the smart adaptor, or 6) the host network interface card of the host machine.
6 . The computer-implemented method of claim 1 , wherein the operational data comprises at least one: 1) a reboot duration value, 2) a panic idle power draw value, 3) console logs or console log related data, 4) system event logs or system event log related data, 5) CPU power consumption data, or 6) network traffic flow data.
7 . A converged infrastructure management service of a cloud computing environment, comprising:
one or more processors; and
one or more memories storing computer-executable instructions that, when executed with the one or more processors, cause the converged infrastructure management service to:
configure, by the converged infrastructure management service, a smart adaptor of a converged network interface card operating at a host machine of the cloud computing environment, the converged network interface card being a hardware component comprising 1) a first storage partition with which first functionality associated with a host network interface card is provided and 2) a second storage partition associated with the smart adaptor and with which second functionality associated with providing network virtualization functions related to at least one virtual cloud network is provided, the second storage partition being hidden from a host operating system executing at the host machine, the converged network interface card being configured with an image that has been booted at a second host machine as part of an auto-qualification process for the image;
monitor, by the converged infrastructure management service, operational data corresponding to the host machine;
identify, by the converged infrastructure management service, a destabilization event that is associated with the converged network interface card based at least in part on the monitoring; and
perform, by the converged infrastructure management service, one or more remedial actions based at least in part on identifying the destabilization event.
8 . The converged infrastructure management service of claim 7 , wherein the smart adaptor is configured with one or more features, and where configuring the smart adaptor comprises:
determining, based at least in part on an identifier corresponding to the host machine, a subset of permitted features selected from the one or more features of the smart adaptor, the subset of permitted features being features that are permitted to be utilized at the host machine; and
executing operations to cause the smart adaptor to 1) expose an interface associated with the subset of permitted features, or 2) restrict access to features other than the subset of permitted features.
9 . The converged infrastructure management service of claim 7 , wherein executing the computer-executable instructions further causes the converged infrastructure management service to:
initiate the auto-qualification process for pre-testing one or more images with the converged network interface card;
boot, at the second host machine, each of the one or more images, the second host machine being configured to operate into an isolated network and configured with an instance of the converge network interface card; and
associate each of the one or more images with a label that indicates a respective image is stable or unstable based at least in part on identifying whether a given image booted fully at the second host machine, wherein a corresponding image that is associated with an unstable label indicating the corresponding image is unstable is restricted from being deployed to infrastructure components, including the host machine, that are configured with the converged network interface card.
10 . The converged infrastructure management service of claim 7 , wherein the operational data is obtained from at least one of: 1) an integrated lights out manager, 2) a baseboard management controller, 3) an agent operating at an operating system executing at the host machine, 4) a load balancer associated with the host machine, 5) the smart adaptor, or 6) the host network interface card of the host machine.
11 . The converged infrastructure management service of claim 7 , wherein the operational data comprises at least one: 1) a reboot duration value, 2) a panic idle power draw value, 3) console logs or console log related data, 4) system event logs or system event log related data, 5) CPU power consumption data, or 6) network traffic flow data.
12 . A non-transitory computer-readable medium comprising executable instructions that, when executed with one or more processors of a converged infrastructure management service within a cloud computing environment cause the converged infrastructure management service to:
configure, by the converged infrastructure management service, a smart adaptor of a converged network interface card operating at a host machine of the cloud computing environment, the converged network interface card being a hardware component comprising 1) a first storage partition with which first functionality associated with a host network interface card is provided and 2) a second storage partition associated with the smart adaptor and with which second functionality associated with-providing network virtualization functions related to at least one virtual cloud network is provided, the second storage partition being hidden from a host operating system executing at the host machine, the converged network interface card being configured with an image that has been booted at a second host machine as part of an auto-qualification process for the image;
monitor, by the converged infrastructure management service, operational data corresponding to the host machine;
identify, by the converged infrastructure management service, a destabilization event that is associated with the converged network interface card based at least in part on the monitoring; and
perform, by the converged infrastructure management service, one or more remedial actions based at least in part on identifying the destabilization event.
13 . The non-transitory computer-readable medium of claim 12 , wherein the smart adaptor is configured with one or more features, and where configuring the smart adaptor comprises:
determining, based at least in part on an identifier corresponding to the host machine, a subset of permitted features selected from the one or more features of the smart adaptor, the subset of permitted features being features that are permitted to be utilized at the host machine; and
executing operations to cause the smart adaptor to 1) expose an interface associated with the subset of permitted features, or 2) restrict access to features other than the subset of permitted features.
14 . The non-transitory computer-readable medium of claim 12 , wherein executing the computer-executable instructions further causes the converged infrastructure management service to:
initiate the auto-qualification process for pre-testing one or more images with the converged network interface card;
boot, at the second host machine, each of the one or more images, the second host machine being configured to operate into an isolated network and configured with an instance of the converge network interface card; and
associate each of the one or more images with a label that indicates a respective image is stable or unstable based at least in part on identifying whether a given image booted fully at the second host machine, wherein a corresponding image that is associated with an unstable label indicating the corresponding image is unstable is restricted from being deployed to infrastructure components, including the host machine, that are configured with the converged network interface card.
15 . The non-transitory computer-readable medium of claim 12 , wherein identifying the destabilization event further causes the converged infrastructure management service to provide input to a machine learning model trained to identify destabilization events from input data, the machine learning model being previously trained using a supervised learning algorithm and training data comprising training data examples individually being associated with corresponding operational data, or data generated from the corresponding operational data, and a known destabilization event label.
16 . The non-transitory computer-readable medium of claim 12 , wherein the operational data is obtained from at least one of: 1) an integrated lights out manager, 2) a baseboard management controller, 3) an agent operating at an operating system executing at the host machine, 4) a load balancer associated with the host machine, 5) the smart adaptor, or 6) the host network interface card of the host machine.
17 . The non-transitory computer-readable medium of claim 12 , wherein the operational data comprises at least one: 1) a reboot duration value, 2) a panic idle power draw value, 3) console logs or console log related data, 4) system event logs or system event log related data, 5) CPU power consumption data, or 6) network traffic flow data.
18 . The computer-implemented method of claim 1 , wherein configuring the smart adaptor comprises deploying, to the second storage partition, a software agent that provides third functionality that differs from the first functionality and the second functionality.
19 . The computer-implemented method of claim 1 , wherein the first storage partition and the second storage partition of the converged network interface card are physically and communicatively connected via a peripheral component interconnect express (PCIe) connection.