Container-based stateful application resilience to node failure
Techniques for managing node failures in container environments are disclosed. In one example, a method determines when a first node executing at least one containerized workload has failed. In response to determining the first node has failed, the method marks a configuration object for the first node with an indicator that the first node is not to be used to schedule execution of a subsequent containerized workload, isolates from the first node one or more storage volumes used by the first node, and deletes configuration objects for the one or more storage volumes and for the containerized workload. The method then causes creation of a replacement containerized workload for execution on a second node, removes one or more artifacts associated with the containerized workload from the first node, and removes the indicator from the configuration object for the first node.
1. An apparatus comprising:
at least one processing platform comprising at least one processor coupled to at least one memory, the at least one processing platform, when executing program code, is configured to:
determine when a first node executing at least one containerized workload in an information processing system has failed;
in response to determining the first node has failed, mark a configuration object for the first node with an indicator that the first node is not to be used to schedule execution of a subsequent containerized workload;
isolate, from the first node, one or more storage volumes used by the first node when executing the at least one containerized workload;
delete configuration objects for the one or more storage volumes and for the at least one containerized workload executed by the first node;
cause creation of a replacement containerized workload for execution on a second node in the information processing system;
remove one or more artifacts associated with the at least one containerized workload from the first node; and
remove the indicator from the configuration object for the first node.
2. The apparatus of claim 1 , wherein determining when a first node executing at least one containerized workload in an information processing system has failed further comprises monitoring for receipt of a periodic signal received from the first node.
3. The apparatus of claim 1 , wherein determining when a first node executing at least one containerized workload in an information processing system has failed further comprises querying a storage system used by the first node when executing the at least one containerized workload to determine whether or not the storage system has, within a given time period, at least one of: (i) maintained one or more data path connections with the first node; and (ii) executed one or more input-output operations for the first node.
4. The apparatus of claim 1 , wherein the processing platform, when executing program code, is further configured to, after determining the first node has failed, update one or more configuration objects associated with the first node to a failed status.
5. The apparatus of claim 1 , wherein marking a configuration object for the first node with an indicator that the first node is not to be used to schedule execution of a subsequent containerized workload further comprises marking the configuration object with a taint label.
6. The apparatus of claim 1 , wherein isolating, from the first node, one or more storage volumes used by the first node when executing the at least one containerized workload further comprises fencing the one or more storage volumes from the first node.
7. The apparatus of claim 1 , wherein causing creation of a replacement containerized workload for execution on a second node in the information processing system further comprises:
causing creation of a configuration object for the replacement containerized workload; and
causing mapping of the one or more storage volumes to the replacement containerized workload.
8. The apparatus of claim 1 , wherein the at least one containerized workload and the replacement containerized workload are executable in respective pods on the first node and the second node.
9. The apparatus of claim 8 , wherein the at least one processing platform further comprises a pod monitoring controller.
10. The apparatus of claim 8 , wherein the at least one processing platform further comprises a pod monitor on the first node.
11. A method comprising:
determining when a first node executing at least one containerized workload in an information processing system has failed;
in response to determining the first node has failed, marking a configuration object for the first node with an indicator that the first node is not to be used to schedule execution of a subsequent containerized workload;
isolating, from the first node, one or more storage volumes used by the first node when executing the at least one containerized workload;
deleting configuration objects for the one or more storage volumes and for the at least one containerized workload executed by the first node;
causing creation of a replacement containerized workload for execution on a second node in the information processing system;
removing one or more artifacts associated with the at least one containerized workload from the first node; and
removing the indicator from the configuration object for the first node.
12. The method of claim 11 , wherein determining when a first node executing at least one containerized workload in an information processing system has failed further comprises monitoring for receipt of a periodic signal received from the first node.
13. The method of claim 11 , wherein determining when a first node executing at least one containerized workload in an information processing system has failed further comprises querying a storage system used by the first node when executing the at least one containerized workload to determine whether or not the storage system has, within a given time period, at least one of: (i) maintained one or more data path connections with the first node; and (ii) executed one or more input-output operations for the first node.
14. The method of claim 11 , further comprising updating one or more configuration objects associated with the first node to a failed status.
15. The method of claim 11 , wherein marking a configuration object for the first node with an indicator that the first node is not to be used to schedule execution of a subsequent containerized workload further comprises marking the configuration object with a taint label.
16. The method of claim 11 , wherein isolating, from the first node, one or more storage volumes used by the first node when executing the at least one containerized workload further comprises fencing the one or more storage volumes from the first node.
17. The method of claim 11 , wherein causing creation of a replacement containerized workload for execution on a second node in the information processing system further comprises:
causing creation of a configuration object for the replacement containerized workload; and
causing mapping of the one or more storage volumes to the replacement containerized workload.
18. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing platform causes the at least one processing platform to:
determine when a first node executing at least one containerized workload in an information processing system has failed;
in response to determining the first node has failed, mark a configuration object for the first node with an indicator that the first node is not to be used to schedule execution of a subsequent containerized workload;
isolate, from the first node, one or more storage volumes used by the first node when executing the at least one containerized workload;
delete configuration objects for the one or more storage volumes and for the at least one containerized workload executed by the first node;
causing creation of a replacement containerized workload for execution on a second node in the information processing system;
remove one or more artifacts associated with the at least one containerized workload from the first node; and
remove the indicator from the configuration object for the first node.
19. The computer program product of claim 18 , wherein marking a configuration object for the first node with an indicator that the first node is not to be used to schedule execution of a subsequent containerized workload further comprises marking the configuration object with a taint label.
20. The computer program product of claim 18 , wherein isolating, from the first node, one or more storage volumes used by the first node when executing the at least one containerized workload further comprises fencing the one or more storage volumes from the first node.