Optimizing application recovery in unmanaged clusters
Various systems, methods, and processes to perform recovery operations in a cluster based on exponential backoff models are disclosed. A node failure is detected. The node is one of multiple nodes in a cluster. In response to the node failure, an application executing on the node is failed over to another node in the cluster. In response to the detecting the node failure, recovery operations are automatically performed to determine whether the node is recovered. A subsequent recovery operation is performed after a prior recovery operation. The subsequent recovery operation is performed periodically based on a frequency that decreases exponentially after performing the prior recovery operation.
1. A computer-implemented method, comprising:
determining a mean time to recovery (MTTR) and an auto-clear retry threshold (ACRT) of a node based on one or more characteristics of the node, wherein
the MTTR comprises a mean time required to recover the node failure, and
the ACRT comprises a frequency at which a plurality of recovery operations are to be performed;
detecting failure of the node, wherein
the node is one of a plurality of nodes comprised in a cluster, and
in response to the node failure, an application executing on the node is failed over to another node in the cluster; and
in response to the detecting the node failure, performing the plurality of recovery operations to determine whether the node is recovered based on the ACRT until an elapsed time since the node failure is equal to the MTTR, wherein
a subsequent recovery operation of the plurality of recovery operations is performed after a prior recovery operation of the plurality of recovery operations, and
the subsequent recovery operation is performed periodically based on the frequency that decreases exponentially after performing the prior recovery operation.
2. The computer-implemented method of claim 1 , wherein
a time period between each recovery operation of the plurality of recovery operations increases exponentially after the elapsed time reaches the MTTR.
3. The method of claim 2 , wherein
one or more recovery operation of the plurality of recovery operations are performed based on the ACRT until the elapsed time is equal to the MTTR.
4. The method of claim 3 , further comprising
if the MTTR and the ACRT cannot be determined based on one or more node characteristics, receiving the MTTR and the ACRT as part of one or more user inputs, or one or more Service Level Agreements (SLAs).
5. The method of claim 2 , wherein
the MTTR comprises information indicating the mean time required to recover the node failure such that the node can be designated as a failover target for the application.
6. The method of claim 3 , wherein
the ACRT comprises information indicating a specific minimum frequency at which each subsequent recovery operation of the plurality of recovery operations is to be performed until the elapsed time is equal to the MTTR.
7. The method of claim 1 , wherein
the cluster is an unmanaged cluster.
8. The method of claim 1 , wherein
the cluster is a minimally managed cluster.
9. The method of claim 2 , wherein
the MTTR is based on the one or more characteristics of the cluster comprising
a geographic location of the cluster,
a size of the cluster, or
complexity of the application.
10. A non-transitory computer readable storage medium storing program instructions executable to:
determine a mean time to recovery (MTTR) and an auto-clear retry threshold (ACRT) of a node based on one or more characteristics of the node, wherein
the MTTR comprises a mean time required to recover the node failure, and
the ACRT comprises a frequency at which a plurality of recovery operations are to be performed;
detect failure of the node, wherein
the node is one of a plurality of nodes comprised in a cluster, and
in response to the node failure, an application executing on the node is failed over to another node in the cluster; and
in response to the detecting the node failure, perform the plurality of recovery operations to determine whether the node is recovered based on the ACRT until an elapsed time since the node failure is equal to the MTTR, wherein
a subsequent recovery operation of the plurality of recovery operations is performed after a prior recovery operation of the plurality of recovery operations, and
the subsequent recovery operation is performed periodically based on the frequency that decreases exponentially after performing the prior recovery operation.
11. The non-transitory computer readable storage medium of claim 10 , wherein
a time period between each recovery operation of the plurality of recovery operations increases exponentially after the elapsed time reaches the MTTR and
one or more recovery operation of the plurality of recovery operations are performed based on the ACRT until the elapsed time is equal to the MTTR.
12. The non-transitory computer readable storage medium of claim 11 , further comprising
if the MTTR and the ACRT cannot be determined based on one or more node characteristics, receiving the MTTR and the ACRT as part of one or more user inputs, or one or more Service Level Agreements (SLAs).
13. The non-transitory computer readable storage medium of claim 12 , wherein
the MTTR comprises information indicating the mean time required to recover the node failure such that the node can be designated as a failover target for the application, and
the ACRT comprises information indicating a specific minimum frequency at which each subsequent recovery operation of the plurality of recovery operations is to be performed until the elapsed time is equal to the MTTR.
14. The non-transitory computer readable storage medium of claim 10 , wherein the cluster is
an unmanaged cluster, or
a minimally managed cluster.
15. The non-transitory computer readable storage medium of claim 11 , wherein
the MTTR is based on one or more characteristics of the cluster comprising
a geographic location of the cluster,
a size of the cluster,
complexity of the application.
16. A system comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory stores program instructions executable by the one or more processors to:
determine a mean time to recovery (MTTR) and an auto-clear retry threshold (ACRT) of a node based on one or more characteristics of the node; wherein
the MTTR comprises a mean time required to recover the node failure, and
the ACRT comprises a frequency at which a plurality of recovery operations are to be performed;
detect failure of the node, wherein
the node is one of a plurality of nodes comprised in a cluster, and
in response to the node failure, an application executing on the node is failed over to another node in the cluster; and
in response to the detecting the node failure, perform the plurality of recovery operations to determine whether the node is recovered based on the ACRT until an elapsed time since the node failure is equal to the MTTR, wherein
a subsequent recovery operation of the plurality of recovery operations is performed after a prior recovery operation of the plurality of recovery operations, and
the subsequent recovery operation is performed periodically based on the frequency that decreases exponentially after performing the prior recovery operation.
17. The system of claim 16 , wherein
a time period between each recovery operation of the plurality of recovery operations increases exponentially after the elapsed time reaches the MTTR, and
one or more recovery operation of the plurality of recovery operations are performed based on the ACRT until the elapsed time is equal to the MTTR.
18. The system of claim 17 , wherein
if the MTTR and the ACRT cannot be determined based on one or more node characteristics, receiving the MTTR and the ACRT as part of one or more user inputs, or one or more Service Level Agreements (SLAs),
the MTTR comprises information indicating a mean time required to recover the node failure such that the node can be designated as a failover target for the application, and
the ACRT comprises information indicating a specific minimum frequency at which each recovery operation of the plurality of recovery operations is to be performed until the elapsed time is equal to the MTTR.
19. The system of claim 16 , wherein
the cluster is
an unmanaged cluster, or
a minimally managed cluster.
20. The system of claim 17 , wherein
the MTTR is based on one or more characteristics of the cluster comprising
a geographic location of the cluster,
a size of the cluster, or
complexity of the application.