Application aware input/output fencing
Disclosed herein are methods, systems, and processes to perform application aware input/output (I/O) fencing operations. A determination is made that a cluster has been partitioned. The cluster includes multiple nodes. As a result of the partitioning, the nodes are split between a first network partition with a first set of nodes and a second network partition with a second set of nodes. Another determination is made that instances of an application are executing on the first set of nodes and the second set of nodes. An application aware I/O fencing operation is then performed that causes termination of instances of the application executing on the first set of nodes or on the second set of nodes.
1. A method comprising:
determining that a cluster has been partitioned, wherein
the cluster comprises a plurality of nodes,
each of a plurality of application instances of an application are executing on a corresponding one of the plurality of nodes,
as a result of the partitioning, the plurality of nodes are split into a plurality of network partitions, and
the plurality of network partitions comprise
a first network partition, comprising a first set of nodes of the plurality of nodes, and
a second network partition, comprising a second set of nodes of the plurality of nodes;
determining whether at least one application instance of the plurality of application instances is executing on at least one of the first set of nodes and at least one other application instance of the plurality of application instances is executing on at least one of the second set of nodes; and
in response to a determination that the at least one application instance of the plurality of application instances is executing on at least one of the first set of nodes and at least one other application instance of the plurality of application instances is executing on at least one of the second set of nodes, performing an application fencing operation, wherein
the application fencing operation causes termination of
the at least one application instance, or
the at least one other application instance, and
the application fencing operation does not prevent the first set of nodes or the second set of nodes from continuing to operate.
2. The method of claim 1 , further comprising:
performing a fencing race, wherein
the performing the fencing race comprises, at least in part,
accessing an application weight matrix, wherein
the application weight matrix comprises a weight assigned to each application instance of the application, and
comparing a first total application weight for the first set of nodes and a second total application weight for the second set of nodes, wherein
the first total application weight is a total of an application weight of each application instance executing on the first set of nodes, and
the second total application weight is a total of an application weight of each application instance executing on the second set of nodes.
3. The method of claim 2 , further comprising:
bypassing the fencing race, if all of the plurality of application instances are executing on the first set of nodes or on the second set of nodes; and
broadcasting a message to one or more nodes on which one or more remaining application instances of the plurality of application instances are executing.
4. The method of claim 2 , further comprising:
determining whether the first total application weight is greater than the second total application weight.
5. The method of claim 4 , wherein performing the fencing race further comprises:
uninstalling a registration key associated with each instance of the application executing on the second set of nodes in the second network partition from one or more coordination points.
6. The method of claim 5 , further comprising:
determining whether the first total application weight is lesser than the second total application weight; and
based on the determining whether the first total application weight is lesser than the second total application weight, performing the fencing race after a delay.
7. The method of claim 6 , wherein
the delay is based on a time required for a second racer node to perform another fencing race, wherein
the second set of nodes comprises the second racer node.
8. A non-transitory computer-readable storage medium storing program instructions executable to:
determine that a cluster has been partitioned, wherein
the cluster comprises a plurality of nodes,
each of a plurality of application instances of an application are executing on a corresponding one of the plurality of nodes,
as a result of the partitioning, the plurality of nodes are split into a plurality of network partitions, and
the plurality of network partitions comprise
a first network partition, comprising a first set of nodes of the plurality of nodes, and
a second network partition, comprising a second set of nodes of the plurality of nodes;
determine whether at least one application instance of the plurality of application instances is executing on at least one of the first set of nodes and at least one other application instance of the plurality of application instances is executing on at least one of the second set of nodes; and
in response to a determination that the at least one application instance of the plurality of application instances is executing on at least one of the first set of nodes and at least one other application instance of the plurality of application instances is executing on at least one of the second set of nodes, perform an application fencing operation, wherein
the application fencing operation causes termination of
the at least one application instance, or
the at least one other application instance, and
the application fencing operation does not prevent the first set of nodes or the second set of nodes from continuing to operate.
9. The non-transitory computer-readable storage medium of claim 8 , wherein the program instructions are further executable to:
perform a fencing race, wherein
the fencing race performed comprises, at least in part,
accessing an application weight matrix, wherein
the application weight matrix comprises a weight assigned to each application instance of the application, and
comparing a first total application weight for the first set of nodes and a second total application weight for the second set of nodes, wherein
the first total application weight is a total of an application weight of each application instance executing on the first set of nodes, and
the second total application weight is a total of an application weight of each application instance executing on the second set of nodes.
10. The non-transitory computer-readable storage medium of claim 9 , wherein the program instructions are further executable to:
bypass the fencing race, if all of the plurality of application instances are executing on the first set of nodes or on the second set of nodes; and
broadcast a message to one or more nodes on which one or more remaining application instances of the plurality of application instances are executing.
11. The non-transitory computer-readable storage medium of claim 9 , wherein the program instructions are further executable to:
determine whether the first total application weight is greater than the second total application weight.
12. The non-transitory computer-readable storage medium of claim 11 , wherein the program instructions to perform the fencing race further comprise program instructions to:
uninstall a registration key associated with each instance of the application executing on the second set of nodes in the second network partition from one or more coordination points.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the program instructions are further executable to:
determine whether the first total application weight is lesser than the second total application weight; and
based on whether the first total application weight is lesser than the second total application weight, perform the fencing race after a delay.
14. The non-transitory computer-readable storage medium of claim 13 , wherein
the delay is based on a time required for a second racer node to perform another fencing race, wherein
the second set of nodes comprises the second racer node.
15. A system comprising:
one or more processors; and
a memory coupled to the one or more processors, wherein the memory stores program instructions executable by the one or more processors to
determine that a cluster has been partitioned, wherein the cluster comprises a plurality of nodes,
each of a plurality of application instances of an application are executing on a corresponding one of the plurality of nodes,
as a result of the partitioning, the plurality of nodes are split into a plurality of network partitions, and
the plurality of network partitions comprise
a first network partition, comprising a first set of nodes of the plurality of nodes, and
a second network partition, comprising a second set of nodes of the plurality of nodes,
a determine whether at least one application instance of the plurality of application instances is executing on at least one of the first set of nodes and at least one other application instance of the plurality of application instances is executing on at least one of the second set of nodes; and
in response to a determination that the at least one application instance of the plurality of application instances is executing on at least one of the first set of nodes and at least one other application instance of the plurality of application instances is executing on at least one of the second set of nodes, perform an application fencing operation, wherein
the application fencing operation causes termination of
the at least one application instance, or
the at least one other application instance, and
the application fencing operation does not prevent the first set of nodes or the second set of nodes from continuing to operate.
16. The system of claim 15 , wherein the program instructions are further executable to:
perform a fencing race, wherein
the fencing race performed comprises, at least in part,
accessing an application weight matrix, wherein
the application weight matrix comprises a weight assigned to each application instance of the application, and
comparing a first total application weight for the first set of nodes and a second total application weight for the second set of nodes, wherein
the first total application weight is a total of an application weight of each application instance executing on the first set of nodes, and
the second total application weight is a total of an application weight of each application instance executing on the second set of nodes.
17. The system of claim 16 , wherein the program instructions are further executable to:
bypass the fencing race, if all of the plurality of application instances are executing on the first set of nodes or on the second set of nodes; and
broadcast a message to one or more nodes on which one or more remaining application instances of the plurality of application instances are executing.
18. The system of claim 16 , wherein the program instructions are further executable to:
determine whether the first total application weight is greater than the second total application weight.
19. The system of claim 18 , wherein the program instructions to perform the fencing race further comprise program instructions to:
uninstall a registration key associated with each instance of the application executing on the second set of nodes in the second network partition from one or more coordination points.
20. The system of claim 19 , further comprising:
determine whether the first total application weight is lesser than the second total application weight on; and
based on whether the first total application weight is lesser than the second total application weight, perform the fencing race after a delay.
21. The system of claim 20 , wherein
the delay is based on a time required for a second racer node to perform another fencing race, wherein
the second set of nodes comprises the second racer node.