Application aware input/output fencing
Disclosed herein are methods, systems, and processes to perform application aware input/output (I/O) fencing operations. Performing such an application aware I/O fencing operation includes installing an identifier that identifies an instance of an application with a node on which the instance of the application is executing, on coordination points. A weight assigned to the instance of the application is determined, and the instance of the application is terminated based on the weight.
1. A computer-implemented method comprising:
determining a first weight assigned to a first application instance of a plurality of application instances and a second weight assigned to a second application instance of the plurality of application instances, wherein
each of the plurality of application instances is an instance of an application,
the first application instance is executed by a first node of a cluster of nodes, and
the second application instance is executed by a second node of the cluster of nodes; and
in response to detection of a network partition event, performing an application fencing operation, wherein
the network partition event results in partitioning of the cluster of nodes into at least a first sub-cluster and a second sub-cluster,
the first node is comprised in the first sub-cluster,
the second node is comprised in the second sub-cluster, and
the application fencing operation comprises
terminating one of the first application instance or the second application instance, wherein
the terminating is based, at least in part, on the first weight and the second weight, and
the terminating is performed without terminating either of the first node or the second node.
2. The computer-implemented method of claim 1 , further comprising:
installing each identifier of a plurality of identifiers on one or more coordination points, wherein
the each identifier of the plurality of identifiers associates a corresponding application instance of the plurality of application instances with a corresponding node of the cluster of nodes.
3. The computer-implemented method of claim 2 , further comprising:
accessing an application weight matrix, wherein
the application weight matrix comprises the first weight and the second weight.
4. The computer-implemented method of claim 3 , further comprising:
receiving the application weight matrix;
generating the plurality of identifiers, wherein
each of the plurality of identifiers is one of a plurality of registration keys; and
as part of the installing, storing the plurality of registration keys on the one or more coordination points.
5. The computer-implemented method of claim 4 , wherein
each of the one or more coordination points comprises at least one of
a coordinator disk, or
a coordination point server.
6. The computer-implemented method of claim 3 , further comprising:
performing a partition arbitration process using the application weight matrix.
7. The computer-implemented method of claim 6 , wherein
the network partition event partitions the cluster of nodes into a plurality of sub-clusters,
the plurality of sub-clusters comprises the first sub-cluster and the second sub-cluster, and
the performing the partition arbitration process comprises
determining a winner sub-cluster of the plurality of sub-clusters and one or more loser sub-clusters of the plurality of sub-clusters by performing a fencing race.
8. The computer-implemented method of claim 7 , further comprising:
excluding a third application instance of a plurality of other application instances and a fourth application instance of the plurality of other application instances from the fencing race, if both the third application instance and the fourth application instance are executed on one or more nodes of the cluster of nodes and the one or more nodes are within one sub-cluster of the plurality of sub-clusters, wherein
each of the plurality of other application instances is an instance of an other application; and
including the third application instance and the fourth application instance in the fencing race, if the third application instance is executed by a third node of the cluster of nodes, the fourth application instance is executed by a fourth node of the cluster of nodes, and the third node and the fourth node belong to different ones of the plurality of sub-clusters.
9. The computer-implemented method of claim 7 , wherein
the fencing race is decided based, at least in part, on information in the application weight matrix.
10. The computer-implemented method of claim 2 , further comprising:
as part of the installing, storing each of a plurality of registration keys on the one or more coordination points, wherein
the each identifier of the plurality of identifiers is one of the plurality of registration keys; and
performing a partition arbitration process that comprises performing a fencing race, wherein
as part of the fencing race, a registration key of the plurality of registration keys for one of the first application instance or the second application instance is uninstalled from a coordination point of the one or more coordination points based, at least in part, on the first weight and the second weight.
11. A non-transitory computer-readable storage medium storing program instructions executable to cause one or more processors to:
determine a first weight assigned to a first application instance of a plurality of application instances and a second weight assigned to a second application instance of the plurality of application instances, wherein
each of the plurality of application instances is an instance of an application,
the first application instance is executed by a first node of a cluster of nodes, and
the second application instance is executed by a second node of the cluster of nodes; and
in response to detection of a network partition event, perform an application fencing operation, wherein
the network partition event results in partitioning of the cluster of nodes into at least a first sub-cluster and a second sub-cluster,
the first node is comprised in the first sub-cluster, and
the second node is comprised in the second sub-cluster, and
the application fencing operation comprises
terminating one of the first application instance or the second application instance, wherein
the terminating is based, at least in part, on the first weight and the second weight, and
the terminating is performed without terminating either of the first node or the second node.
12. The non-transitory computer-readable storage medium of claim 11 , wherein the program instructions are further executable to cause the one or more processors to:
install each identifier of a plurality of identifiers on one or more coordination points, wherein
the each identifier of the plurality of identifiers associates a corresponding application instance of the plurality of application instances with a corresponding node of the cluster of nodes.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the program instructions are further executable to cause the one or more processors to:
access an application weight matrix, wherein
the application weight matrix comprises the first weight and the second weight;
receive the application weight matrix;
generate the plurality of identifiers, wherein
each of the plurality of identifiers is one of a plurality of registration keys; and
store the plurality of registration keys on the one or more coordination points.
14. The non-transitory computer-readable storage medium of claim 13 , wherein
each of the one or more coordination points comprises at least one of
a coordinator disk, or
a coordination point server.
15. The non-transitory computer-readable storage medium of claim 13 , wherein the program instructions are further executable to cause the one or more processors to:
perform a partition arbitration process using the application weight matrix, wherein
the partition arbitration process comprises
determining a winner sub-cluster of the plurality of sub-clusters and one or more loser sub-clusters of the plurality of sub-clusters by performing a fencing race, wherein
the fencing race is decided based, at least in part, on information in the application weight matrix, and
as part of the fencing race, a registration key of the plurality of registration keys for one of the first application instance or the second application instance is uninstalled from a coordination point of the one or more coordination points based, at least in part, on the first weight and the second weight.
16. A system comprising:
one or more processors; and
a computer-readable storage medium coupled to the one or more processors, wherein the computer-readable storage medium stores program instructions executable by the one or more processors to perform a method comprising
determining a first weight assigned to a first application instance of a plurality of application instances and a second weight assigned to a second application instance of the plurality of application instances, wherein
each of the plurality of application instances is an instance of an application,
the first application instance is executed by a first node of a cluster of nodes, and
the second application instance is executed by a second node of the cluster of nodes, and
in response to detection of a network partition event, performing an application fencing operation, wherein
the network partition event results in partitioning of the cluster of nodes into at least a first sub-cluster and a second sub-cluster,
the first node is comprised in the first sub-cluster,
the second node is comprised in the second sub-cluster, and
the application fencing operation comprises
terminating one of the first application instance or the second application instance, wherein
the terminating is based, at least in part, on the first weight and the second weight, and
the terminating is performed without terminating either of the first node or the second node.
17. The system of claim 16 , wherein the method further comprises:
installing each identifier of a plurality of identifiers on one or more coordination points, wherein
the each identifier of the plurality of identifiers associates a corresponding application instance of the plurality of application instances with a corresponding node of the cluster of nodes.
18. The system of claim 17 , wherein the method further comprises:
accessing an application weight matrix, wherein
the application weight matrix comprises the first weight and the second weight;
receiving the application weight matrix;
generating the plurality of identifiers, wherein
each of the plurality of identifiers is one of a plurality of registration keys; and
as part of the installing, storing the plurality of registration keys on the one or more coordination points.
19. The system of claim 18 , wherein
each of the one or more coordination points comprises at least one of
a coordinator disk, or
a coordination point server.
20. The system of claim 18 , wherein the method further comprises:
performing a partition arbitration process using the application weight matrix.
21. The system of claim 20 , wherein
the network partition event partitions the cluster of nodes into a plurality of sub-clusters,
the plurality of sub-clusters comprises the first sub-cluster and the second sub-cluster, and
the performing the partition arbitration process comprises
determining a winner sub-cluster of the plurality of sub-clusters and one or more loser sub-clusters of the plurality of sub-clusters by performing a fencing race, wherein
the fencing race is decided based, at least in part, on information in the application weight matrix, and
as part of the fencing race, a registration key of the plurality of registration keys for one of the first application instance or the second application instance is uninstalled from a coordination point of the one or more coordination points based, at least in part, on the first weight and the second weight.
22. The system of claim 21 , wherein the method further comprises:
excluding a third application instance of a plurality of other application instances and a fourth application instance of the plurality of other application instances from the fencing race, if both the third application instance and the fourth application instance are executed on one or more nodes of the cluster of nodes and the one or more nodes are within one sub-cluster of the plurality of sub-clusters, wherein
each of the plurality of other application instances is an instance of an other application; and
including the third application instance and the fourth application instance in the fencing race, if the third application instance is executed by a third node of the cluster of nodes, the fourth application instance is executed by a fourth node of the cluster of nodes, and the third node and the fourth node belong to different ones of the plurality of sub-clusters.