Application aware input/output fencing
Disclosed herein are various embodiments that perform application-aware input/output (I/O) fencing operations, certain of which embodiments include, in response to detection of a network partition event in a cluster, determining a first application weight, determining whether the first one or more application instances should be delayed in a cumulative fencing race, performing the cumulative fencing race (in response to a determination that the first one or more application instances should be delayed in the cumulative fencing race), and performing the cumulative fencing race without introducing the delay with regard to the first one or more application instances (in response to a determination that the first one or more application instances should not be delayed in the cumulative fencing race).
1. A computer-implemented method comprising:
in response to detection of a network partition event in a cluster comprising a plurality of nodes,
determining whether the network partition event resulted in partitioning of the plurality of nodes into a first sub-cluster and a second sub-cluster,
wherein
each node of the plurality of nodes executes at least one application instance of a first plurality of application instances of a first application,
in response to a determination that the network partition event partitioned the plurality of nodes into the first sub-cluster and the second sub-cluster,
determining an application weight of at least one application instance of the first plurality of application instances,
determining whether to include the first application in a first group of applications, wherein
the determining whether to include the first application in the first group of applications is based, at least in part, on the application weight, and
in response to a determination that the first application should be included in the first group of applications, including the first application in the first group of applications, and
performing a cumulative fencing race, wherein the cumulative fencing race includes the first group of applications.
2. The computer-implemented method of claim 1 , further comprising:
ejecting one or more application-aware registration keys, of a plurality of application-aware registration keys, from a coordination point, wherein
each of the one or more application-aware registration keys is associated with one or more corresponding application instances executed by one or more nodes of a leaving sub-cluster, and
the leaving sub-cluster is one of the first sub-cluster or the second sub-cluster.
3. The computer-implemented method of claim 2 , wherein
each application-aware registration key of the plurality of application-aware registration keys is associated with a corresponding application instance of the first plurality of application instances.
4. The computer-implemented method of claim 1 , further comprising:
creating the first group of applications, wherein
the application weight indicates that the network partition event partitioned the first plurality of application instances into the first sub-cluster and the second sub-cluster.
5. The computer-implemented method of claim 1 , wherein the determining whether the network partition event resulted in partitioning of the plurality of nodes into the first sub-cluster and the second sub-cluster comprises:
determining whether the network partition event resulted in
all application instances of the first plurality of application instances being executed by one or more nodes of the first sub-cluster, or
the all application instances of the first plurality of application instances being executed by one or more nodes of the second sub-cluster.
6. The computer-implemented method of claim 5 , wherein the determining whether the network partition event resulted in the all application instances of the first plurality of application instances being executed by the one or more nodes of the first sub-cluster or the all application instances of the first plurality of application instances being executed by the one or more nodes of the second sub-cluster comprises:
determining whether the application weight is equal to zero or the application weight is equal to a total application weight of the all application instances.
7. The computer-implemented method of claim 1 , wherein the determining the application weight of the at least one application instance comprises:
determining a first application weight, wherein
the first application weight is based, at least in part, on an application weight of a first one or more application instances, and
the first one or more application instances are ones of the at least one application instance that are executed by one or more nodes of the first sub-cluster,
determining a second application weight, wherein
the second application weight is based, at least in part, on an application weight of a second one or more application instances,
the second one or more application instances are ones of the at least one application instance that are executed by one or more nodes of the second sub-cluster,
the determining whether to include the first application in the first group of applications is based, at least in part, on the first application weight and the second application weight, and
the application weight is comprised in at least one of the first application weight or the second application weight.
8. The computer-implemented method of claim 7 , wherein the determining whether to include the first application in a first group of applications comprises:
comparing the first application weight and the second application weight; and
in response to a result of the comparing indicating that the first application should be included in the first group of applications, indicating that the first application should be included in the first group of applications.
9. The computer-implemented method of claim 8 , wherein
the result of the comparing indicates that the first application should be included in the first group of applications, if the first application weight is greater than or equal to the second application weight.
10. The computer-implemented method of claim 7 , wherein the first application weight is a first number of application instances of the first plurality of application instances executed in the first sub-cluster, the second application weight is a second number of application instances of the first plurality of application instances executed in the second sub-cluster, and the determining whether to include the first application in a first group of applications comprises:
comparing the first number and the second number; and
in response to the first number being greater than or equal to the second number, indicating that the first application should be included in the first group of applications.
11. The computer-implemented method of claim 7 , further comprising:
comparing the first application weight and the second application weight;
in response to the first application weight being greater than or equal to the second application weight, performing the cumulative fencing race without introducing a delay with regard to the first one or more application instances; and
in response to the first application weight being less than the second application weight, performing the cumulative fencing race without introducing a delay with regard to the second one or more application instances.
12. The computer-implemented method of claim 7 , further comprising:
determining whether the first application weight is equal to a total application weight for the first application, wherein
the total application weight is a total of an application weight for each application instance of the first plurality of application instances; and
in response to the first application weight being equal to the total application weight for the first application, indicating that the first one or more application instances should not be delayed in the cumulative fencing race.
13. The computer-implemented method of claim 1 , further comprising:
further in response to the detection of the network partition event and a determination that the network partition event partitioned the plurality of nodes into the first sub-cluster and the second sub-cluster,
in response to a determination that the first application should be included in a second group of applications, including the first application in the second group of applications, and
performing the cumulative fencing race, wherein
the cumulative fencing race includes the second group of applications.
14. The computer-implemented method of claim 13 , further comprising:
determining whether one or more applications in the second group of applications should be delayed in the cumulative fencing race, wherein
the determining is based, at least in part, on the application weight, and
the cumulative fencing race is configured to terminate an application instance executed by a node of the plurality of nodes without terminating the node, and
in response to a determination that the applications in the second group of applications should be delayed in the cumulative fencing race,
performing the cumulative fencing race, wherein
the cumulative fencing race is performed such that a delay is introduced with regard to application instances of the applications in the second group of applications.
15. The computer-implemented method of claim 14 , further comprising:
in response to a determination that the applications in the second group of applications should not be delayed in the cumulative fencing race,
creating a first data structure, and
storing first information, regarding a first one or more application instances, in the first data structure, wherein
the first one or more application instances are executed by one or more nodes of the first sub-cluster; and
in response to the determination that the applications in the second group of applications should be delayed in the cumulative fencing race,
creating a second data structure, and
storing second information, regarding a second one or more application instances, in the first data structure, wherein
the second one or more application instances are executed by one or more nodes of the second sub-cluster.
16. A non-transitory computer-readable storage medium, comprising program instructions, which, when executed by one or more processors of a computing system, perform a method comprising:
in response to detection of a network partition event in a cluster comprising a plurality of nodes,
determining whether the network partition event resulted in partitioning of the plurality of nodes into a first sub-cluster and a second sub-cluster,
wherein
each node of the plurality of nodes executes at least one application instance of a first plurality of application instances of a first application,
in response to a determination that the network partition event partitioned the plurality of nodes into the first sub-cluster and the second sub-cluster,
determining an application weight of at least one application instance of the first plurality of application instances,
determining whether to include the first application in a first group of applications, wherein
the determining whether to include the first application in the first group of applications is based, at least in part, on the application weight, and
in response to a determination that the first application should be included in the first group of applications, including the first application in the first group of applications, and
performing a cumulative fencing race, wherein
the cumulative fencing race includes the first group of applications.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the determining whether the network partition event resulted in partitioning of the plurality of nodes into the first sub-cluster and the second sub-cluster comprises:
determining whether the network partition event resulted in
all application instances of the first plurality of application instances being executed by one or more nodes of the first sub-cluster, or
the all application instances of the first plurality of application instances being executed by one or more nodes of the second sub-cluster.
18. The non-transitory computer-readable storage medium of claim 16 , wherein the determining the application weight of the at least one application instance comprises:
determining a first application weight, wherein
the first application weight is based, at least in part, on an application weight of a first one or more application instances, and
the first one or more application instances are ones of the at least one application instance that are executed by one or more nodes of the first sub-cluster,
determining a second application weight, wherein
the second application weight is based, at least in part, on an application weight of a second one or more application instances,
the second one or more application instances are ones of the at least one application instance that are executed by one or more nodes of the second sub-cluster,
the determining whether to include the first application in the first group of applications is based, at least in part, on the first application weight and the second application weight, and
the application weight is comprised in at least one of the first application weight or the second application weight.
19. The non-transitory computer-readable storage medium of claim 18 , wherein the determining whether to include the first application in a first group of applications comprises:
comparing the first application weight and the second application weight; and
in response to a result of the comparing indicating that the first application should be included in the first group of applications, indicating that the first application should be included in the first group of applications.
20. A computing system comprising:
one or more processors; and
a computer-readable storage medium coupled to the one or more processors, comprising program instructions, which, when executed by the one or more processors, perform a method comprising
in response to detection of a network partition event in a cluster comprising a plurality of nodes,
determining whether the network partition event resulted in partitioning of the plurality of nodes into a first sub-cluster and a second sub-cluster, wherein
each node of the plurality of nodes executes at least one application instance of a first plurality of application instances of a first application,
in response to a determination that the network partition event partitioned the plurality of nodes into the first sub-cluster and the second sub-cluster,
determining an application weight of at least one application instance of the first plurality of application instances,
determining whether to include the first application in a first group of applications, wherein
the determining whether to include the first application in the first group of applications is based, at least in part, on the application weight, and
in response to a determination that the first application should be included in the first group of applications, including the first application in the first group of applications, and
performing a cumulative fencing race, wherein
the cumulative fencing race includes the first group of applications.