Scalable leadership election in a multi-processing computing environment
A scalable multi-cluster resource sharing facility. A global witness process runs on a first computing platform that communicates over one or more networks to any number of nodes situated over two or more clusters. The global witness process listens on the network for occurrences of leadership and/or resource requests from nodes of different clusters. The global witness processes a request by retrieving a resource request and a respective last known state value, comparing the last known state value to a global stored state value, then storing a new state value when the respective last known state value is equal to the stored state value. Any number of contemporaneous requests can be processed by the global witness process, however only one request can be granted. The other requestors each receive a rejection of their resource request when their proffered last known state value is not equal to the stored state value.
1. A non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor performs a set of acts comprising:
receiving, at a witness process on a first node in a first fault domain, a request from a second node in a second fault domain to become an owner of a resource shared across the second fault domain and a third fault domain, wherein a third node in the third fault domain was previously established as the owner of the resource using the witness process and the witness process maintains an ownership state for the resource and the second and third fault domains are at different sites accessible over a network;
determining that the second node in the second fault domain should become the owner of the resource;
transmitting an authorization from the witness process on the first node to the second node granting ownership of the resource to the second node to become a new owner of the resource; and
updating the ownership state for the resource at the witness process on the first node, wherein the new owner of the resource communicates a copy of storage changes to both a fault domain of a follower node and the second fault domain of the second node.
2. The non-transitory computer readable medium of claim 1 , wherein the resource to be accessed is a leadership relationship.
3. The non-transitory computer readable medium of claim 1 , wherein multiple requestors contact the witness process to establish a leadership relationship.
4. The non-transitory computer readable medium of claim 3 , wherein the set of acts further comprise sending an affirmative acknowledgement to one of the multiple requestors to identify a single owner of the resource, and sending a negative acknowledgement to a set of remaining requestors to generate a set of followers.
5. The non-transitory computer readable medium of claim 4 , wherein the set of acts further comprise sending a periodic liveness signal by the owner of the resource.
6. The non-transitory computer readable medium of claim 4 , wherein at least some of the set of followers perform a liveness measure check.
7. The non-transitory computer readable medium of claim 1 , wherein the set of acts further comprise comparing a respective last known state value to a stored state value that is stored in a database.
8. The non-transitory computer readable medium of claim 7 , wherein the database comprises a plurality of state tables or a plurality of state values.
9. A method comprising:
receiving, at a witness process on a first node in a first fault domain, a request from a second node in a second fault domain to become an owner of a resource shared across the second fault domain and a third fault domain, wherein a third node in the third fault domain was previously established as the owner of the resource using the witness process and the witness process maintains an ownership state for the resource and the second and third fault domains are at different sites accessible over a network;
determining that the second node in the second fault domain should become the owner of the resource;
transmitting an authorization from the witness process on the first node to the second node granting ownership of the resource to the second node to become a new owner of the resource; and
updating the ownership state for the resource at the witness process on the first node, wherein the new owner of the resource communicates a copy of storage changes to both a fault domain of a follower node and the second fault domain of the second node.
10. The method of claim 9 , wherein the resource to be accessed is a leadership relationship.
11. The method of claim 9 , wherein multiple requestors contact the witness process to establish a leadership relationship.
12. The method of claim 11 , further comprising sending an affirmative acknowledgement to one of the multiple requestors to identify a single owner of the resource, and sending a negative acknowledgement to a set of remaining requestors to generate a set of followers.
13. The method of claim 12 , further comprising sending a periodic liveness signal by the owner of the resource.
14. The method of claim 12 , wherein at least some of the set of followers perform a liveness measure check.
15. The method of claim 9 , further comprising comparing a respective last known state value to a stored state value that is stored in a database.
16. The method of claim 15 , wherein the database comprises a plurality of state tables or a plurality of state values.
17. A system comprising:
a non-transitory storage medium having stored thereon a sequence of instructions; and
a processor that executes the sequence of instructions to perform a set of acts comprising:
receiving, at a witness process on a first node in a first fault domain, a request from a second node in a second fault domain to become an owner of a resource shared across the second fault domain and a third fault domain, wherein a third node in the third fault domain was previously established as the owner of the resource using the witness process and the witness process maintains an ownership state for the resource and the second and third fault domains are at different sites accessible over a network;
determining that the second node in the second fault domain should become the owner of the resource;
transmitting an authorization from the witness process on the first node to the second node granting ownership of the resource to the second node; and
updating the ownership state for the resource at the witness process on the first node;
transmitting an authorization from the witness process on the first node to the second node granting ownership of the resource to the second node to become a new owner of the resource; and
updating the ownership state for the resource at the witness process on the first node, wherein the new owner of the resource communicates a copy of storage changes to both a fault domain of a follower node and the second fault domain of the second node.
18. The system of claim 17 , wherein the resource to be accessed is a leadership relationship.
19. The system of claim 17 , wherein multiple requestors contact the witness process to establish a leadership relationship.
20. The system of claim 19 , wherein the set of acts further comprise sending an affirmative acknowledgement to one of the multiple requestors to identify a single owner of the resource, and sending a negative acknowledgement to a set of remaining requestors to generate a set of followers.
21. The system of claim 20 , wherein the set of acts further comprise sending a periodic liveness signal by the owner of the resource.
22. The system of claim 20 , wherein at least some of the set of followers perform a liveness measure check.
23. The system of claim 17 , wherein the set of acts further comprise, comparing a respective last known state value to a stored state value that is stored in a database.
24. The system of claim 23 , wherein the database comprises a plurality of state tables or a plurality of state values.