Distributed fair allocation of shared resources to constituents of a cluster
View Patent ↗Techniques are disclosed for allocating shared resources to nodes in a distributed computing network system. Nodes request a lock for each instance of a computing resource (e.g., a virtual IP address associated with a service provided by the distributed computing network system) from a distributed lock manager. The distributed lock manager maintains a queue of requests for each instance of the shared resource. Upon receiving a lock from the distributed lock manager, the receiving node performs a fairness allocation protocol to determine whether to accept the lock. If so determined, the shared computing resources associated with the lock is configured.
1. A method for allocating shared computing resources in a distributed computing cluster, the method comprising:
electing, by an instance of a distributed lock manager executing on each of a plurality of nodes in the distributed computing cluster, one of the plurality of nodes to act as a leader;
requesting, from the instance of the distributed lock manager on the elected node, a lock for each instance of a computing resource, wherein the instance of the distributed lock manager on the elected node maintains a queue of requests for each instance of the shared resource; and
upon receiving a lock from the instance of the distributed lock manager on the elected node:
determining, based on a fairness allocation protocol, whether to accept the lock, wherein the fairness allocation protocol identifies a maximum number of locks to instances of the shared resource that a given node can obtain;
upon determining to accept the lock, configuring the shared computing resources associated with the lock, wherein a first M % N of the plurality of nodes in a sorted list of the plurality of nodes obtain a remainder of M % N instances of the computing resource, where M is a total number of instances of the computing resource and N is a total number of nodes in the distributed computing cluster; and
advertising the configuration to each of the plurality of nodes in the distributed computing cluster.
2. The method of claim 1 , further comprising, upon determining to not accept the lock:
rejecting the lock;
requesting the lock from the instance of the distributed lock manager on the elected node.
3. The method of claim 1 , wherein each instance of the computing resource is a distinct virtual IP address associated with a service provided by the distributed computing cluster.
4. The method of claim 1 , wherein the distributed computing cluster is a secondary storage system.
5. The method of claim 1 , further comprising, upon detecting a change in cluster membership within the distributed computing cluster, prior to determining whether to accept the lock, delaying for a specified period of time after receiving the lock.
6. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform an operation for allocating shared computing resources in a distributed computing cluster, the operation comprising:
electing, by an instance of a distributed lock manager executing on each of a plurality of nodes in the distributed computing cluster, one of the plurality of nodes to act as a leader;
requesting, from the instance of the distributed lock manager on the elected node, a lock for each instance of a computing resource, wherein the instance of the distributed lock manager on the elected node maintains a queue of requests for each instance of the shared resource; and
upon receiving a lock from the instance of the distributed lock manager on the elected node:
determining, based on a fairness allocation protocol, whether to accept the lock, wherein the fairness allocation protocol identifies a maximum number of locks to instances of the shared resource that a given node can obtain;
upon determining to accept the lock, configuring the shared computing resources associated with the lock, wherein a first M % N of the plurality of nodes in a sorted list of the plurality of nodes obtain a remainder of M % N instances of the computing resource, where M is a total number of instances of the computing resource and N is a total number of nodes in the distributed computing cluster; and
advertising the configuration to each of the plurality of nodes in the distributed computing cluster.
7. The computer-readable storage medium of claim 6 , wherein the operation further comprises, upon determining to not accept the lock:
rejecting the lock;
requesting the lock from the instance of the distributed lock manager on the elected node.
8. The computer-readable storage medium of claim 6 , wherein each instance of the computing resource is a distinct virtual IP address associated with a service provided by the distributed computing cluster.
9. The computer-readable storage medium of claim 6 , wherein the operation further comprises, upon detecting a change in cluster membership within the distributed computing cluster, prior to determining whether to accept the lock, delaying for a specified period of time after receiving the lock.
10. A system comprising:
a processor; and
a memory storing program code, which, when executed on the processor performs an operation for allocating shared computing resources in a distributed computing cluster, the operation comprising:
electing, by an instance of a distributed lock manager executing on each of a plurality of nodes in the distributed computing cluster, one of the plurality of nodes to act as a leader;
requesting, from the instance of the distributed lock manager on the elected node, a lock for each instance of a computing resource, wherein the instance of the distributed lock manager on the elected node maintains a queue of requests for each instance of the shared resource; and
upon receiving a lock from the instance of the distributed lock manager on the elected node:
determining, based on a fairness allocation protocol, whether to accept the lock, wherein the fairness allocation protocol identifies a maximum number of locks to instances of the shared resource that a given node can obtain;
upon determining to accept the lock, configuring the shared computing resources associated with the lock, wherein a first M % N of the plurality of nodes in a sorted list of the plurality of nodes obtain a remainder of M % N instances of the computing resource, where M is a total number of instances of the computing resource and N is a total number of nodes in the distributed computing cluster; and
advertising the configuration to each of the plurality of nodes in the distributed computing cluster.
11. The system of claim 10 , wherein the operation further comprises, upon determining to not accept the lock:
rejecting the lock;
requesting the lock from the instance of the distributed lock manager on the elected node.
12. The system of claim 10 , wherein each instance of the computing resource is a distinct virtual IP address associated with a service provided by the distributed computing cluster.
13. The system of claim 10 , wherein the operation further comprises, upon detecting a change in cluster membership within the distributed computing cluster, prior to determining whether to accept the lock, delaying for a specified period of time after receiving the lock.