Fencing off cluster services based on shared storage access keys
A cluster service receives a cluster message based on a removal of a first service from membership in a first node in a cluster, and requests a shared storage to generate and then provide a new access key, thereby enabling the shared storage to use the new access key to validate requests to access resources stored by the shared storage. The cluster service receives the new access key from the shared storage, and sends the new access key in at least some cluster messages to at least a second service in a second node in the cluster of nodes. The second service creates a key based on the new access key. The second service uses the key to create a request to access a resource stored by the shared storage, sends the request to the shared storage, and then accesses the resource, which was previously accessed by the first service.
1. A system for fencing off cluster services based on shared storage access keys comprising:
one or more processors; and
a non-transitory computer readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
request, by a cluster service, in response to receiving a cluster message upon a loss by a first service of cluster membership communications in a first node in a cluster, a shared storage to generate and then provide a new access key, thereby enabling the shared storage to use the new access key to validate requests from a plurality of services which have cluster membership communications in the cluster to access resources stored by the shared storage;
receive, by the cluster service, the new access key from the shared storage;
send, by the cluster service, the new access key in at least one or more cluster messages to at least a second service which has cluster membership communications in a second node in the cluster of nodes, in response to the cluster service receiving the new access key from the shared storage;
create, by the second service, a key based on the new access key; and
access, by the second service, a resource, which was previously accessed by the first service, by using the key to create a request to access the resource stored by the shared storage, and then sending the request to the shared storage.
2. The system of claim 1 , wherein using the new access key to validate requests to access resources stored by the shared storage comprises using the new access key to create signing keys that are used to validate signed requests to access resources stored by the shared storage; creating the key based on the new access key comprises creating a signing key based on the new access key; and using the key to create a request to access the resource stored by the shared storage, and then sending the request to the shared storage comprises using the signing key to sign a request to access the resource stored by the shared storage, and then sending the request signed by the signing key to the shared storage.
3. The system of claim 1 , wherein the cluster message is sent only to services which have cluster membership communications in the cluster, and the cluster message is sent only to nodes which have cluster membership communications in the cluster, the cluster message being based on a loss, by any node, of cluster membership communications in the cluster.
4. The system of claim 1 , wherein the resource comprises one of a local fingerprint index, a namespace, a local post-deduplication log, and a range of similarity group identifiers, and the shared storage comprises one of an object storage, a distributed key value store, and a distributed log.
5. The system of claim 1 , wherein the plurality of instructions further causes the processor to relocate the resource, which was previously accessed by the first service, from the first node to the second node, in response to the first service in the first node losing cluster membership communications in the first node in the cluster.
6. The system of claim 1 , wherein the cluster message further comprises a current list of member services and a previous list of member services, which the cluster service uses to identify the first service which lost cluster membership communications in the cluster, which enables the shared storage to create access keys that are used to create requests from only a type of the first service to access resources stored by the shared storage.
7. The system of claim 1 , wherein the cluster service identifies the resource accessed by a service which lost cluster membership communications in the cluster, and requests the shared storage to create access keys which are used to create requests from cluster services to access only a type of the resource stored by the shared storage.
8. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:
request, by a cluster service, in response to receiving a cluster message upon a removal of loss by a first service of cluster membership communications in a first node in a cluster, a shared storage to generate and then provide a new access key, thereby enabling the shared storage to use the new access key to validate requests from a plurality of services which have cluster membership communications in the cluster to access resources stored by the shared storage;
receive, by the cluster service, the new access key from the shared storage;
send, by the cluster service, the new access key in at least one or more cluster messages to at least a second service which has cluster membership communications in a second node in the cluster of nodes, in response to the cluster service receiving the new access key from the shared storage;
create, by the second service, a key based on the new access key; and
access, by the second service, a resource, which was previously accessed by the first service, by using the key to create a request to access the resource stored by the shared storage, and then sending the request to the shared storage.
9. The computer program product of claim 8 , wherein using the new access key to validate requests to access resources stored by the shared storage comprises using the new access key to create signing keys that are used to validate signed requests to access resources stored by the shared storage; creating the key based on the new access key comprises creating a signing key based on the new access key; and using the key to create a request to access the resource stored by the shared storage, and sending the request to the shared storage comprises using the signing key to sign a request to access the resource stored by the shared storage, and then sending the request signed by the signing key to the shared storage.
10. The computer program product of claim 8 , wherein the cluster message is sent only to services which have cluster membership communications in the cluster, and the cluster message is sent only to nodes which have cluster membership communications in the cluster, the cluster message being based on a loss, by any node, of cluster membership communications in the cluster.
11. The computer program product of claim 8 , wherein the resource comprises one of a local fingerprint index, a namespace, a local post-deduplication log, and a range of similarity group identifiers, and the shared storage comprises one of an object storage, a distributed key value store, and a distributed log.
12. The computer program product of claim 8 , wherein the program code includes further instructions to relocate the resource, which was previously accessed by the first service, from the first node to the second node, in response to the first service in the first node losing cluster membership communications in the first node in the cluster.
13. The computer program product of claim 8 , wherein the cluster message further comprises a current list of member services and a previous list of member services, which the cluster service uses to identify the first service which lost cluster membership communications, which enables the shared storage to create access keys that are used to create requests from only a type of the first service to access resources stored by the shared storage.
14. The computer program product of claim 8 , wherein the cluster service identifies the resource accessed by a service which lost cluster membership communications in the cluster, and requests the shared storage to create access keys which are used to create requests from cluster services to access only a type of the resource stored by the shared storage.
15. A computer-implemented method for fencing off cluster services based on shared storage access keys, comprising:
requesting, by a cluster service, in response to receiving a cluster message upon a loss by a first service of cluster membership communications in a first node in a cluster, a shared storage to generate and then provide a new access key, thereby enabling the shared storage to use the new access key to validate requests from a plurality of services which have cluster membership communications in the cluster to access resources stored by the shared storage;
receive, by the cluster service, the new access key from the shared storage;
sending, by the cluster service, the new access key in at least one or more cluster messages to at least a second service which has cluster membership communications in a second node in the cluster of nodes, in response to the cluster service receiving the new access key from the shared storage;
creating, by the second service, a key based on the new access key; and
accessing, by the second service, a resource, which was previously accessed by the first service, by using the key to create a request to access the resource stored by the shared storage, and then sending the request to the shared storage.
16. The computer-implemented method of claim 15 , wherein using the new access key to validate requests to access resources stored by the shared storage comprises using the new access key to create signing keys that are used to validate signed requests to access resources stored by the shared storage; creating the key based on the new access key comprises creating a signing key based on the new access key; and using the key to create a request to access the resource stored by the shared storage, and then sending the request to the shared storage comprises using the signing key to sign a request to access the resource stored by the shared storage, and then sending the request signed by the signing key to the shared storage.
17. The computer-implemented method of claim 15 , wherein the cluster message is sent only to services which have cluster membership communications in the cluster; the cluster message is sent only to nodes which have cluster membership communications in the cluster, the cluster message being based on a loss, by any node, of cluster membership communications in the cluster, and the resource comprises one of a local fingerprint index, a namespace, a local post-deduplication log, and a range of similarity group identifiers, and the shared storage comprises one of an object storage, a distributed key value store, and a distributed log.
18. The computer-implemented method of claim 15 , wherein the computer-implemented method further comprises relocating the resource, which was previously accessed by the first service, from the first node to the second node, in response to the first service in the first node losing cluster membership communications in the first node in the cluster.
19. The computer-implemented method of claim 15 , wherein the cluster message further comprises a current list of member services and a previous list of member services, which the cluster service uses to identify the first service which lost cluster membership communications, which enables the shared storage to create access keys which are used to create requests from only a type of the first service to access resources stored by the shared storage.
20. The computer-implemented method of claim 15 , wherein the cluster service identifies the resource accessed by a service which lost cluster membership communications in the cluster, and requests the shared storage to create access keys which are used to create requests from cluster services to access only a type of the resource stored by the shared storage.