Secure encryption in deduplication cluster

1. A method comprising:

initiating, by a node among a cluster of nodes in a deduplication cluster, a leadership transition that includes transmitting an updated secret key identifier to each of the cluster of nodes, wherein the deduplication cluster splits a stream of data into blocks of data and for each block of data:

determines, whether a hash value for the block of data is a duplicate of a hash value for a stored block of data; and

responsive to the hash value for the block of data not matching the hash value for the stored block of data, routing the block of data to a process from among the cluster of nodes;

receiving, at the node, an alternative secret key identifier that is different from both a current secret key identifier and from the updated secret key identifier;

updating, based at least in part on a resolution policy, the current secret key identifier to be the updated secret key identifier instead of the alternative secret key identifier; and

transitioning, based at least in part on the updated secret key identifier being selected to be the current secret key identifier, the node to be a leader node of the cluster of nodes.

2. The method of claim 1 , wherein the deduplication cluster comprises multiple servers within an intermediate computing system between one or more client devices and a backend cloud storage service.

3. The method of claim 2 , wherein a data store on the intermediate computing system uses a consistent data storage model, and wherein the data store for the cloud storage service uses an eventually consistent data storage model.

4. The method of claim 1 , wherein the method further comprises:

receiving, via an application program interface of a front-end process, the stream of data; and

hashing the blocks of data.

5. The method of claim 1 , further comprising:

distributing, to different ones of the cluster of nodes, the hashed blocks of data that are not duplicates of stored data.

6. The method of claim 5 , further comprising:

sending, from a process that has received a hashed block of data to a remote data store, one or more portions of the block of data, wherein the one or more portions of the block of data correspond to one or more transactions.

7. The method of claim 6 , wherein the one or more transactions are recorded within a transaction log for the process, and wherein each process among the cluster of nodes generates a transaction log corresponding to data sent to the remote data store.

8. The method of claim 7 , wherein the front-end process receives the stream of bytes of data from a client device via a communication interface that is compatible with a communication interface provided by the object store.

9. The method of claim 7 , wherein the remote data store is an object store provided by a cloud services provider.

10. The method of claim 1 , wherein at least one process among the cluster of nodes operates in parallel with at least one other process among the cluster of nodes.

11. An apparatus comprising:

one or more hardware processors;

one or more data storage resources; and

a consistent transactions module configured to:

initiate, by a node among a cluster of nodes in a deduplication cluster, a leadership transition that includes transmitting an updated secret key identifier to each of the cluster of nodes, wherein the deduplication cluster splits a stream of data into blocks of data and for each block of data:

determines, whether a hash value for the block of data is a duplicate of a hash value for a stored block of data; and

responsive to the hash value for the block of data not matching the hash value for the stored block of data, routing the block of data to a process from among the cluster of nodes;

receive, at the node, an alternative secret key identifier that is different from both a current secret key identifier and from the updated secret key identifier;

update, based at least in part on a resolution policy, the current secret key identifier to be the updated secret key identifier instead of the alternative secret key identifier; and

transition, based at least in part on the updated secret key identifier being selected to be the current secret key identifier, the node to be a leader node of the cluster of nodes.

12. The apparatus of claim 11 , wherein the deduplication cluster comprises multiple servers within an intermediate computing system between one or more client devices and a backend cloud storage service.

13. The apparatus of claim 11 , wherein a data store on the intermediate computing system uses a consistent data storage model, and wherein the data store for the cloud storage service uses an eventually consistent data storage model.

14. The apparatus of claim 11 , wherein the consistent transactions module is further configured to:

receive, via an application program interface of a front-end process, the stream of data;

hash the blocks of data.

15. The apparatus of claim 11 , wherein the consistent transactions module is further configured to:

distribute, to different ones of the cluster of nodes, the hashed blocks of data that are not duplicates of stored data.

16. The apparatus of claim 15 , wherein the consistent transactions module is further configured to:

sending, from a process that has received a hashed block of data to a remote data store, one or more portions of the block of data, wherein the one or more portions of the block of data correspond to one or more transactions.

17. The apparatus of claim 16 , wherein the one or more transactions are recorded within a transaction log for the process, and wherein each process among the cluster of nodes generates a transaction log corresponding to data sent to the remote data store.

18. The apparatus of claim 17 , wherein the front-end process receives the stream of bytes of data from a client device via a communication interface that is compatible with a communication interface provided by the object store.

19. The apparatus of claim 17 , wherein the remote data store is an object store provided by a cloud services provider.

20. A computer program product disposed upon a non-transitory computer readable medium, the computer program product comprising computer program instructions that, when executed, cause a computer to carry out the steps of:

initiating, by a node among a cluster of nodes in a deduplication cluster, a leadership transition that includes transmitting an updated secret key identifier to each of the cluster of nodes, wherein the deduplication cluster splits a stream of data into blocks of data and for each block of data:

determines, whether a hash value for the block of data is a duplicate of a hash value for a stored block of data; and

responsive to the hash value for the block of data not matching the hash value for the stored block of data, routing the block of data to a process from among the cluster of nodes;

receiving, at the node, an alternative secret key identifier that is different from both a current secret key identifier and from the updated secret key identifier;

updating, based at least in part on a resolution policy, the current secret key identifier to be the updated secret key identifier instead of the alternative secret key identifier; and

transitioning, based at least in part on the updated secret key identifier being selected to be the current secret key identifier, the node to be a leader node of the cluster of nodes.