System and method for distributed deduplication of encrypted chunks
The present disclosure relates to an advantageous system and related methods for distributed deduplication of encrypted chunks. One embodiment relates to a method for storing encrypted chunks in which an encryption key is generated independently from a chunk payload. With this method, two encrypted chunks are identifiable as having identical chunk payloads even when the chunk payloads are encrypted with different encryption keys. Other embodiments, aspects and features are also disclosed.
1. A method of putting an encrypted chunk to an object storage system supporting distributed deduplication, the method comprising:
calculating a cryptographic hash to fingerprint an unencrypted payload by a client/proxy (a client, or proxy acting on behalf of the client), after optional compression has been applied at the client/proxy;
sending a chunk put request to a chunk server that identifies the chunk to be put by a fingerprint, wherein the chunk server was selected either because it is co-located with the client/proxy or by a consistent hashing algorithm.
2. The method of claim 1 further comprising:
determining by the chunk server whether a specified chunk identifier is already stored within a federated set of chunk servers by checking local storage of the chunk server and at least one of the chunk servers designated to store the specified chunk identifier by the consistent hashing algorithm.
3. The method of claim 2 further comprising:
indicating by the chunk server to the client/proxy whether the chunk was already stored, or whether a chunk payload is to be provided.
4. The method of claim 3 further comprising:
when the chunk payload is to be provided, the client/proxy performs steps including generating an effectively random chunk key that is not correlated with the chunk payload, encrypting the chunk payload with the generated chunk key, sending the encrypted chunk payload to the chunk server, encrypting the chunk key for the chunk server, and sending encrypted chunk key to the chunk server.
5. The method of claim 4 further comprising:
receiving the encrypted chunk payload and the encrypted chunk key by the chunk server; and
validating the encrypted chunk payload by the chunk server, wherein the validating is performed by decrypting the encrypted chunk key, decrypting the received payload with the decrypted chunk key, calculating the cryptographic hash fingerprint of the decrypted chunk payload, comparing the calculated fingerprint with the fingerprint identifying the chunk, and declaring the chunk to be invalid if said fingerprints do not match.
6. The method of claim 5 , wherein if said fingerprints do match, then
storing the encrypted chunk payload as a chunk image at the chunk server,
encrypting the chunk key with a private key shared with no other server,
including the encrypted chunk key in chunk metadata for the chunk image, and
writing the chunk metadata at the chunk server.