Method and system for differential deduplication in untrusted storage
The present disclosure provides a method and system for differential deduplication of data blocks in an untrusted storage. An example system determines first data blocks of similar data blocks as reference data blocks and second data blocks of the identified similar data blocks which are compressed using the reference data blocks to enable differential deduplication of data blocks.
1. A method for differential deduplication of one or more data blocks in an untrusted storage, the method comprising:
providing a hash value of the one or more data blocks, one or more similarity hash values of the one or more data blocks, and an identification (KeyID) of an encryption key of one data block of the one or more data blocks;
performing, by a trusted server, a first compression and a first encryption of the one or more data blocks;
sending the compressed and encrypted one or more data blocks from the trusted server to the untrusted storage, by executing a write function of a first application programming interface (API);
identifying, by the untrusted storage, similar data blocks of the compressed and encrypted one or more data blocks with identical similarity hash values;
performing, by the trusted server, a request for the similar data blocks from the untrusted storage by executing a second API;
receiving, by the trusted server, a vector comprising the similar data blocks from the untrusted storage by performing a read function of the second API;
decrypting and decompressing, by the trusted server, the similar data blocks comprised in the received vector;
identifying similarities between the decrypted and decompressed similar data blocks;
determining first data blocks of the decrypted and decompressed similar data blocks as reference data blocks and second data blocks of the decrypted and decompressed similar data blocks to be compressed using the reference data blocks;
performing, by the trusted server, a second compression by using the reference data blocks and performing a second encryption to re-compress and re-encrypt the second data blocks;
performing a write function, by executing the second API, to the untrusted storage, and sending the re-compressed and re-encrypted second data blocks compressed with the reference data blocks, hash values of the second data blocks, hash values of the reference data blocks, KeyIDs of encryption keys of the second data blocks, KeyIDs of encryption keys of the reference data blocks from the trusted server to the untrusted storage; and
replacing in the untrusted storage the compressed and encrypted one or more data blocks with the re-compressed and re-encrypted second data blocks and obtaining differential deduplication data blocks.
2. The method according to claim 1 , wherein the trusted server belongs to one or more sets of trusted servers.
3. The method according to claim 1 , wherein the method further comprises decompressing the differential deduplication data blocks by performing a decryption of the differential deduplication data blocks, performing a decryption of the compressed reference data blocks, performing a decompression of the decrypted reference data blocks, performing a decompression of the differential deduplication data blocks by using decompressed reference data blocks.
4. The method according to claim 1 , wherein the sending the compressed and encrypted one or more data blocks comprises sending a size of the one or more data blocks, the hash value of the one or more data blocks, the one or more similarity hash values of the one or more data blocks, and the KeyID of the encryption key of the one block of the one or more data blocks.
5. The method according to claim 1 , wherein the receiving the vector comprises receiving a size of the compressed and encrypted one or more data blocks, the KeyID of the encryption key of the one block of the one or more data blocks, the hash value of the one or more data blocks, and the one or more similarity hash values of the one or more data blocks.
6. The method according to claim 1 , wherein the vector comprises the similar data blocks with largest common similarity hash values.
7. A trusted server for differential deduplication of one or more data blocks in an untrusted storage, wherein:
the trusted server comprises an interface; at least one memory; and at least one processor coupled to the interface and the at least one memory, wherein the at least one memory stores programming instructions for execution by the at least one processor to cause the trusted server to perform operations comprising:
implementing a first application programming interface for executing at least one of a write function and a read function between the trusted server and the untrusted storage;
implementing a second application programming interface for executing at least one of a request function and a replacing function between the trusted server and the untrusted storage;
calculating a hash value and one or more similarity hash values for the one or more data blocks;
performing in the untrusted storage differential deduplication of the one or more data blocks;
decrypting encrypted data blocks; and
performing a data deduplication and compression to the untrusted storage.
8. The trusted server according to claim 7 , wherein the operations further comprise:
performing a first compression and a first encryption of the one or more data blocks;
sending the compressed and encrypted one or more data blocks to the untrusted storage;
performing a request for similar data blocks from the untrusted storage;
receiving a vector comprising the similar data blocks from the untrusted storage;
decrypting and decompressing the similar data blocks comprised in the received vector;
determining first data blocks of the decrypted and decompressed similar data blocks as reference data blocks and second data blocks of the decrypted and decompressed similar data blocks to be compressed using the reference data blocks;
re-compressing and re-encrypting the second data blocks by performing a second compression using the reference data blocks and performing a second encryption; and
sending the re-compressed and re-encrypted second data blocks compressed with the reference data blocks, hash values of the second data blocks, hash values of the reference data blocks, identifications (KeyIDs) of encryption keys of the second data blocks, and KeyIDs of encryption keys of the reference data blocks to the untrusted storage.
9. A system for differential deduplication of one or more data blocks in an untrusted storage, the system comprises:
the untrusted storage;
one or more sets of trusted servers comprising one or more trusted servers;
a first application programming interface for executing at least one of a write function and a read function between the one or more trusted servers and the untrusted storage;
a second application programming interface for executing at least one of a request function and a replacing function between the one or more trusted servers and the untrusted storage;
at least one processor; and
at least one memory coupled to the at least one processor and storing programming instructions for execution by the at least one processor to cause the system to perform operations comprising:
receiving, by the untrusted storage, compressed and encrypted one or more data blocks;
identifying, by the untrusted storage, similar data blocks of the compressed and encrypted data blocks;
decrypting, by the untrusted storage, compressed reference data blocks;
calculating, by the one or more trusted servers, a hash value, and one or more similarity hash values for the one or more data blocks;
performing, by the one or more trusted servers, in the untrusted storage differential deduplication of the one or more data blocks;
decrypting, by the one or more trusted servers, encrypted data blocks; and
performing, by the one or more trusted servers, a data deduplication and compression to the untrusted storage.
10. The system according to claim 9 , wherein the system further comprises a key manager running in the at least one processor, and wherein the operations further comprise generating, by the key manager, to the one or more trusted servers one or more encryption keys.
11. The system according to claim 10 , wherein the operations further comprise sharing, by each of the one or more sets of trusted servers, the one or more encryption keys, the hash values, and the one or more similarity hash values between trusted servers within the same set.
12. The method according to claim 2 , wherein each of the one or more sets of trusted servers shares the hash value of the one or more data blocks, the one or more similarity hash values of the one or more data blocks, and the at least one encryption key of the one or more data blocks between one or more trusted servers within the same set.
13. The trusted server according to claim 7 , wherein the trusted server belongs to one or more sets of trusted servers, and wherein each of the one or more sets of trusted servers shares the hash value of the one or more data blocks and the one or more similarity hash values of the one or more data blocks between one or more trusted servers within the same set.
14. The trusted server according to claim 8 , wherein the sending the compressed and encrypted one or more data blocks to the untrusted storage comprises sending a size of the one or more data blocks, the hash value of the one or more data blocks, and the one or more similarity hash values of the one or more data blocks to the untrusted storage.
15. The trusted server according to claim 8 , wherein the receiving the vector comprises receiving a size of the compressed and encrypted one or more data blocks, the hash value of the one or more data blocks, and the one or more similarity hash values of the one or more data blocks.
16. The trusted server according to claim 15 , wherein the vector comprises the similar data blocks with largest common similarity hash values.
17. The system according to claim 9 , wherein the operations further comprise obtaining differential deduplication data blocks, performing a decryption of the differential deduplication data blocks, performing a decryption of the compressed reference data blocks, performing a decompression of the decrypted reference data blocks, and performing a decompression of the differential deduplication data blocks by using the decompressed reference data blocks.
18. The system according to claim 9 , wherein the operations further comprise:
performing, by the one or more trusted servers, a first compression and a first encryption of the one or more data blocks to obtain the compressed and encrypted one or more data blocks.
19. The system according to claim 9 , wherein the receiving the compressed and encrypted one or more data blocks comprises receiving a size of the one or more data blocks, the hash value of the one or more data blocks, and the one or more similarity hash values of the one or more data blocks.
20. The system according to claim 9 , wherein the operations further comprise:
receiving, by the one or more trusted servers, a vector comprising the similar data blocks from the untrusted storage.