Efficient deduplication of compressed files
The present disclosure describes a technique for performing an efficient deduplication of compressed source data. The techniques may reduce the required storage footprint required for deduplication of compressed data. In order to reduce the storage size required, the system may perform additional decompression/recompression processes by identifying particular compression algorithms used by a source storage system. Once the compression algorithm is identified, the system may initiate decompression and then perform fingerprint analysis of the segment in the file of the uncompressed data. When a recovery process is initiated, the system may recompress the deduplicated data using the same compression algorithm used by the source storage system. Accordingly, the data recovery process may be performed in manner in which the client device receives restored data as expected and in the original compression format.
1. A computer-implemented method of storing data within a storage system, comprising:
in response to receiving an indication to backup one or more files stored on a first source storage system to a target storage system, determining that the one or more files are stored in a compressed format on the first source storage system using an index, wherein the index indicates which of the one or more files are compressed, the index being stored on the first source storage system and the target storage system, wherein the target storage system implements data deduplication, the indication being provided by the first source storage that works in conjunction with an application agent being a part of the first source storage system;
determining a first compression algorithm used to compress the one or more files stored on the first source storage system by analyzing metadata associated with the one or more files;
initiating a decompression using the determined first compression algorithm of the one or more files on the source storage system to create one or more decompressed files, wherein a library of compression algorithms including the first compression algorithm used for the decompression is shared by the source storage system, target storage system, and the application agent, wherein the decompression is scheduled based on determining when a Central Processing Unit (CPU) usage capacity drops below a predefined threshold;
performing a fingerprint analysis on segments of the one or more created decompressed files to identify segments that are already stored by the target storage system based on the performed fingerprint analysis, wherein the fingerprint analysis includes comparing fingerprints of the segments of the one or more created decompressed files with a fingerprint index associated with the target storage system, wherein the fingerprint index is a portion of the metadata and at least a portion of the fingerprint index is stored or cached in a memory; and
transmitting, to the target storage system, the segments of the one or more created decompressed files not already stored on the target storage system.
2. The method of claim 1 , further comprising:
in response to transmitting the segments of the one or more created decompressed files to the target storage system, initiating storage of the segments on the target storage system, wherein the segments are stored in a compressed format using a second compression algorithm.
3. The method of claim 2 , wherein the second compression algorithm is different than the first compression algorithm.
4. The method of claim 3 , further comprising:
in response to receiving an indication to restore one or more files of the first source storage system stored on the target storage system, determining the one or more files were originally stored in the compressed format on the first source storage system using the first compression algorithm;
decompressing the segments of the one or more files stored on the target storage system using the second compression algorithm;
initiating a recompression of the decompressed segments using the first compression algorithm originally used by the source storage system; and
restoring the one or more files of the first source storage system to the original compressed format with the recompressed segments.
5. The method of claim 4 , further comprising:
determining the first compression algorithm originally used by the source storage system by analyzing metadata stored during a storage process performed by the target storage system.
6. The method of claim 1 , wherein the decompression of the one or more files is performed on the source storage system prior to transmitting segments of the one or more files to the target storage system.
7. The method of claim 4 , wherein the recompression of the decompressed segments is performed on the target storage system prior to transmitting the segments to the source storage system for restoration.
8. The method of claim 1 , further comprising:
determining the one or more files stored on the first source storage system are encrypted; and
in response to receiving an input to decrypt the one of more files, performing a decryption of the files prior to performing the fingerprint analysis on segments of the one or more files stored on the first source storage system.
9. A system, comprising:
a memory storing instructions; and
a processor coupled to the memory to execute the instructions from the memory, the processor being configured to perform operations, comprising:
in response to receiving an indication to backup one or more files stored on a first source storage system to a target storage system, determining that the one or more files are stored in a compressed format on the first source storage system using an index, wherein the index indicates which of the one or more files are compressed, the index being stored on the first source storage system and the target storage system, wherein the target storage system implements data deduplication, the indication being provided by the first source storage that works in conjunction with an application agent being a part of the first source storage system;
determining a first compression algorithm used to compress the one or more files stored on the first source storage system by analyzing metadata associated with the one or more files;
initiating a decompression using the determined first compression algorithm of the one or more files on the source storage system to create one or more decompressed files, wherein a library of compression algorithms including the first compression algorithm used for the decompression is shared by the source storage system, target storage system, and the application agent, wherein the decompression is scheduled based on determining when a Central Processing Unit (CPU) usage capacity drops below a predefined threshold;
performing a fingerprint analysis on segments of the one or more created decompressed files to identify segments that are already stored by the target storage system based on the performed fingerprint analysis, wherein the fingerprint analysis includes comparing fingerprints of the segments of the one or more created decompressed files with a fingerprint index associated with the target storage system; and
transmitting, to the target storage system, the segments of the one or more created decompressed files not already stored on the target storage system.
10. The system of claim 9 , the operations further comprising:
in response to transmitting the segments of the one or more created decompressed files to the target storage system, initiating storage of the segments on the target storage system, wherein the segments are stored in a compressed format using a second compression algorithm different than the first compression algorithm.
11. The system of claim 10 , the operations further comprising:
in response to receiving an indication to restore one or more files of the first source storage system stored on the target storage system, determining the one or more files were originally stored in the compressed format on the first source storage system using the first compression algorithm;
decompressing the segments of the one or more files stored on the target storage system using the second compression algorithm;
initiating a recompression of the decompressed segments using the first compression algorithm originally used by the source storage system; and
restoring the one or more files of the first source storage system to the original compressed format with the recompressed segments.
12. The system of claim 11 , the operations further comprising:
determining the first compression algorithm originally used by the source storage system by analyzing metadata stored during a storage process performed by the target storage system.
13. The system of claim 9 , wherein the decompression of the one or more files is performed on the source storage system prior to transmitting segments of the one or more files to the target storage system.
14. The system of claim 11 , wherein the recompression of the decompressed segments is performed on the target storage system prior to transmitting the segments to the source storage system for restoration.
15. The system of claim 9 , the operations further comprising:
determining the one or more files stored on the first source storage system are encrypted; and
in response to receiving an input to decrypt the one of more files, performing a decryption of the files prior to performing the fingerprint analysis on segments of the one or more files stored on the first source storage system.
16. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations, comprising:
in response to receiving an indication to backup one or more files stored on a first source storage system to a target storage system, determining that the one or more files are stored in a compressed format on the first source storage system using an index, wherein the index indicates which of the one or more files are compressed, the index being stored on the first source storage system and the target storage system, wherein the target storage system implements data deduplication, the indication being provided by the first source storage that works in conjunction with an application agent being a part of the first source storage system;
determining a first compression algorithm used to compress the one or more files stored on the first source storage system by analyzing metadata associated with the one or more files;
initiating a decompression using the determined first compression algorithm of the one or more files on the source storage system to create one or more decompressed files, wherein a library of compression algorithms including the first compression algorithm used for the decompression is shared by the source storage system, target storage system, and the application agent, wherein the decompression is scheduled based on determining when a Central Processing Unit (CPU) usage capacity drops below a predefined threshold;
performing a fingerprint analysis on segments of the one or more created decompressed files to identify segments that are already stored by the target storage system based on the performed fingerprint analysis, wherein the fingerprint analysis includes comparing fingerprints of the segments of the one or more created decompressed files with a fingerprint index associated with the target storage system; and
transmitting, to the target storage system, the segments of the one or more created decompressed files not already stored on the target storage system.
17. The medium of claim 16 , the operations further comprising:
in response to transmitting the segments of the one or more created decompressed files to the target storage system, initiating storage of the segments on the target storage system, wherein the segments are stored in a compressed format using a second compression algorithm different than the first compression algorithm.
18. The medium of claim 17 , the operations further comprising:
in response to receiving an indication to restore one or more files of the first source storage system stored on the target storage system, determining the one or more files were originally stored in the compressed format on the first source storage system using the first compression algorithm;
decompressing the segments of the one or more files stored on the target storage system using the second compression algorithm;
initiating a recompression of the decompressed segments using the first compression algorithm originally used by the source storage system; and
restoring the one or more files of the first source storage system to the original compressed format with the recompressed segments.
19. The medium of claim 16 , wherein the decompression of the one or more files is performed on the source storage system prior to transmitting segments of the one or more files to the target storage system.
20. The medium of claim 18 , wherein the recompression of the decompressed segments is performed on the target storage system prior to transmitting the segments to the source storage system for restoration.