Segment deduplication system with compression of segments
A system for storing compressed data comprises a processor and a memory. The processor is configured to receive a compressed segment. The compressed segment is determined by breaking a data stream, a data block, or a data file into one or more segments and compressing each of the one or more segments. The processor is further configured to determine whether the compressed segment has been previously stored, and in the event that the compressed segment has not been previously stored, store the compressed segment. The memory is coupled to the processor and configured to provide the processor with instructions.
1. A system for storing compressed data, comprising:
a processor configured to:
break a data stream, a data block, or a data file into one or more segments, wherein one segment of the one or more segments is used to reconstruct more than one data streams, data blocks, or data files;
compress each of the one or more segments into one or more compressed segments;
receive one of the one or more compressed segments;
receive metadata information associated with the compressed segment, wherein the received metadata information includes a segment identifier and compression metadata information, wherein the compression metadata information includes a compression type and a compression state, wherein the compression state comprises a state used for compressing the compressed segment, wherein the compression state comprises a key;
determine whether the received compressed segment has been previously stored based at least in part on a comparison of the segment identifier associated with the received compressed segment and at least one segment identifier associated with a previously stored compressed segment;
in a first event that the received compressed segment is determined to have been previously stored, store a reference to the previously stored compressed segment;
in a second event that the received compressed segment is determined to have not been previously stored, store the received compressed segment, wherein the storing of the received compressed segment further comprises generating an index entry from the received compressed segment, wherein the index entry comprises one or more of the following: a digital fingerprint, SHA-1 hash, or Rabin Hash; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. A system as in claim 1 , wherein the breaking of the data stream, the data block, or the data file is based at least in part on one of the following: a content-based technique or a non-content-based technique.
3. A system as in claim 1 , wherein the compression of the each of the one or more segments comprises compressing using at least one of: a lossless compression system, a Huffman coding system, or a Lempel-Ziv Welch coding system.
4. A system as in claim 1 , wherein for a first segment and a second segment that is identical to the first segment, a compressed first segment is identical to a compressed second segment.
5. A method for storing compressed data, comprising:
breaking a data stream, a data block, or a data file into one or more segments, wherein one segment of the one or more segments is used to reconstruct more than one data streams, data blocks, or data files;
compressing each of the one or more segments into one or more compressed segments;
receiving one of the one or more compressed segments;
receiving metadata information associated with the compressed segment, wherein the received metadata information includes a segment identifier and compression metadata information, wherein the compression metadata information includes a compression type and a compression state, wherein the compression state comprises a state used for compressing the compressed segment, wherein the compression state comprises a key;
determining whether the received compressed segment has been previously stored based at least in part on a comparison of the segment identifier associated with the received compressed segment and at least one segment identifier associated with a previously stored compressed segment;
in a first event that the received compressed segment is determined to have been previously stored, storing a reference to the previously stored compressed segment;
in a second event that the received compressed segment is determined to have not been previously stored, storing the received compressed segment, wherein the storing of the received compressed segment further comprises generating an index entry from the received compressed segment, wherein the index entry comprises one or more of the following: a digital fingerprint, SHA-1 hash, or Rabin Hash.
6. A method as in claim 5 , wherein the breaking of the data stream, the data block, or the data file is based at least in part on one of the following: a content-based technique or a non-content-based technique.
7. A method as in claim 5 , wherein the compression of the each of the one or more segments comprises compressing using at least one of: a lossless compression system, a Huffman coding system, or a Lempel-Ziv Welch coding system.
8. A method as in claim 5 , wherein for a first segment and a second segment that is identical to the first segment, a compressed first segment is identical to a compressed second segment.
9. A computer program product for storing compressed data, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
breaking a data stream, a data block, or a data file into one or more segments, wherein one segment of the one or more segments is used to reconstruct more than one data streams, data blocks, or data files;
compressing each of the one or more segments into one or more compressed segments;
receiving one of the one or more compressed segments;
receiving metadata information associated with the compressed segment, wherein the received metadata information includes a segment identifier and compression metadata information, wherein the compression metadata information includes a compression type and a compression state, wherein the compression state comprises a state used for compressing the compressed segment, wherein the compression state comprises a key;
determining whether the received compressed segment has been previously stored based at least in part on a comparison of the segment identifier associated with the received compressed segment and at least one segment identifier associated with a previously stored compressed segment;
in a first event that the received compressed segment is determined to have been previously stored, storing a reference to the previously stored compressed segment;
in a second event that the received compressed segment is determined to have not been previously stored, storing the received compressed segment, wherein the storing of the received compressed segment further comprises generating an index entry from the received compressed segment, wherein the index entry comprises one or more of the following: a digital fingerprint, SHA-1 hash, or Rabin Hash.
10. A computer program product as in claim 9 , wherein the breaking of the data stream, the data block, or the data file is based at least in part on one of the following: a content-based technique or a non-content-based technique.
11. A computer program product as in claim 9 , wherein the compression of the each of the one or more segments comprises compressing using at least one of: a lossless compression system, a Huffman coding system, or a Lempel-Ziv Welch coding system.
12. A computer program product as in claim 9 , wherein for a first segment and a second segment that is identical to the first segment, a compressed first segment is identical to a compressed second segment.