Scalable deduplication system with small blocks

1. A system for scalable data deduplication working with small data chunks in a computing environment, comprising:

at least one processor device, operated in the computing environment, wherein the at least one processor device is configured to:

for each of the small data chunks, generate a signature based on a combination of a representation of characters that appear in the small data chunk and a representation of frequencies of the characters in the small data chunk, wherein the signature is used to help in selecting data to be deduplicated,

wherein the signature is generated based on a c-spectrum of the small data chunk, an f-spectrum of the small data chunk, and a p-spectrum of the small data chunk, wherein the c-spectrum of the small data chunk is a sequence of representations of different characters ordered by their frequency of occurrence in the small data chunk, the f-spectrum of the small data chunk is a corresponding sequence of frequencies of the different characters in the small data chunk, and the p-spectrum of the small data chunk is a sequence of different character pairs ordered by their frequency of occurrence in the small data chunk.

2. The system of claim 1 , wherein generation of the signature provides for small modifications in the small data chunk to cause one of small changes or no changes in the signature.

3. The system of claim 1 , wherein the at least one processor device is further configured to, in conjunction with generating the signature based on a c-spectrum, f-spectrum and p-spectrum, perform one or more of:

combining characters of the c-spectrum having similar frequencies into blocks, by weighing gaps d i =f i −f i+1 between frequencies f i of characters that are adjacent in an ordered sequence, wherein the gaps are normalized based upon weights,

selecting a subset of the gaps for defining boundaries of blocks of characters of the c-spectrum,

performing a logical XOR operation on a combination of characters in blocks of characters of the c-spectrum,

selecting a subset of bits of a subset of frequencies of the f-spectrum, and

selecting a subset of pairs of the p-spectrum and performing logical operations on pairs of representations of characters for each of the selected subset of pairs of the p-spectrum.

4. The system of claim 3 , wherein the at least one processor device is further configured to perform shifting operations on the selected subset of bits and the selected subset of pairs and performing a logical XOR operation by combining the selected subsets into a bitstring representing a signature.

5. The system of claim 4 , wherein the at least one processor device is further configured to perform the shifting operations according to parameter tables.

6. The system of claim 3 , wherein weights w i by which the gaps d i =f i −f i+1 are normalized are

w

i

=

1

i

.

7. The system of claim 3 , wherein the logical operations performed on pairs of representations of characters include at least one of:

cyclically rotating a string of bits,

shifting the string of bits, and

performing a logical XOR operation.

8. A computer program product for scalable data deduplication working with small data chunks by a processor device, the computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion for, for each of the small data chunks, generating a signature based on a combination of a representation of characters that appear in the small data chunk and a representation of frequencies of the characters in the small data chunk, wherein the signature is used to help in selecting data to be deduplicated,

wherein the signature is generated based on a c-spectrum of the small data chunk, an f-spectrum of the small data chunk, and a p-spectrum of the small data chunk, wherein the c-spectrum of the small data chunk is a sequence of representations of different characters ordered by their frequency of occurrence in the small data chunk, the f-spectrum of the small data chunk is a corresponding sequence of frequencies of the different characters in the small data chunk, and the p-spectrum of the small data chunk is a sequence of different character pairs ordered by their frequency of occurrence in the small data chunk.

9. The computer program product of claim 8 , wherein the generation of the signature provides for small modifications in the small data chunks to cause one of small changes or no changes in the signature.

10. The computer program product of claim 8 , further including a second executable portion for, in conjunction with generating the signature based on a c-spectrum, f-spectrum and p-spectrum, performing one or more of:

combining characters of the c-spectrum having similar frequencies into blocks, by weighing gaps d i =f i −f i+1 between frequencies f i of characters that are adjacent in an ordered sequence, wherein the gaps are normalized based upon weights,

selecting a subset of the gaps for defining boundaries of blocks of characters of the c-spectrum,

performing a logical XOR operation on a combination of characters in blocks of characters of the c-spectrum,

selecting a subset of bits of a subset of frequencies of the f-spectrum, and

selecting a subset of pairs of the p-spectrum and performing logical operations on pairs of representations of characters for each of the selected subset of pairs of the p-spectrum.

11. The computer program product of claim 10 , further including a third executable portion for one of:

performing shifting operations on the selected subset of bits and the selected subset of pairs and performing a logical XOR operation by combining the selected subsets into a bitstring representing a signature, and

performing the shifting operations according to parameter tables.

12. The computer program product of claim 10 ,

wherein weights w i by which the gaps d i =f i −f i+1 are normalized are

w

i

=

1

i

.

13. The computer program product of claim 10 , wherein the logical operations performed on pairs of representations of characters include at least one of:

cyclically rotating a string of bits,

shifting the string of bits, and

performing a logical XOR operation.