System and method for performing compression/encryption on data such that the number of duplicate blocks in the transformed data is increased

1. A method for applying desired transformations to data such that the number of duplicate chunks in the transformed data is increased and the chunks are predominantly of a fixed size, the method comprising:

(a) selecting a group of data from a set of data, the selected group of data comprising a beginning and an end and being continuous between the beginning and the end of the selected group of data and such that a working chunk of data generated from the selected group of data is of size equal to a predetermined size;

(b) generating the working chunk of data from the selected group of data by performing at least one of a compression, an encryption and an encoding of the selected group of data;

(c) determining whether the working chunk comprises a duplicate chunk of data;

(d) when the working chunk does not comprise a duplicate chunk of data, selecting a next group of data from the set of data, the next selected group of data comprising a beginning and an end and being continuous between the beginning and the end of the next selected group of data, the beginning of the next group of data being a predetermined number of data units after the beginning of the current selected group of data, and returning to (b); and

(e) when the working chunk comprises a duplicate chunk of data, dividing the data of the data set between the end of the last selected group of data that generated a working chunk that was a duplicate chunk of data and the beginning of the current selected group of data into one or more unique chunks.

2. The method according to claim 1 , wherein the data units comprise one of a bit, a byte and a word.

3. The method according to claim 1 , wherein the working chunk of data is considered a duplicate chunk of data if it is likely to be identical to a chunk of data that has already been processed and classified.

4. The method according to claim 1 , wherein determining whether the working chunk of data comprises a duplicate chunk of data comprises:

computing a mathematical value based on the working chunk of data; and

comparing the mathematical value to contents of a data structure.

5. The method according to claim 4 , wherein the data structure comprises a hash table.

6. The method according to claim 1 , wherein determining whether the working chunk of data comprises a duplicate chunk of data is based on a checksum generated from the working chunk of data.

7. The method according to claim 1 , wherein determining whether the working chunk of data comprises a duplicate chunk of data is based on a cryptographic hash of the working chunk of data.

8. The method according to claim 1 , wherein determining whether the working chunk of data comprises a duplicate chunk of data is based on a comparison of the working chunk of data to previously processed and classified chunks of data.

9. The method according to claim 1 , further comprising storing each unique chunk of data.

10. The method according to claim 1 , wherein selecting a next group of data in step (d) comprises:

locating a marker in a working chunk of data generated from the next group of data;

determining whether the marker has been seen before;

retrieving a remembered offset for the marker;

shifting the location of the beginning of the next group of data by an amount based on the remembered offset for the marker and its current offset from an edge of the working chunk of data; and

returning to (b).

11. The method according to claim 10 , wherein determining whether the marker has been seen before comprises:

computing a mathematical value of the data around the marker; and

comparing the mathematical value to contents of a data structure.

12. The method according to claim 11 , wherein the data structure comprises a maximum predetermined size, and

the method further comprising removing an oldest value from the data structure when storing a new mathematical value in the data structure causes the data structure to exceed the maximum predetermined size.

13. The method according to claim 10 , further comprising storing the mathematical value for a marker in the data structure when the marker has not been seen before.

14. The method according to claim 13 , wherein storing the mathematical value stores the mathematical value for a predetermined period of time.

15. The method according to claim 1 , wherein selecting a next group of data in step (d) comprises:

locating a marker in the next group of data;

determining whether the marker has been seen before;

retrieving a remembered offset for the marker;

shifting the location of the beginning of the next group of data by an amount based on the remembered offset for the marker and its current offset from an edge of the next group of data; and

returning to (b).

16. The method according to claim 1 , wherein selecting a next group of data in step (d) comprises:

locating a marker in a working chunk of data generated from the next group of data;

computing a mathematical function of data around the marker;

determining whether a remembered offset is greater than or equal to a current offset of the marker from an edge of the working chunk of data when the computed mathematical function of data around the marker has been previously seen;

setting a number y to be equal to the predetermined size minus a quantity of a remembered offset minus the current offset of the marker in the working block of data when the remembered offset is greater than or equal to the current offset of the marker, the number y being greater than or equal to zero;

setting the number y to be equal to the remembered offset minus the current offset of the marker in the working block of data when the remembered offset is less than the current offset of the marker;

setting the number y to be equal to the offset of the marker when the computed mathematical function of data around the marker has not been previously seen;

setting the number y to be equal to the predetermined size when the working chunk does not contain the marker;

setting a number x equal to an offset in the untransformed set of data corresponding to the offset of the number y in the working chunk of data, the number x being greater than or equal to zero;

shifting the location of the beginning of the next group of data by x data units; and

returning to (b).

17. The method according to claim 1 , wherein selecting a next group of data in step (d) comprises:

locating a marker in the next group of data;

computing a mathematical function of data around the marker;

determining whether a remembered offset is greater than or equal to a current offset of the marker when the computed mathematical function of data around the marker has been previously seen;

setting a number x to be equal to the predetermined size minus a quantity of a remembered offset minus the current offset of the marker in the next group of data when the remembered offset is greater than or equal to the current offset of the marker from an edge of the next group of data, the number x being greater than or equal to zero;

setting the number x to be equal to the remembered offset minus the current offset of the marker in the next group of data when the remembered offset is less than the current offset of the marker;

setting the number x to be equal to the offset of the marker when the computed mathematical function of data around the marker has not been previously seen;

setting the number x to be equal to the predetermined size when the next group of data does not contain the marker; shifting the location of the beginning of the next group of data by x data units; and returning to (b).

18. A program storage device, comprising:

a storage area; and information stored in the storage area, the information being readable by a machine and tangibly embodying a program of instructions executable by the machine for performing method steps for applying desired transformations to data such that the number of duplicate chunks in the transformed data is increased and the chunks are predominantly of a fixed size, the method comprising:

(a) selecting a group of data from the set of data, the selected group of data comprising a beginning and an end and being continuous between the beginning and the end of the selected group of data and such that a working chunk of data generated from the selected group of data is of size equal to a predetermined size;

(b) generating the working chunk of data from the selected group of data; by performing at least one of compression, an encoding of the selected group of data;

(c) determining whether the working chunk is a duplicate chunk of data;

(d) when the working chunk is not a duplicate chunk of data, selecting a next group of data from the set of data, the next selected group of data comprising a beginning and an end and being continuous between the beginning and the end of the next selected group of data, the beginning of the next group of data being a predetermined number of data units after the beginning of the current selected group of data, and returning to (b);

(e) when the working chunk comprises a duplicate chunk of data, dividing the data of the data set between the end of the last selected group of data that generated a working chunk that was a duplicate chunk of data and the beginning of the current selected group of data into one or more unique chunks.