Asymmetric end host redundancy elimination for networks

1. A method, comprising:

receiving, on a socket layer from an application, a new data block at a server on a computer network configured to support communication between the server and a client;

accessing a server packet cache, the server packet cache containing data blocks that have been previously sent over the computer network between the server and the client on the computer network;

comparing the new data block to individual data blocks in the server packet cache to identify contiguous strings of repeated data in the new data block;

encoding the identified contiguous strings of repeated data as pointer lookups to the server packet cache to reduce redundant data transmission over the computer network; and,

communicating with the client to adaptively tune a synchronized cache size of a client packet cache resident on the client and the server packet cache resident on the server based upon memory available at the server.

2. The method of claim 1 , further comprising:

transmitting an individual data block that includes the encoded identified contiguous strings of repeated data over the computer network to the client.

3. The method of claim 2 , further comprising:

generating a lookup table, from training data;

selecting one byte at a time from the new data block; and

determining whether an entry corresponding to the selected byte in the lookup table is set.

4. The method of claim 3 , further comprising:

in an instance where the entry is set, then designating the selected byte as a marker;

computing a fingerprint for the selected byte by hashing a contiguous window of bytes of the new data block that begins at the marker;

identifying representative regions from a stored fingerprint hash table of markers and the fingerprint;

copying the new data block to the server packet cache; and

storing the fingerprint and an offset of the marker in the server packet cache in the fingerprint hash table on the server.

5. The method of claim 4 , further comprising using an index of the fingerprint hash table to implicitly represent at least some part of the fingerprint.

6. The method of claim 4 , further comprising:

defining a sampling period;

determining that there are more bytes in the new data block;

defining a configurable number of bytes of content, where the configurable number is controlled by the sampling period;

skipping the configurable number of bytes of content in the new data block; and

selecting another byte of the new data block based on the skipping.

7. The method of claim 6 , further comprising:

determining the sampling period based on a load at the server such that in an instance where the server has a high application load then the sampling period is increased so that redundancy elimination does not affect the application.

8. The method of claim 6 , further comprising setting the configurable number of bytes of content equal to the sampling period divided by two.

9. The method of claim 8 , further comprising maintaining the server packet cache and the client packet cache as a fixed-size circular first-in first-out (FIFO) log of data blocks.

10. The method of claim 1 , further comprising:

obtaining representative regions using a fingerprinting technique; and

identifying maximal matches around the representative regions that are repeated across multiple data blocks.

11. The method of claim 10 , further comprising:

computing a fingerprint from the representative regions;

generating and maintaining a fingerprint hash table that contains fingerprints and marker offsets from previous data blocks in the server cache;

overwriting a previous entry in the fingerprint hash table with a current fingerprint and marker offset value; and

avoiding explicitly deleting stale fingerprints and marker offset values.

12. A method, comprising:

receiving on an IP layer or a socket layer from an application, a data block containing redundancies;

defining a minimum redundant string of contiguous bytes in the data block that will define redundant data, wherein the minimum redundant string is in a range from 12 bytes to 64 bytes;

processing the data block in a byte-by-byte manner to determine a location of the redundancies using the minimum redundant string of contiguous bytes;

identifying contiguous strings of redundant data in the data block by using a fingerprinting technique and a server packet cache to compare the data block and data from previous data blocks transmitted over a computer network from a server to a client;

communicating with the client to adaptively tune a synchronized cache size of a client packet cache and the server packet cache based upon memory available at the server;

encoding the identified contiguous strings of redundant data to reduce data redundancies; and

sending the encoded data to the client on the computer network.

13. The method of claim 12 , further comprising:

inputting training data to the server;

generating a lookup table from the training data;

selecting one byte at a time from the data block to obtain a selected byte; and

determining whether an entry corresponding to the selected byte in the lookup table is set for the selected byte.

14. The method of claim 13 , further comprising:

in an instance where the selected byte is set, then designating the selected byte as a marker;

computing on the server a fingerprint for the selected byte using a hash technique;

skipping a sampling period divided by two bytes of content and then selecting another byte of the data block for processing; and

using the server to identify representative regions from the markers and fingerprints.

15. The method of claim 12 , further comprising:

obtaining the previous data blocks from one of a client packet cache and a server packet cache, where the previous data blocks are payload from past traffic exchanged between the client and the server and the client packet cache and the server packet cache are synchronized with each other;

defining the synchronized cache size of both the client packet cache and the server packet cache; and

adaptively tuning the synchronized cache size based on available memory at the server such that lower server memory corresponds to a lower synchronized cache size.

16. The method of claim 12 , further comprising providing data redundancy elimination as an end system service in a computer network prior to any encryption being used on the data.

17. A method, comprising:

receiving a data block containing redundancies;

accessing a lookup table;

determining that a selected byte from the data block has an entry in the lookup table that is set for the selected byte;

designating the selected byte as a marker;

computing a first fingerprint for the selected byte using a hash technique;

storing the first fingerprint and an offset of the marker in a server packet cache in a fingerprint hash table;

skipping an amount of data equal to a half of a sampling period bytes and processing a byte at that location in the data block;

identifying representative regions from stored marker offsets and stored fingerprints in the fingerprint hash table;

identifying maximal matches around the identified representative regions that are repeated across multiple data blocks; and,

encoding the maximal matches to obtain encoded redundant data.

18. The method of claim 17 , further comprising:

defining fixed-size windows for computing second fingerprints over the data block;

selecting one of the fixed-size windows;

using the server to compute the second fingerprints over the selected fixed-size window using the identified representative regions;

generating an optimized representation of the fingerprint hash table that maps a fingerprint hash value to an offset value; and

comparing individual fingerprints for the data block with the fingerprint hash table that contains other individual fingerprints of previous data blocks.

19. The method of claim 18 , further comprising:

retrieving fingerprint matches of the previous data blocks from a client packet cache;

determining a matching region where data from the data block and the previous data blocks match; and

expanding the matching region one byte at a time in both directions to obtain a maximal region of redundant bytes.

20. The method of claim 19 , further comprising:

encoding the matching region using tuples that identify the matching region in the client packet cache;

computing hash values for the computed second fingerprints to obtain current fingerprint hash values using a hash function that generates a few bytes of hash value as one of the individual fingerprints; and

storing the current fingerprint hash values in the fingerprint hash table by overwriting a previous entry in the fingerprint hash table.