Method for optimizing WAN traffic with deduplicated storage

1. A computer-implemented method comprising:

caching, in one or more transmitted data files (TDFs) in a deduplicated manner by a local proxy, a plurality of chunks of one or more streams that have been transmitted to a remote proxy, each of the one or more streams being identified by a stream identifier (ID);

for each of the one or more streams, maintaining a stream object having one or more TDF references, each TDF reference corresponding to at least a segment of the stream, wherein each TDF reference includes information identifying a file location within one of the TDFs at which the segment of the stream is located; and

maintaining a chunk index having a plurality of chunk index entries corresponding to a plurality of deduplicated chunks of the one or more streams, wherein each chunk index entry includes information identifying a stream location within one of the one or more streams at which a corresponding chunk is located.

2. The method of claim 1 , wherein each TDF reference includes a TDF ID identifying one of the TDFs and a file offset within that TDF at which the segment of the stream is located.

3. The method of claim 2 , wherein each chunk index entry includes the stream ID identifying one of the one or more streams and a stream offset within that stream at which a corresponding chunk is located.

4. The method of claim 3 , further comprising:

in response to receiving a flow of data to be transmitted to the remote proxy, allocating a new stream object identified by a new stream ID that identifies a new stream representing at least a portion of the flow;

chunking the flow of data into a plurality of chunks in sequence;

identifying one or more contiguous chunks that have been previously stored in one of the TDFs; and

adding a new TDF reference in the new stream object having a TDF ID and a TDF offset identifying a location of one of the TDFs storing the one or more contiguous chunks.

5. The method of claim 4 , wherein identifying one or more contiguous chunks that have been previously stored in one of the TDFs comprises:

for each of the plurality of chunks, hashing content of the chunk to generate a fingerprint;

performing a lookup in the chunk index based on the fingerprint to determine whether the chunk has been previously stored in one of the TDFs;

if a matched chunk index entry is found, retrieving a stream ID and a stream offset from the matched chunk index entry;

performing a lookup in a stream object associated with the retrieved stream ID to determine a TDF ID and a file offset identifying a file location of a TDF at which the chunk is located; and

populating the new TDF reference based on the determined TDF ID and the file offset.

6. The method of claim 5 , further comprising:

if no matched chunk index entry is found, appending the chunk in an existing TDF or in a new TDF and recording a file offset of the chunk;

updating the new TDF reference to include a corresponding TDF ID and the file offset associated with the chunk; and

adding a new chunk index entry for the chunk including the new stream ID of the new stream.

7. The method of claim 1 , wherein each of the chunk index entries further includes a direction indicator indicating whether the corresponding chunk has been received from or transmitted to the remote proxy in a stream identified by a stream ID of the chunk index entry.

8. The method of claim 7 , wherein each of the chunk index entries further includes a peer ID identifying the remote proxy, which is one of a plurality of remote proxies the local proxy communicates with concurrently.

9. The method of claim 1 , wherein each of the stream objects further includes a direction indicator indicating whether a corresponding stream has been received from or transmitted to the remote proxy in a stream identified by a stream ID of the stream object.

10. The method of claim 9 , wherein each of the stream objects further includes a peer ID identifying the remote proxy, which is one of a plurality of remote proxies the local proxy communicates with concurrently.

11. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations comprising:

caching, in one or more transmitted data files (TDFs) in a deduplicated manner by a local proxy, a plurality of chunks of one or more streams that have been transmitted to a remote proxy, each of the streams being identified by a stream identifier (ID);

for each of the one or more streams, maintaining a stream object having one or more TDF references, each TDF reference corresponding to at least a segment of the stream, wherein each TDF reference includes information identifying a file location within one of the TDFs at which the segment of the stream is located; and

maintaining a chunk index having a plurality of chunk index entries corresponding to a plurality of deduplicated chunks of the one or more streams, wherein each chunk index entry includes information identifying a stream location within one of the one or more streams at which a corresponding chunk is located.

12. The medium of claim 11 , wherein each TDF reference includes a TDF ID identifying one of the TDFs and a file offset within that TDF at which the segment of the stream is located.

13. The medium of claim 12 , wherein each chunk index entry includes the stream ID identifying one of the one or more streams and a stream offset within that stream at which a corresponding chunk is located.

14. The medium of claim 13 , wherein the method further comprises:

in response to receiving a flow of data to be transmitted to the remote proxy, allocating a new stream object identified by a new stream ID that identifies a new stream representing at least a portion of the flow;

chunking the flow of data into a plurality of chunks in sequence;

identifying one or more contiguous chunks that have been previously stored in one of the TDFs; and

adding a new TDF reference in the new stream object having a TDF ID and a TDF offset identifying a location of one of the TDFs storing the one or more contiguous chunks.

15. The medium of claim 14 , wherein identifying one or more contiguous chunks that have been previously stored in one of the TDFs comprises:

for each of the plurality of chunks, hashing content of the chunk to generate a fingerprint;

performing a lookup in the chunk index based on the fingerprint to determine whether the chunk has been previously stored in one of the TDFs;

if a matched chunk index entry is found, retrieving a stream ID and a stream offset from the matched chunk index entry;

performing a lookup in a stream object associated with the retrieved stream ID to determine a TDF ID and a file offset identifying a file location of a TDF at which the chunk is located; and

populating the new TDF reference based on the determined TDF ID and the file offset.

16. The medium of claim 15 , wherein the method further comprises:

if no matched chunk index entry is found, appending the chunk in an existing TDF or in a new TDF and recording a file offset of the chunk;

updating the new TDF reference to include a corresponding TDF ID and the file offset associated with the chunk; and

adding a new chunk index entry for the chunk including a stream ID of the new stream.

17. The medium of claim 11 , wherein each of the chunk index entries further includes a direction indicator indicating whether the corresponding chunk has been received from or transmitted to the remote proxy in a stream identified by a respective stream ID of the chunk index entry.

18. The medium of claim 17 , wherein each of the chunk index entries further includes a peer ID identifying the remote proxy, which is one of a plurality of remote proxies the local proxy communicates with concurrently.

19. The medium of claim 11 , wherein each of the stream objects further includes a direction indicator indicating whether the corresponding stream has been received from or transmitted to the remote proxy in a stream identified by a respective stream ID of the stream object.

20. The medium of claim 19 , wherein each of the stream objects further includes a peer ID identifying the remote proxy, which is one of a plurality of remote proxies the local proxy communicates with concurrently.

21. A data processing system, comprising:

a processor; and

a memory coupled to the processor for storing instructions, which when executed by the processor, cause the processor to

cache, in one or more transmitted data files (TDFs) in a deduplicated manner by a local proxy represented by the data processing system, a plurality of chunks of one or more streams that have been transmitted to a remote proxy, each of the one or more streams being identified by a stream identifier (ID),

for each of the one or more streams, maintain a stream object having one or more TDF references, each TDF reference corresponding to at least a segment of the stream, wherein each TDF reference includes information identifying a file location within one of the TDFs at which the segment of the stream is located, and

maintain a chunk index having a plurality of chunk index entries corresponding to a plurality of deduplicated chunks of the one or more streams, wherein each chunk index entry includes information identifying a stream location within one of the one or more streams at which a corresponding chunk is located.

22. The system of claim 21 , wherein each TDF reference includes a TDF ID identifying one of the TDFs and a file offset within that TDF at which the segment of the stream is located.

23. The system of claim 22 , wherein each chunk index entry includes a respective stream ID identifying one of the one or more streams and a stream offset within that stream at which a corresponding chunk is located.

24. The system of claim 23 , wherein the processor is further to:

in response to receiving a flow of data to be transmitted to the remote proxy, allocate a new stream object identified by a new stream ID that identifies a new stream representing at least a portion of the flow;

chunk the flow of data into a plurality of chunks in sequence;

identify one or more contiguous chunks that have been previously stored in one of the TDFs; and

add a new TDF reference in the new stream object having a TDF ID and a TDF offset identifying a location of one of the TDFs storing the one or more contiguous chunks.

25. The system of claim 24 , wherein identifying one or more contiguous chunks that have been previously stored in one of the TDFs comprises:

for each of the plurality of chunks, hashing content of the chunk to generate a fingerprint;

performing a lookup in the chunk index based on the fingerprint to determine whether the chunk has been previously stored in one of the TDFs;

if a matched chunk index entry is found, retrieving a stream ID and a stream offset from the matched chunk index entry;

performing a lookup in a stream object associated with the retrieved stream ID to determine a TDF ID and a file offset identifying a file location of a TDF at which the chunk is located; and

populating the new TDF reference based on the determined TDF ID and the file offset.