Dictionary synchronization over a transport with guaranteed ordering

1. A method for transporting chunks over a transport using guaranteed ordering, the method comprising:

storing an index on a transmitter side and storing a dictionary on a receiver side;

receiving a chunk for transmission from a transmitter to a receiver;

accessing the index to determine whether the chunk is known to the receiver;

when the chunk is known to the receiver:

transmitting the address of the chunk to the receiver without transmitting the chunk;

updating the index to reflect that the chunk has been used recently and is not stale for eviction purposes, wherein the receiver receives the address and updates the dictionary for eviction purposes;

evicting stale entries in the index by the transmitter; and

evicting stale entries in the dictionary by the receiver;

when the chunk is not known to the receiver:

deterministically determining by the transmitter, when the chunk is not known to the receiver, an address of the chunk;

storing the address in the index;

updating the index to reflect that the chunk is not stale for eviction purposes;

evicting stale entries in the index by the transmitter;

transmitting the chunk that is not known to the receiver to the receiver without conveying the address of the chunk over the transport;

deterministically determining a dictionary address of the chunk by the receiver;

storing the chunk in the dictionary at the deterministically determined dictionary address, wherein the dictionary address is the same as the address in the index; and

updating the dictionary by the receiver to reflect that the chunk is not stale for eviction purposes; and

evicting stale entries in the dictionary by the receiver.

2. The method of claim 1 , further comprising:

accessing the index based on the determined address; and

determining that the chunk is known to the receiver when the determined address is present in the index.

3. The method of claim 1 , further comprising transmitting only the address of the chunk when determining that the chunk is known to the receiver.

4. The method of claim 1 , further comprising transmitting the chunk or the address to the receiver with guaranteed packet ordering.

5. The method of claim 1 , further comprising deterministically determining the dictionary address of the chunk such that the address calculated by the transmitter is the same as the dictionary address calculated by the receiver.

6. The method of claim 1 , further comprising entering the address into the index and into the dictionary without requiring the receiver to transmit the address to the transmitter.

7. The method of claim 1 , further comprising evicting the stale entries from the index and/or from the dictionary according to a predetermined strategy, wherein the index comprises a cache memory or other memory and wherein the dictionary comprises a cache memory or other memory.

8. The method of claim 1 , further comprising, when only the address of the chunk is transmitted by the transmitter, accessing the dictionary to retrieve the chunk at the address and providing the chunk to an application on the receiver side.

9. A method for a transmitter and a receiver to transmit data between a first application and a second application using guaranteed ordering, wherein the transmitter is associated with an index and the receiver is associated with a dictionary, the method comprising:

transmitting, for each data chunk processed by the transmitter, either the data chunk itself or its address to the receiver;

updating the index, for each data chunk or address transmitted to the receiver, for eviction purposes;

receiving the data chunks or addresses of data chunks as entries in an input queue at the receiver;

accessing the input queue to obtain an entry;

determining that the entry contains a new data chunk from the first application;

calculating a dictionary address, by the receiver, for the new data chunk;

updating the dictionary at the transmitter with the dictionary address, by the transmitter, for eviction purposes such that the new data chunk is not stale;

evicting stale entries from the index by the receiver;

entering the new data chunk into the dictionary at the calculated dictionary address, wherein the transmitter deterministically calculates the same dictionary address such that the dictionary address is entered into the index without requiring the receiver to communicate the dictionary address to the transmitter;

updating the dictionary to reflect that the new data chunk is not stale for eviction purposes;

when the input queue contains a second entry that only includes an address of a second chunk, updating the dictionary to reflect that that second chunk has been recently used and is not stale for eviction purposes; and

outputting the new data chunk to the second application.

10. The method of claim 9 , further comprising, when the second entry from the input queue only includes an address, outputting the second chunk, which is stored at the address in the dictionary, to the second application.

11. The method of claim 9 , further comprising receiving a data transport from the transmitter over a transport that provides guaranteed packet ordering.

12. The method of claim 9 , further comprising evicting the stale chunks and/or stale addresses from the dictionary and/or the index according to a predetermined eviction strategy.

13. The method of claim 9 , further comprising synchronizing the index with the dictionary by deterministically determining the dictionary address of the new data chunk at both the transmitter and the receiver.

14. The method of claim 13 , wherein the dictionary address determined by the transmitter and the receiver is computed using the same inputs and the same algorithm at both the transmitter and the receiver.

15. The method of claim 14 , wherein a data chunk considered for transmission by the transmitter is known to the receiver when the dictionary address is present in the index.

16. The method of claim 9 , further comprising de-duplicating the data by only sending the dictionary address when the data chunk is known to the receiver.