Low latency replication techniques with content addressable storage
A CAS data storage method and apparatus comprising: receiving input data including a succession of data items with corresponding logical addresses at a source CAS data storage space for storage therein and for replication at a destination CAS data storage space, generating a hash key for each data item at the source storage space, comparing respective hash keys with hash keys stored at a hash key storage table, to determine whether respective further data items are already present at the destination storage device; transferring respective data items to the destination storage space if no match is made to a hash key stored at the hash key storage table, but not transferring respective further data items if a match is made to a hash key stored at the hash key storage table, thereby transferring to the destination storage space only unique data items.
1. A CAS data storage system comprising:
at least one source CAS data storage space,
a destination CAS data storage space, and
a communication line connecting said source storage space and said destination storage space, wherein input data for storage in said system arrives at said source storage space for storage in said source storage space and replication in said destination storage space, and wherein a hash key of a data item is generated at said source storage space, said source storage space comprising:
a hash key storage table configured to store respective generated hash keys in said source storage space to indicate that said respective data item is stored at said destination storage space, wherein the hash key storage table is enabled to contain multiple indexes indicating which destination storage device is associated with respective generated hash keys and data associated with each respective generated hash key, wherein the hash key storage at the source storage space is enabled to provide determination of whether data is stored at the destination storage device; and
a hash key inspection unit configured to compare hash keys stored at said hash key storage table against hash keys generated from further data items arriving at said source storage space to determine whether respective further data items are already present at said destination storage device, wherein if a match is made an associated data item is present at said destination storage device and the associated data item is not unique to said destination storage device, said source storage system configured to transfer respective further data items if no match is made to a hash key stored at said hash key storage table, but not to transfer respective further data items if a match is made to a hash key stored at said hash key storage table, thereby only transferring unique data items;
wherein the hash key inspection unit is enabled to compare hash keys and make a determination to send an associated data item without communicating with the destination storage device;
wherein said source storage space is enabled to utilize write folding on data determined to be sent to the destination storage space.
2. The CAS data storage system of claim 1 , wherein said hash key storage table is configured with an index field to hold for each stored hash key an index indicating a number of times said respective hash is currently replicated at said destination storage space, said index field being incremented or decremented respectively each time a corresponding data item is written or deleted.
3. The CAS data storage system of claim 1 , wherein:
each incoming data item is associated with a logical address, the source storage space and destination storage spaces respectively further comprising an address to hash table to store said generated hash keys together with corresponding logical addresses; and
each unique data item transferred is associated with a physical storage address at said destination storage space, said destination storage space comprising a hash to physical table to store said generated hash keys together with respective physical storage addresses.
4. The CAS data storage system of claim 3 , wherein, i f said match is made between a hash key of a respective incoming data item and a hash key in said hash key storage table so that a respective data item is not unique and is not transferred to said destination storage space, then a respective corresponding logical address is transferred to said destination storage device together with said respective hash key for storage in said destination storage space address to hash table.
5. The CAS storage system of claim 1 , wherein data transfer between said source storage space and said destination storage space is asynchronous.
6. The CAS storage system of claim 1 , wherein data transfer between said source storage space and said destination storage space is synchronous.
7. The CAS data storage system of claim 1 , wherein said source storage space is configured to generate hash keys corresponding to data items such that identical data items correspond to identical hash keys but non-identical data items correspond to non-identical hash keys.
8. A CAS data storage method comprising:
receiving input data including a succession of data items with corresponding logical addresses at a source CAS data storage space for storage therein and for replication at a destination CAS data storage space,
generating a hash key for each data item at said source storage space,
comparing respective hash keys with hash keys stored at a hash key storage table at said source storage space, to determine at said hash key storage space whether respective further data items are already present at said destination storage device, wherein if a match is made an associated data item is present at said destination storage device and the associated data item is not unique to said destination storage device; wherein the hash key storage table is enabled to contain multiple indexes indicating which destination storage device is associated with respective generated hash keys, wherein the hash key storage at the source storage space is enabled to provide determination of whether data is stored at the destination storage device;
wherein comparing respective hash keys and making a determination to send an associated data item is enabled to be completed without communicating with the destination storage device
wherein said source storage space is enabled to utilize write folding on data determined to be sent to said destination storage space; and
transferring respective data items to said destination storage space if no match is made to a hash key stored at said hash key storage table, but not transferring respective further data items if a match is made to a hash key stored at said hash key storage table, thereby transferring to said destination storage space only unique data items.
9. The CAS data storage method according to claim 8 , wherein:
if no match is made of a respective incoming data item hash key to a hash key stored at said hash key storage table, then said respective incoming data item hash key is stored at said hash key storage table along with said transferring and associated with an index, said index being set at unity to indicate that said hash key is present once at said destination storage space; and
if a match is made then a corresponding associated index is incremented, thereby to provide, for each stored hash key, an index indicating a number of times said respective hash is currently replicated at said destination storage space, said index field further being decremented each time a corresponding data item is deleted.
10. The CAS data storage method of claim 8 , further comprising providing the source storage space and destination storage spaces respectively with an address to hash table to store said generated hash keys together with corresponding logical addresses.
11. The CAS data storage method of claim 8 comprising associating each unique data item transferred to said destination storage space with a physical storage address at said destination storage space, and providing said destination storage space with a hash to physical table to store said generated hash keys together with respective physical storage addresses.
12. The CAS data storage method of claim 10 , wherein, if said match is made between a hash key of a respective incoming data item and a hash key in said hash key storage table so that a respective data item is not transferred to said destination storage space, then a respective corresponding logical address is transferred to said destination storage device together with said respective hash key for storage in said destination storage space address to hash table.
13. The CAS data storage method of claim 8 , wherein data transfer between said source storage space and said destination storage space is asynchronous.
14. The CAS data strange method of claim 8 , wherein data transfer between said source storage space and said destination storage space is synchronous.
15. The CAS data storage method of claim 8 , wherein said source storage space is configured to generate hash keys corresponding to data items such that identical data items correspond to identical hash keys but non-identical data items correspond to non-identical hash keys.