Method and system for enabling deduplication for asynchronous replication in a storage system
Techniques are used for enabling deduplication for asynchronous replication in a storage system. The techniques may be used to provide, among other things, retrieving, on a source system, a virtual index key corresponding to a logical address, and determining whether a deduplication index table for a target system includes an entry with the same virtual index key.
1. A method for reducing bandwidth consumed during asynchronous replication of deduplicated data in a storage system, the method comprising:
identifying unique data on a source storage system of the storage system during a deduplication process using a virtual index stored on the source storage system, wherein the virtual index stored on the source storage system comprises a virtual index key;
retrieving, on the source storage system, a first virtual index key corresponding to a pointer stored at a logical address on the source storage system;
determining whether data associated with the logical address on the source storage system has been transferred to a target storage system by searching a deduplication index table stored on the target storage system for an entry that matches the virtual index key stored on the source storage system, wherein deduplicated data at a logical address on the target storage system of the storage system is replaced with the pointer stored at the logical address on the source storage system.
2. The method of claim 1 , further comprising:
upon determining that the deduplication index table does not include the virtual index key, augmenting the deduplication index table to include the virtual index key and the next logical address on the target system to be processed in the asynchronous replication.
3. The method of claim 2 , further comprising:
sending, to the target system, data to store in association with the next logical address to be processed in the asynchronous replication.
4. The method of claim 1 , further comprising:
upon determining that the deduplication index table includes the virtual index key, (1) retrieving the logical address in the deduplication index table corresponding to the virtual index key, and (2) instructing the target system to perform a metadata copy from the retrieved logical address to the next logical address on the target system to be processed in the asynchronous replication.
5. The method of claim 1 , wherein retrieving the virtual index key comprises:
retrieving a virtual address referenced by the pointer, the virtual address associated with an entry in the virtual index.
6. A system for reducing bandwidth consumed during asynchronous replication of deduplicated data in a storage system, the system comprising a processor configured to:
identify unique data on a source storage system of the storage system during a deduplication process using a virtual index stored on the source storage system, wherein the virtual index stored on the source storage system comprises a virtual index key;
retrieve, on the source storage system, a first virtual index key corresponding to a pointer stored at a logical address on the source storage system;
determine whether data associated with the logical address on the source storage system has been transferred to a target storage system by searching a deduplication index table stored on the target storage system for an entry that matches the virtual index key stored on the source storage system, wherein deduplicated data at a logical address on the target storage system of the storage system is replaced with the pointer stored at the logical address on the source storage system.
7. The system of claim 6 , wherein the processor is further configured to:
upon determining that the deduplication index table does not include the virtual index key, augment the deduplication index table to include the virtual index key and the next logical address on the target system to be processed in the asynchronous replication.
8. The system of claim 7 , wherein the processor is further configured to:
send, to the target system, data to store in association with the next logical address to be processed in the asynchronous replication.
9. The system of claim 6 , wherein the processor is further configured to:
upon determining that the deduplication index table includes the virtual index key, (1) retrieve the logical address in the deduplication index table corresponding to the virtual index key, and (2) instruct the target system to perform a metadata copy from the retrieved logical address to the next logical address on the target system to be processed in the asynchronous replication.
10. The system of claim 6 , wherein the processor is further configured to:
retrieve a virtual address referenced by the pointer, the virtual address associated with an entry in the virtual index.
11. The method of claim 1 , further comprising:
traversing, on the source system, a set of logical addresses corresponding to deduplicated data to replicate to the target system, the set of logical addresses including data and pointers associated with deduplicated data.
12. The method of claim 1 , wherein the virtual address is associated with a pointer to a physical address of unique data stored on the source system.
13. The method of claim 4 , wherein the logical address in the deduplication index table corresponding to the virtual index key is a logical address on the target system where unique data has been stored.
14. The system of claim 6 , wherein the processor is further configured to:
traverse a set of logical addresses corresponding to deduplicated data to replicate to the target system, the set of logical addresses including data and pointers associated with deduplicated data.
15. The system of claim 6 , wherein the virtual address is associated with a pointer to a physical address of unique data stored on the source system.
16. The system of claim 6 , wherein the logical address in the deduplication index table corresponding to the virtual index key is a logical address on the target system where unique data has been stored.
17. A non-transitory computer readable media storing instructions executable by one or more processors to perform operations comprising:
identifying unique data on a source storage system of the storage system during a deduplication process using a virtual index stored on the source storage system, wherein the virtual index stored on the source storage system comprises a virtual index key;
retrieving, on the source storage system, a first virtual index key corresponding to a pointer stored at a logical address on the source storage system;
determining whether data associated with the logical address on the source storage system has been transferred to a target storage system by searching a deduplication index table stored on the target storage system for an entry that matches the virtual index key stored on the source storage system, wherein deduplicated data at a logical address on the target storage system of the storage system is replaced with the pointer stored at the logical address on the source storage system.
18. The non-transitory computer readable media of claim 17 , further comprising instructions executable to perform:
upon determining that the deduplication index table does not include the virtual index key, augmenting the deduplication index table to include the virtual index key and the next logical address on the target system to be processed in the asynchronous replication.
19. The non-transitory computer readable media of claim 18 , further comprising instructions executable to perform:
sending, to the target system, data to store in association with the next logical address to be processed in the asynchronous replication.
20. The non-transitory computer readable media of claim 17 , further comprising instructions executable to perform:
upon determining that the deduplication index table includes the virtual index key, (1) retrieving the logical address in the deduplication index table corresponding to the virtual index key, and (2) instructing the target system to perform a metadata copy from the retrieved logical address to the next logical address on the target system to be processed in the asynchronous replication.