Method and system for accelerating data movement using change information concerning difference between current and previous data movements
View Patent ↗According to one embodiment, a first storage system receives a first data stream from a second storage system over a network. The first data stream includes data objects and differential object information identifying at least one data object missing from the first data stream. A difference between the first data stream and a second data stream that has been previously received is determined based on the differential object information, including identifying a data object that has been added, deleted, or modified in view of the second data stream. The first data stream is reconstructed based on the second data stream and the difference between the first data stream and the second data stream, generating a third data stream. The third data stream is stored in a persistent storage device of the first storage system, the third data stream representing a complete first data stream without a missing data object.
1. A computer-implemented method for data movement over a network, the method comprising:
receiving, at a first storage system, a first data stream from a second storage system over a network, the first data stream having a plurality of data objects, each identified by a data object identifier (ID), the first data stream including differential object information identifying at least one data object missing from the first data stream, wherein the differential object information comprises a changed object table having a plurality of entries, each entry corresponding to a data object identified by an object identifier, wherein each entry further includes an attribute indicating whether the corresponding data object has been modified, inserted, or deleted with respect to a second data stream;
determining a difference between the first data stream and the second data stream that has been previously received from the second storage system based on the differential object information, including identifying a data object of the first data stream that has been added from the second data stream, a data object of the second data stream that has been deleted, or a data object of the second data stream that has been modified in the first data stream;
generating a third data stream based on the second data stream and the difference between the first data stream and the second data stream; and
storing the third data stream in a persistent storage device of the first storage system, the third data stream representing a complete first data stream without a missing data object.
2. The method of claim 1 , further comprising:
obtaining from the differential object information differential block information representing one or more modified data blocks of one or more modified data objects, wherein the differential block information identifies one or more data blocks within a data object that have been modified with respect to a corresponding data object of the second data stream; and
adding the one or more data blocks of a modified data object from the first data stream into the third data stream.
3. The method of claim 2 , wherein the differential block information further identifies one or more blocks that have been added or deleted with respect to the second data stream.
4. The method of claim 1 , wherein the first storage system is a deduplicated storage system.
5. The method of claim 3 , wherein each entry of the changed object table associated with a modified data object further includes a reference linking to a changed block table storing one or more data blocks within the data object that have been modified, added, or deleted with respect to a corresponding data object of the second data stream.
6. The method of claim 5 , wherein the changed block table comprises a plurality of entries, each entry corresponding to a data block, including an offset of the first data stream at which the corresponding data block has been modified, added, or deleted.
7. The method of claim 6 , wherein each entry of the changed block table further includes information specifying a size of the data block, wherein information of the changed block table is utilized to generate differential block information.
8. The method of claim 1 , wherein a data object or data block that has been deleted from the second data stream is not included in the first data stream received from the second storage system.
9. The method of claim 1 , wherein a data object or data block as a part of the first data stream but also exists in the second data stream is not included in the first data stream.
10. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations of data movement over a network, the operations comprising:
receiving, at a first storage system, a first data stream from a second storage system over a network, the first data stream having a plurality of data objects, each identified by a data object identifier (ID), the first data stream including differential object information identifying at least one data object missing from the first data stream, wherein the differential object information comprises a changed object table having a plurality of entries, each entry corresponding to a data object identified by an object identifier, wherein each entry further includes an attribute indicating whether the corresponding data object has been modified, inserted, or deleted with respect to a second data stream;
determining a difference between the first data stream and the second data stream that has been previously received from the second storage system based on the differential object information, including identifying a data object of the first data stream that has been added from the second data stream, a data object of the second data stream that has been deleted, or a data object of the second data stream that has been modified in the first data stream;
generating a third data stream based on the second data stream and the difference between the first data stream and the second data stream; and
storing the third data stream in a persistent storage device of the first storage system, the third data stream representing a complete first data stream without a missing data object.
11. The non-transitory machine-readable medium of claim 10 , wherein the operations further comprise:
obtaining from the differential object information differential block information representing one or more modified data blocks of one or more modified data objects, wherein the differential block information identifies one or more data blocks within a data object that have been modified with respect to a corresponding data object of the second data stream; and
adding the one or more data blocks of a modified data object from the first data stream into the third data stream.
12. The non-transitory machine-readable medium of claim 11 , wherein the differential block information further identifies one or more blocks that have been added or deleted with respect to the second data stream.
13. The non-transitory machine-readable medium of claim 10 , wherein the first storage system is a deduplicated storage system.
14. The non-transitory machine-readable medium of claim 12 , wherein each entry of the changed object table associated with a modified data object further includes a reference linking to a changed block table storing one or more data blocks within the data object that have been modified, added, or deleted with respect to a corresponding data object of the second data stream.
15. The non-transitory machine-readable medium of claim 14 , wherein the changed block table comprises a plurality of entries, each entry corresponding to a data block, including an offset of the first data stream at which the corresponding data block has been modified, added, or deleted.
16. The non-transitory machine-readable medium of claim 15 , wherein each entry of the changed block table further includes information specifying a size of the data block, wherein information of the changed block table is utilized to generate differential block information.
17. The non-transitory machine-readable medium of claim 10 , wherein a data object or data block that has been deleted from the second data stream is not included in the first data stream received from the second storage system.
18. The non-transitory machine-readable medium of claim 10 , wherein a data object or data block as a part of the first data stream but also exists in the second data stream is not included in the first data stream.
19. A storage system, comprising:
a storage device; and
a data stream reconstructor coupled to the storage device to
receive a first data stream from a second storage system over a network, the first data stream having a plurality of data objects, each identified by a data object identifier (ID), the first data stream including differential object information identifying at least one data object missing from the first data stream, wherein the differential object information comprises a changed object table having a plurality of entries, each entry corresponding to a data object identified by an object identifier, wherein each entry further includes an attribute indicating whether the corresponding data object has been modified, inserted, or deleted with respect to a second data stream,
determine a difference between the first data stream and the second data stream that has been previously received from the second storage system based on the differential object information, including identifying a data object of the first data stream that has been added from the second data stream, a data object of the second data stream that has been deleted, or a data object of the second data stream that has been modified in the first data stream,
generate a third data stream based on the second data stream and the difference between the first data stream and the second data stream, and
store the third data stream in the storage device, the third data stream representing a complete first data stream without a missing data object.
20. The system of claim 19 , wherein the data stream reconstructor is further to
obtain from the differential object information differential block information representing one or more modified data blocks of one or more modified data objects, wherein the differential block information identifies one or more data blocks within a data object that have been modified with respect to a corresponding data object of the second data stream, and
add the one or more data blocks of a modified data object from the first data stream into the third data stream.
21. The system of claim 20 , wherein the differential block information further identifies one or more blocks that have been added or deleted with respect to the second data stream.
22. The system of claim 19 , wherein the storage system is a deduplicated storage system.
23. The system of claim 21 , wherein each entry of the changed object table associated with a modified data object further includes a reference linking to a changed block table storing one or more data blocks within the data object that have been modified, added, or deleted with respect to a corresponding data object of the second data stream.
24. The system of claim 23 , wherein the changed block table comprises a plurality of entries, each entry corresponding to a data block, including an offset of the first data stream at which the corresponding data block has been modified, added, or deleted.
25. The system of claim 24 , wherein each entry of the changed block table further includes information specifying a size of the data block, wherein information of the changed block table is utilized to generate differential block information.
26. The method of claim 1 , wherein generating a third data stream comprises:
generating a third data stream based on the second data stream, the difference between the first data stream and the second data stream, and a fourth data stream received at the first storage system from the second storage system over a network, wherein the fourth data stream contains the at least one missing data object of the first data stream, and the fourth data stream is received separately from the first data stream after confirming the first storage system has the second data stream.