Incremental replication between foreign system dataset stores
Incremental replication between foreign system dataset stores is described herein. A method as described herein can include connecting to a target storage system to facilitate an initial transfer of a dataset from the system to the target storage system, wherein the system is distinct from the target storage system; in response to determining that a first version of the dataset is present on the target storage system, comparing the first version of the dataset to a second version of the dataset stored on the system; and, in response to determining that the second version of the dataset is more recent than the first version of the dataset, transferring a differential between the first version of the dataset and the second version of the dataset to the target storage system, the differential comprising less than all of the second version of the dataset.
1. A file storage system, comprising:
a memory that stores executable components; and
a processor that executes the executable components stored in the memory, wherein the executable components comprise:
a handshake component that establishes a connection with a cloud storage system, in which data is stored in an object format that is distinct from a file format used by the file storage system, to facilitate an initial transfer of a dataset from the file storage system to the cloud storage system;
a dataset analysis component that, in response to the cloud storage system indicating via the connection that a first version of the dataset is stored in the object format at the cloud storage system, compares the first version of the dataset to a second version of the dataset as stored in the file format on the file storage system; and
an incremental transfer component that, in response to the dataset analysis component determining that the second version of the dataset is more recent than the first version of the dataset, creates, in the object format, a differential between the first version of the dataset and the second version of the dataset and transfers the differential to the cloud storage system, wherein the differential comprises less than all of the second version of the dataset.
2. The file storage system of claim 1 , wherein the executable components further comprise:
a dataset management component that assigns a dataset identifier to the second version of the dataset, wherein the dataset identifier comprises a dataset tree identifier and a revision identifier.
3. The file storage system of claim 2 , wherein the dataset identifier is a second dataset identifier comprising a second dataset tree identifier and a second revision identifier, and wherein the dataset analysis component receives a first dataset identifier associated with the first version of the dataset from the cloud storage system, the first dataset identifier comprising a first dataset tree identifier and a first revision identifier.
4. The file storage system of claim 1 , wherein the dataset analysis component compares the first version of the dataset as stored on the cloud storage system to the second version of the dataset further in response to determining that the first version of the dataset is present in a dataset history corresponding to the dataset as maintained by the file storage system.
5. The file storage system of claim 4 , wherein the dataset history is a first dataset history, and wherein the dataset analysis component compares the first version of the dataset to the second version of the dataset by comparing the first dataset history to a second dataset history corresponding to the dataset as maintained by the cloud storage system.
6. The file storage system of claim 1 , wherein the second version of the dataset comprises a group of files, and wherein the executable components further comprise:
a file management component that assigns respective global file identifiers to respective files of the group of files.
7. The file storage system of claim 6 , wherein the executable components further comprise:
a mapping component that maps the respective global file identifiers to respective local file identifiers associated with the file storage system.
8. The file storage system of claim 6 , wherein the incremental transfer component creates the differential by determining global file identifiers for respective modified files of the group of files that have been altered between the first version of the dataset and the second version of the dataset and adding the respective modified files of the group of files to the incremental according to their respective global file identifiers.
9. A method, comprising:
connecting, by a file storage system operatively coupled to a processor, to a cloud storage system, in which data is stored in an object format that is distinct from a file format used by the file storage system, to facilitate an initial transfer of a dataset from the file storage system to the cloud storage system;
in response to determining that a first version of the dataset is stored in the object format on the cloud storage system, comparing, by the file storage system, the first version of the dataset to a second version of the dataset stored in the file format on the file storage system; and
in response to determining that the second version of the dataset is more recent than the first version of the dataset, generating, by the file storage system in the object format, a differential between the first version of the dataset and the second version of the dataset, the differential comprising less than all of the second version of the dataset, and transferring, by the file storage system, the differential to the cloud storage system.
10. The method of claim 9 , further comprising:
receiving, by the system, a first dataset identifier associated with the first version of the dataset from the cloud storage system, the first dataset identifier comprising a first dataset tree indicator and a first revision indicator; and
assigning, by the file storage system, a second dataset identifier to the second version of the dataset, the second dataset identifier comprising a second dataset tree indicator and a second revision indicator.
11. The method of claim 9 , wherein the comparing comprises comparing the first version of the dataset to the second version of the dataset further in response to determining that the first version of the dataset is present in a dataset history corresponding to the dataset as maintained by the file storage system.
12. The method of claim 11 , wherein the dataset history is a first dataset history, and wherein the comparing further comprises comparing the first dataset history to a second dataset history corresponding to the dataset as maintained by the cloud storage system.
13. The method of claim 9 , wherein the second version of the dataset comprises respective files, and wherein the method further comprise:
assigning, by the file storage system, respectively assigning respective global file identifiers to the respective files.
14. The method of claim 13 , further comprising:
mapping, by the file storage system, the respective global file identifiers to respectively corresponding local file identifiers associated with the file storage system.
15. A non-transitory machine-readable medium comprising computer executable instructions that, when executed by a processor of a file storage system, facilitate performance of operations, the operations comprising:
initiating an initial transfer of a dataset from the file storage system to a cloud storage system that is distinct from the file storage system and stores data in an object format that is distinct from a file format used by the file storage system;
in response to the cloud storage system indicating that a first revision of the dataset is present in the object format on the cloud storage system, comparing the first revision of the dataset to a second revision of the dataset as stored in the file format on the file storage system; and
in response to the first revision of the dataset being earlier than the second revision of the dataset, generating, in the object format, a differential between the first revision of the dataset and the second revision of the dataset, wherein the differential comprises less than all of the dataset, and transferring the differential to the cloud storage system.
16. The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:
receiving a first dataset tree identifier and a first revision identifier associated with the first revision of the dataset from the cloud storage system; and
associating a second dataset tree identifier and a second revision identifier to the second revision of the dataset as stored on the file storage system.
17. The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:
comparing the first revision of the dataset to the second revision of the dataset, further in response to the first revision of the dataset being present in a first dataset history corresponding to the dataset as maintained by the file storage system by comparing the first dataset history to a second dataset history corresponding to the dataset as maintained by the cloud storage system.
18. The non-transitory machine-readable medium of claim 15 , wherein the second version of the dataset comprises respective files, and wherein the operations further comprise:
assigning respective global file identifiers to the respective files of the dataset.
19. The file storage system of claim 1 , wherein the incremental transfer component creates the differential in the object format by generating the differential in the file format and converting the differential from the file format to the object format.
20. The file storage system of claim 6 , wherein the differential comprises a data object in the object format, and wherein the incremental transfer component assigns a name to the data object based on a global file identifier, of the respective global file identifiers, corresponding to the data object.