Systems and methods for data replication synchronization
Systems and methods are provided for storing a first data object comprising a first set of immutable components, the first data object being associated with a corresponding second data object stored by a remote replication system. A difference is determined between the first set of immutable components of the first data object and a second set of immutable components of the corresponding second data object. A subset of immutable components is identified from the first set of immutable components based on the difference. The subset of immutable components from the first set of immutable components is provided to the remote replication system over a communication network.
1. A system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the system to perform:
storing a first data object by a first replication system comprising a first set of immutable components, the first data object being associated with a corresponding second data object stored by a remote replication system;
determining a difference between the first set of immutable components of the first data object and a second set of immutable components of the corresponding second data object;
identifying a subset of immutable components from the first set of immutable components based on the difference; and
scheduling an export action to export the subset from the first replication system that provides, over a communication network, the subset to the remote replication system, according to a scheduling criteria based on a performance load characteristic indicating a manner in which an increased load affects a performance on the communication network, the scheduling criteria further comprising:
in response to the export action being dependent upon and ordered ahead of an import action to import one or more data objects associated with the first data object to the first replication system, permitting the import action to be reordered ahead of the export action while maintaining an order of the import action with respect to other import actions to the first replication system.
2. The system of claim 1 , wherein the immutable components of the first set of immutable components and the immutable components of the second set of immutable components are associated with respective unique identifiers and a respective hash values.
3. The system of claim 2 , wherein the determining a difference between the first set of immutable components of the first data object and a second set of immutable components of the corresponding second data object comprises comparing the respective unique identifiers and the respective hash values to the respective unique identifiers and the respective hash values of the corresponding second object; and determining the difference based on the comparison.
4. The system of claim 1 , wherein the scheduling import actions and one or more export actions facilitates an update of the second data object using the subset of immutable components, the update causing the remote replication system to add the subset of immutable components to the second set of immutable components.
5. The system of claim 1 , further comprising facilitating an update of the second data object using the subset of immutable components, the facilitating comprising:
causing the remote replication system to add the subset of immutable components to the second set of immutable components, and remove the immutable components of the second set of immutable components corresponding to the subset of immutable components.
6. The system of claim 1 , wherein the first set of immutable components comprises one or more immutable components selected based on a respective file size relative to a threshold size value.
7. The system of claim 1 , wherein the first set of immutable components comprises one or more immutable components selected based on a respective predicted transfer time relative to a threshold transfer time value.
8. The system of claim 6 , wherein the threshold size value comprises a dynamic threshold size value determined based on one or more performance characteristics of the communication network.
9. The system of 8 , wherein the one or more performance characteristics comprise any of latency or bandwidth.
10. The system of claim 1 , wherein the instructions, when executed by the one or more processors, further cause the system to perform:
detecting an update to the first data object by the first replication system based on a version vector at the first replication system;
determining whether a concurrent update has been made to the first data object by any of remote replication systems based on comparisons between the version vector and version vectors of each of the remote replication systems; and
in response to determining that a concurrent update has been made, holding the first data object from replication until a correct version vector, out of the version vector and the version vectors of the remote replication systems, is identified; and
the determining a difference is in response to the correct version vector being identified.
11. The system of claim 1 , wherein the scheduling import actions and one or more export actions further comprises:
refraining from reordering a priority of another import action or the export action over the import action.
12. The system of claim 1 , wherein the scheduling criteria further comprises scheduling multiple export actions concurrently while scheduling import actions one at a time.
13. A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:
storing a first data object by a first replication system comprising a first set of immutable components, the first data object being associated with a corresponding second data object stored by a remote replication system;
determining a difference between the first set of immutable components of the first data object and a second set of immutable components of the corresponding second data object;
identifying a subset of immutable components from the first set of immutable components based on the difference; and
scheduling an export action to export the subset from the first replication system that provides, over a communication network, the subset to the remote replication system, according to a scheduling criteria based on a performance load characteristic indicating a manner in which an increased load affects a performance on the communication network, the scheduling criteria further comprising:
in response to the export action being dependent upon and ordered ahead of an import action to import one or more data objects associated with the first data object to the first replication system, permitting the import action to be reordered ahead of the export action while maintaining an order of the import action with respect to other import actions to the first replication system.
14. The method of claim 13 , wherein the immutable components of the first set of immutable components and the immutable components of the second set of immutable components are associated with respective unique identifiers and a respective hash values.
15. The method of claim 13 , wherein the determining a difference between the first set of immutable components of the first data object and a second set of immutable components of the corresponding second data object comprises comparing at the respective unique identifiers and the respective hash values to the respective unique identifiers and the respective hash values of the corresponding second object; and determining the difference based on the comparison.
16. The method of claim 13 , wherein the scheduling import actions and one or more export actions facilitates an update of the second data object using the subset of immutable components, the update causing the remote replication system to add the subset of immutable components to the second set of immutable components.
17. The method of claim 13 , further comprising facilitating an update of the second data object using the subset of immutable components, the facilitating comprising:
causing the remote replication system to add the subset of immutable components to the second set of immutable components, and remove the immutable components of the second set of immutable components corresponding to the subset of immutable components.
18. The method of claim 13 , wherein the first set of immutable components comprises one or more immutable components selected based on a respective file size relative to a threshold size value.
19. The method of claim 13 , wherein the first set of immutable components comprises one or more immutable components selected based on a respective predicted transfer time relative to a threshold transfer time value.
20. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:
storing a first data object by a first replication system comprising a first set of immutable components, the first data object being associated with a corresponding second data object stored by a remote replication system;
determining a difference between the first set of immutable components of the first data object and a second set of immutable components of the corresponding second data object;
identifying a subset of immutable components from the first set of immutable components based on the difference; and
scheduling an export action to export the subset from the first replication system that provides, over a communication network, the subset to the remote replication system, according to a scheduling criteria based on a performance load characteristic indicating a manner in which an increased load affects a performance on the communication network, the scheduling criteria further comprising:
in response to the export action being dependent upon and ordered ahead of an import action to import one or more data objects associated with the first data object to the first replication system, permitting the import action to be reordered ahead of the export action while maintaining an order of the import action with respect to other import actions to the first replication system.