Systems and methods for data replication synchronization
Systems and methods are provided for storing a first data object comprising a first set of immutable components, the first data object being associated with a corresponding second data object stored by a remote replication system. A difference is determined between the first set of immutable components of the first data object and a second set of immutable components of the corresponding second data object. A subset of immutable components is identified from the first set of immutable components based on the difference. The subset of immutable components from the first set of immutable components is provided to the remote replication system over a communication network.
1. A system comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the system to perform:
storing, by a first replication system, a first data object comprising a first set of immutable components, the first data object being associated with a corresponding second data object stored by a second replication system, the corresponding second data object comprising a second set of immutable components;
determining, by the first replication system, a difference between the first set of immutable components of the first data object and the second set of immutable components of the second data object;
identifying, by the first replication system, a first subset of immutable components from the first set of immutable components based on the difference, the first subset of immutable components comprising immutable components of the first set of immutable components that do not have corresponding immutable components in the second data object;
identifying, by the first replication system, a second subset of immutable components from the first set of immutable components based on the difference, each immutable component of the second subset of immutable components corresponding to at least one immutable component of at least one third data object stored by the second replication system;
identifying, by the first replication system, a respective first identifier for each immutable component of the second subset of immutable components; and
providing, by the first replication system, the first subset of immutable components from the first set of immutable components and the respective first identifier for each immutable component of the second subset of immutable components to the second replication system over a communication network, thereby facilitating an update of the corresponding second data object by the second replication system, the update including adding the first subset of immutable components to the second data object, and the update including adding the at least one immutable component of the at least one third data object to the corresponding second data object based on a respective second identifier of each of the at least one immutable component of the at least one third data object, each of the at least one respective second identifiers determined based on the at least one respective first identifiers for the second subset of immutable components.
2. The system of claim 1 , wherein each immutable component of the first set of immutable components and each immutable component of the second set of immutable components are associated with a respective unique identifier and a respective hash value.
3. The system of claim 2 , wherein the determining a difference between the first set of immutable components of the first data object and a second set of immutable components of the corresponding second data object comprises comparing the respective unique identifiers and the respective hash values to the respective unique identifiers and the respective hash values of the corresponding second object; and determining the difference based on the comparison.
4. The system of claim 1 , wherein the providing the first subset of immutable components from the first set of immutable components to the second replication system over the communication network comprises the facilitating the update of the second data object, the update causing the second replication system to add the first subset of immutable components to the second set of immutable components.
5. The system of claim 1 , wherein the communication network comprises a low-quality communication network.
6. The system of claim 1 , wherein the first set of immutable components comprises one or more immutable components selected based on a respective file size relative to a threshold size value.
7. The system of claim 1 , wherein the first set of immutable components comprises one or more immutable components selected based on a respective predicted transfer time relative to a threshold transfer time value.
8. The system of claim 6 , wherein the threshold size value comprises a dynamic threshold size value determined based on one or more performance characteristics of the communication network.
9. The system of claim 8 , wherein the one or more performance characteristics comprise any of latency or bandwidth.
10. A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:
storing, by a first replication system, a first data object comprising a first set of immutable components, the first data object being associated with a corresponding second data object stored by a second replication system, the corresponding second data object comprising a second set of immutable components;
determining, by the first replication system, a difference between the first set of immutable components of the first data object and the second set of immutable components of the second data object;
identifying, by the first replication system, a first subset of immutable components from the first set of immutable components based on the difference, the first subset of immutable components comprising immutable components of the first set of immutable components that do not have corresponding immutable components in the second data object;
identifying, by the first replication system, a second subset of immutable components from the first set of immutable components based on the difference, each immutable component of the second subset of immutable components corresponding to at least one immutable component of at least one third data object stored by the second replication system;
identifying, by the first replication system, a respective first identifier for each immutable component of the second subset of immutable components; and
providing, by the first replication system, the first subset of immutable components from the first set of immutable components and the respective first identifier for each immutable component of the second subset of immutable components to the second replication system over a communication network, thereby facilitating an update of the corresponding second data object by the second replication system, the update including adding the first subset of immutable components to the second data object, and the update including adding the at least one immutable component of the at least one third data object to the corresponding second data object based on a respective second identifier of each of the at least one immutable component of the at least one third data object, each of the at least one respective second identifiers determined based on the at least one respective first identifiers for the second subset of immutable components.
11. The method of claim 10 , wherein each immutable component of the first set of immutable components and each immutable component of the second set of immutable components are associated with a respective unique identifier and a respective hash value.
12. The method of claim 11 , wherein the determining a difference between the first set of immutable components of the first data object and a second set of immutable components of the corresponding second data object comprises comparing the respective unique identifiers and the respective hash values to the respective unique identifiers and the respective hash values of the corresponding second object; and determining the difference based on the comparison.
13. The method of claim 10 , wherein the providing the first subset of immutable components from the first set of immutable components to the second replication system over the communication network comprises the facilitating the update of the second data object, the update causing the second replication system to add the first subset of immutable components to the second set of immutable components.
14. The method of claim 10 , wherein the communication network comprises a low-quality communication network.
15. The method of claim 10 , wherein the first set of immutable components comprises one or more immutable components selected based on a respective file size relative to a threshold size value.
16. The method of claim 10 , wherein the first set of immutable components comprises one or more immutable components selected based on a respective predicted transfer time relative to a threshold transfer time value.
17. The method of claim 15 , wherein the threshold size value comprises a dynamic threshold size value determined based on one or more performance characteristics of the communication network.
18. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:
storing, by a first replication system, a first data object comprising a first set of immutable components, the first data object being associated with a corresponding second data object stored by a second replication system, the corresponding second data object comprising a second set of immutable components;
determining, by the first replication system, a difference between the first set of immutable components of the first data object and the second set of immutable components of the second data object;
identifying, by the first replication system, a first subset of immutable components from the first set of immutable components based on the difference, the first subset of immutable components comprising immutable components of the first set of immutable components that do not have corresponding immutable components in the second data object;
identifying, by the first replication system, a second subset of immutable components from the first set of immutable components based on the difference, each immutable component of the second subset of immutable components corresponding to at least one immutable component of at least one third data object stored by the second replication system;
identifying, by the first replication system, a respective first identifier for each immutable component of the second subset of immutable components; and
providing, by the first replication system, the first subset of immutable components from the first set of immutable components and the respective first identifier for each immutable component of the second subset of immutable components to the second replication system over a communication network, thereby facilitating an update of the corresponding second data object by the second replication system, the update including adding the first subset of immutable components to the second data object, and the update including adding the at least one immutable component of the at least one third data object to the corresponding second data object based on a respective second identifier of each of the at least one immutable component of the at least one third data object, each of the at least one respective second identifiers determined based on the at least one respective first identifiers for the second subset of immutable components.