IP Library Granted Patent US 11,829,383
Granted Patent B2
US 11,829,383 · App. 17/517,481 · Granted Nov 28, 2023

Systems and methods for data replication synchronization

Inventors: Stephen Freiberg (Palo Alto, CA); Alexander Landau (Palo Alto, CA); Andrew Greene (Palo Alto, CA); Brian Dorne (San Francisco, CA); Bryan Offutt (Palo Alto, CA); Ernest Zeidman (Palo Alto, CA); Ilya Nepomnyaschchiy (Mountain View, CA); John Garrod (San Mateo, CA); Katherine Brainard (East Orange, NJ); Kolin Purcell (San Francisco, CA); Michael Levin (Bethesda, MD); Simon Swanson (San Francisco, CA); Spencer Stecko (Palo Alto, CA)
Assignee: Palantir Technologies Inc.
G06F16/273G06F7/02
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,829,383
App. No.
17/517,481
Granted
Nov 28, 2023
Kind
B2
Abstract

Systems and methods are provided for storing a first data object comprising a first set of immutable components, the first data object being associated with a corresponding second data object stored by a remote replication system. A difference is determined between the first set of immutable components of the first data object and a second set of immutable components of the corresponding second data object. A subset of immutable components is identified from the first set of immutable components based on the difference. The subset of immutable components from the first set of immutable components is provided to the remote replication system over a communication network.

Claims (42)

1. A system comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, cause the system to perform:

storing a first data object by a first replication system comprising a first set of immutable components, the first data object being associated with a corresponding second data object stored by a remote replication system;

determining a difference between the first set of immutable components of the first data object and a second set of immutable components of the corresponding second data object;

identifying a subset of immutable components from the first set of immutable components based on the difference; and

scheduling a first export action or a second export action to export the subset from the first replication system that provides, over a communication network, the subset to the remote replication system, the scheduling further comprising:

determining whether the first export action or the second export action is dependent on an import action to the first replication system; and

based on the determining of whether the first export action or the second export action is dependent on the import action, and in response to the first export action being scheduled behind the second export action, selectively rescheduling the first export action or the second export action to be concurrent with each other.

2. The system of claim 1 , wherein the scheduling further comprises scheduling import actions one at a time.

3. The system of claim 1 , wherein the scheduling comprises refraining from reordering a priority of the import action or the first export action with respect to a different import action.

4. The system of claim 1 , wherein the scheduling comprises refraining from reordering the first export action to occur ahead of the second export action.

5. The system of claim 1 , wherein the scheduling of the first export action is based on a predicted saturation state of a network connection resulting from the first export action.

6. The system of claim 1 , wherein the instructions further cause the system to propagate version clock acknowledgements corresponding to both the first replication system and the remote replication system to corresponding acknowledgement version vectors stored in the first replication system, wherein each of the corresponding acknowledgement version vectors receive a respective version clock acknowledgement of the version clock acknowledgements.

7. The system of claim 1 , wherein the instructions further cause the system to propagate version clock acknowledgements corresponding to both the first replication system and the remote replication system to corresponding second acknowledgement version vectors stored in the remote replication system, wherein each of the corresponding second acknowledgement version vectors receive a respective version clock acknowledgement of the version clock acknowledgements.

8. The system of claim 1 , wherein the first set of immutable components comprises one or more immutable components selected based on a respective file size relative to a threshold size value.

9. The system of claim 8 , wherein the threshold value comprises a dynamic threshold size value determined based on one or more performance characteristics of the communication network.

10. The system of claim 9 , wherein the one or more performance characteristics comprise any of a latency or a bandwidth.

11. The system of claim 1 , wherein the first set of immutable components comprises one or more immutable components selected based on a respective predicted transfer time relative to a threshold transfer time value.

12. A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:

storing a first data object by a first replication system comprising a first set of immutable components, the first data object being associated with a corresponding second data object stored by a remote replication system;

determining a difference between the first set of immutable components of the first data object and a second set of immutable components of the corresponding second data object;

identifying a subset of immutable components from the first set of immutable components based on the difference; and

scheduling a first export action or a second export action to export the subset from the first replication system that provides, over a communication network, the subset to the remote replication system, the scheduling further comprising:

determining whether the first export action or the second export action is dependent on an import action to the first replication system; and

based on the determining of whether the first export action or the second export action is dependent on the import action, and in response to the first export action being scheduled behind the second export action, selectively rescheduling the first export action or the second export action to be concurrent with each other.

13. The method of claim 12 , wherein the scheduling further comprises scheduling import actions one at a time.

14. The method of claim 12 , wherein the scheduling comprises refraining from reordering a priority of the import action or the first export action with respect to a different import action.

15. The method of claim 12 , wherein the scheduling comprises refraining from reordering the first export action to occur ahead of the second export action.

16. The method of claim 12 , wherein the scheduling of the first export action is based on a predicted saturation state of a network connection resulting from the first export action.

17. The method of claim 12 , further comprising:

propagating version clock acknowledgements corresponding to both the first replication system and the remote replication system to corresponding acknowledgement version vectors stored in the first replication system, wherein each of the corresponding acknowledgement version vectors receive a respective version clock acknowledgement of the version clock acknowledgements.

18. The method of claim 12 , further comprising:

propagating version clock acknowledgements corresponding to both the first replication system and the remote replication system to corresponding acknowledgement version vectors stored in the first replication system, wherein each of the corresponding acknowledgement version vectors receive a respective version clock acknowledgement of the version clock acknowledgements.

19. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:

storing a first data object by a first replication system comprising a first set of immutable components, the first data object being associated with a corresponding second data object stored by a remote replication system;

determining a difference between the first set of immutable components of the first data object and a second set of immutable components of the corresponding second data object;

identifying a subset of immutable components from the first set of immutable components based on the difference; and

scheduling a first export action or a second export action to export the subset from the first replication system that provides, over a communication network, the subset to the remote replication system, the scheduling further comprising:

determining whether the first export action or the second export action is dependent on an import action to the first replication system; and

based on the determining of whether the first export action or the second export action is dependent on the import action, and in response to the first export action being scheduled behind the second export action, selectively rescheduling the first export action or the second export action to be concurrent with each other.

20. The non-transitory computer readable medium of claim 19 , wherein the scheduling further comprises scheduling import actions one at a time.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 2, 2022
From: FREIBERG, STEPHEN; LANDAU, ALEXANDER; GREENE, ANDREW; DORNE, BRIAN; OFFUTT, BRYAN; ZEIDMAN, ERNEST; NEPOMNYASCHCHIY, ILYA; GARROD, JOHN; BRAINARD, KATHERINE; PURCELL, KOLIN; LEVIN, MICHAEL; SWANSON, SIMON; STECKO, SPENCER
To: PALANTIR TECHNOLOGIES INC.
Reel/Frame 060701/0940 →
Continuity (4)
Continuation 16359398 · Mar 20, 2019
Continuation 15456256 · Mar 10, 2017
Provisional Application 62438190 · Dec 22, 2016
Related Publication 20220058209A1 · Feb 24, 2022