Distributed object replication architecture

1. A method of replicating data comprising:

providing, at a source site, a controller, a plurality of source worker nodes, and a message queue, coupled between the controller and the plurality of source worker nodes;

providing, at a destination site, a plurality of destination worker nodes;

defining a replication context specifying a replication frequency, and information identifying objects at the source site to replicate;

identifying, by the controller, differences between a first snapshot created at the source site at a first time and a second snapshot created at the source site at a second time, after the first time;

based on the differences, generating a set of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing, by the controller, the tasks onto the message queue;

retrieving, by a first source worker node, a first task from the message queue;

coordinating, by the first source worker node, with a first destination worker node to perform the first task;

retrieving, by a second source worker node, a second task from the message queue; and

coordinating, by the second source worker node, with a second destination worker node to perform the second task, wherein the first snapshot comprises a first listing of objects stored in an object storage at the source site as of the first time, the second snapshot comprises a second listing of objects stored in the source object storage as of the second time, and the method comprises:

receiving the first and second listings in an alphabetical sort;

comparing the alphabetically sorted first and second listings;

determining from the comparison that the first listing includes an object not included in the second listing; and

upon the determination, generating the first task, the first task comprising an instruction to delete the object from the destination site.

2. The method of claim 1 comprising:

storing a schedule indicating when replication from the source to destination site is to occur;

creating the first snapshot according to the schedule; and

creating the second snapshot according to the schedule.

3. The method of claim 1 wherein the first and second tasks are performed in parallel with each other.

4. A method of replicating data comprising:

providing, at a source site, a controller, a plurality of source worker nodes, and a message queue, coupled between the controller and the plurality of source worker nodes;

providing, at a destination site, a plurality of destination worker nodes;

defining a replication context specifying a replication frequency, and information identifying objects at the source site to replicate;

identifying, by the controller, differences between a first snapshot created at the source site at a first time and a second snapshot created at the source site at a second time, after the first time;

based on the differences, generating a set of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing, by the controller, the tasks onto the message queue;

retrieving, by a first source worker node, a first task from the message queue;

coordinating, by the first source worker node, with a first destination worker node to perform the first task;

retrieving, by a second source worker node, a second task from the message queue; and

coordinating, by the second source worker node, with a second destination worker node to perform the second task, wherein the first snapshot comprises a first listing of objects stored in an object storage at the source site as of the first time, the second snapshot comprises a second listing of objects stored in the source object storage as of the second time, and the method comprises:

receiving the first and second listings in an alphabetical sort;

comparing the alphabetically sorted first and second listings;

determining from the comparison that the second listing includes an object not included in the first listing; and

upon the determination, generating the first task, the first task comprising an instruction to copy the object to the destination site.

5. A method of replicating data comprising:

providing, at a source site, a controller, a plurality of source worker nodes, and a message queue, coupled between the controller and the plurality of source worker nodes;

providing, at a destination site, a plurality of destination worker nodes;

defining a replication context specifying a replication frequency, and information identifying objects at the source site to replicate;

identifying, by the controller, differences between a first snapshot created at the source site at a first time and a second snapshot created at the source site at a second time, after the first time;

based on the differences, generating a set of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing, by the controller, the tasks onto the message queue;

retrieving, by a first source worker node, a first task from the message queue;

coordinating, by the first source worker node, with a first destination worker node to perform the first task;

retrieving, by a second source worker node, a second task from the message queue; and

coordinating, by the second source worker node, with a second destination worker node to perform the second task, wherein the first snapshot comprises a first listing of objects stored in an object storage at the source site as of the first time, the second snapshot comprises a second listing of objects stored in the object storage as of the second time, and the method comprises:

receiving the first and second listings in an alphabetical sort;

comparing the alphabetically sorted first and second listings;

determining from the comparison that an object listed in both the first and second listings has a timestamp that changed between the first and second snapshots, the object listed in the first listing having an earlier timestamp, and the object listed in the second listing having a later timestamp; and

upon the determination, generating the first task and the second task, the first task comprising an instruction to copy the object with the later timestamp to the destination site, and the second task comprising an instruction to delete the object with the earlier timestamp from the destination site.

6. A system for replicating data, the system comprising: a processor, and memory storing one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:

providing, at a source site, a controller, a plurality of source worker nodes, and a message queue, coupled between the controller and the plurality of source worker nodes;

providing, at a destination site, a plurality of destination worker nodes;

defining a replication context specifying a replication frequency, and information identifying objects at the source site to replicate;

identifying, by the controller, differences between a first snapshot created at the source site at a first time and a second snapshot created at the source site at a second time, after the first time;

based on the differences, generating a set of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing, by the controller, the tasks onto the message queue;

retrieving, by a first source worker node, a first task from the message queue;

coordinating, by the first source worker node, with a first destination worker node to perform the first task;

retrieving, by a second source worker node, a second task from the message queue; and

coordinating, by the second source worker node, with a second destination worker node to perform the second task, wherein the first snapshot comprises a first listing of objects stored in an object storage at the source site as of the first time, the second snapshot comprises a second listing of objects stored in the source object storage as of the second time, and the processor further carries out the steps of:

receiving the first and second listings in an alphabetical sort;

comparing the alphabetically sorted first and second listings;

determining from the comparison that the second listing includes an object not included in the first listing; and

upon the determination, generating the first task, the first task comprising an instruction to copy the object to the destination site.

7. The system of claim 6 wherein the processor further carries out the steps of:

storing a schedule indicating when replication from the source to destination site is to occur;

creating the first snapshot according to the schedule; and

creating the second snapshot according to the schedule.

8. The system of claim 6 wherein the first and second tasks are performed in parallel with each other.

9. A system for replicating data, the system comprising: a processor, and memory storing one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:

providing, at a source site, a controller, a plurality of source worker nodes, and a message queue, coupled between the controller and the plurality of source worker nodes;

providing, at a destination site, a plurality of destination worker nodes;

defining a replication context specifying a replication frequency, and information identifying objects at the source site to replicate;

identifying, by the controller, differences between a first snapshot created at the source site at a first time and a second snapshot created at the source site at a second time, after the first time;

based on the differences, generating a set of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing, by the controller, the tasks onto the message queue;

retrieving, by a first source worker node, a first task from the message queue;

coordinating, by the first source worker node, with a first destination worker node to perform the first task;

retrieving, by a second source worker node, a second task from the message queue; and

coordinating, by the second source worker node, with a second destination worker node to perform the second task, wherein the first snapshot comprises a first listing of objects stored in an object storage at the source site as of the first time, the second snapshot comprises a second listing of objects stored in the source object storage as of the second time, and the processor further carries out the steps of:

receiving the first and second listings in an alphabetical sort;

comparing the alphabetically sorted first and second listings;

determining from the comparison that the first listing includes an object not included in the second listing; and

upon the determination, generating the first task, the first task comprising an instruction to delete the object from the destination site.

10. A system for replicating data, the system comprising: a processor, and memory storing one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:

providing, at a source site, a controller, a plurality of source worker nodes, and a message queue, coupled between the controller and the plurality of source worker nodes;

providing, at a destination site, a plurality of destination worker nodes;

defining a replication context specifying a replication frequency, and information identifying objects at the source site to replicate;

identifying, by the controller, differences between a first snapshot created at the source site at a first time and a second snapshot created at the source site at a second time, after the first time;

based on the differences, generating a set of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing, by the controller, the tasks onto the message queue;

retrieving, by a first source worker node, a first task from the message queue;

coordinating, by the first source worker node, with a first destination worker node to perform the first task;

retrieving, by a second source worker node, a second task from the message queue; and

coordinating, by the second source worker node, with a second destination worker node to perform the second task, wherein the first snapshot comprises a first listing of objects stored in an object storage at the source site as of the first time, the second snapshot comprises a second listing of objects stored in the object storage as of the second time, and the processor further carries out the steps of:

receiving the first and second listings in an alphabetical sort;

comparing the alphabetically sorted first and second listings;

determining from the comparison that an object listed in both the first and second listings has a timestamp that changed between the first and second snapshots, the object listed in the first listing having an earlier timestamp, and the object listed in the second listing having a later timestamp; and

upon the determination, generating the first task and the second task, the first task comprising an instruction to copy the object with the later timestamp to the destination site, and the second task comprising an instruction to delete the object with the earlier timestamp from the destination site.

11. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:

providing, at the source site, a controller, a plurality of source worker nodes, and a message queue, coupled between the controller and the plurality of source worker nodes;

providing, at the destination site, a plurality of destination worker nodes;

defining a replication context specifying a replication frequency, and information identifying objects at the source site to replicate;

identifying, by the controller, differences between a first snapshot created at the source site at a first time and a second snapshot created at the source site at a second time, after the first time;

based on the differences, generating a set of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing, by the controller, the tasks onto the message queue;

retrieving, by a first source worker node, a first task from the message queue;

coordinating, by the first source worker node, with a first destination worker node to perform the first task;

retrieving, by a second source worker node, a second task from the message queue; and

coordinating, by the second source worker node, with a second destination worker node to perform the second task, wherein the first snapshot comprises a first listing of objects stored in an object storage at the source site as of the first time, the second snapshot comprises a second listing of objects stored in the object storage as of the second time, and the method comprises:

receiving the first and second listings in an alphabetical sort;

comparing the alphabetically sorted first and second listings;

determining from the comparison that an object listed in both the first and second listings has a timestamp that changed between the first and second snapshots, the object listed in the first listing having an earlier timestamp, and the object listed in the second listing having a later timestamp; and

upon the determination, generating the first task and the second task, the first task comprising an instruction to copy the object with the later timestamp to the destination site, and the second task comprising an instruction to delete the object with the earlier timestamp from the destination site.

12. The computer program product of claim 11 wherein the method comprises:

storing a schedule indicating when replication from the source to destination site is to occur;

creating the first snapshot according to the schedule; and

creating the second snapshot according to the schedule.

13. The computer program product of claim 11 wherein the first and second tasks are performed in parallel with each other.

14. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:

providing, at the source site, a controller, a plurality of source worker nodes, and a message queue, coupled between the controller and the plurality of source worker nodes;

providing, at the destination site, a plurality of destination worker nodes;

defining a replication context specifying a replication frequency, and information identifying objects at the source site to replicate;

identifying, by the controller, differences between a first snapshot created at the source site at a first time and a second snapshot created at the source site at a second time, after the first time;

based on the differences, generating a set of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing, by the controller, the tasks onto the message queue;

retrieving, by a first source worker node, a first task from the message queue;

coordinating, by the first source worker node, with a first destination worker node to perform the first task;

retrieving, by a second source worker node, a second task from the message queue; and

coordinating, by the second source worker node, with a second destination worker node to perform the second task, wherein the first snapshot comprises a first listing of objects stored in an object storage at the source site as of the first time, the second snapshot comprises a second listing of objects stored in the source object storage as of the second time, and the method comprises:

receiving the first and second listings in an alphabetical sort;

comparing the alphabetically sorted first and second listings;

determining from the comparison that the first listing includes an object not included in the second listing; and

upon the determination, generating the first task, the first task comprising an instruction to delete the object from the destination site.

15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:

providing, at the source site, a controller, a plurality of source worker nodes, and a message queue, coupled between the controller and the plurality of source worker nodes;

providing, at the destination site, a plurality of destination worker nodes;

defining a replication context specifying a replication frequency, and information identifying objects at the source site to replicate;

identifying, by the controller, differences between a first snapshot created at the source site at a first time and a second snapshot created at the source site at a second time, after the first time;

based on the differences, generating a set of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing, by the controller, the tasks onto the message queue;

retrieving, by a first source worker node, a first task from the message queue;

coordinating, by the first source worker node, with a first destination worker node to perform the first task;

retrieving, by a second source worker node, a second task from the message queue; and

coordinating, by the second source worker node, with a second destination worker node to perform the second task, wherein the first snapshot comprises a first listing of objects stored in an object storage at the source site as of the first time, the second snapshot comprises a second listing of objects stored in the source object storage as of the second time, and the method comprises:

receiving the first and second listings in an alphabetical sort;

comparing the alphabetically sorted first and second listings;

determining from the comparison that the second listing includes an object not included in the first listing; and

upon the determination, generating the first task, the first task comprising an instruction to copy the object to the destination site.