Tracking status and restarting distributed replication

1. A method comprising:

generating, by a controller at a source site, a plurality of tasks associated with a replication job between the source site and a destination site, the plurality of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing the plurality of tasks onto a message queue at the source site;

allowing source worker nodes at the source site to retrieve the tasks from the source site message queue for processing in conjunction with destination worker nodes at the destination site,

wherein a destination worker node, upon receiving a task from a source worker node, places the task onto a message queue at the destination site for retrieval by a backend worker node that handles writing to an object store at the destination site;

monitoring the replication job;

based on the monitoring, determining that the replication job has failed;

upon the failure, determining that first portions of the object have been replicated from the source site to the destination site while second portions of the object have not yet been replicated from the source site to the destination site; and

generating new tasks to replicate only the second portions of the object that have not yet been replicated.

2. The method of claim 1 further comprising:

based on the monitoring, adjusting at least one of a number of the source worker nodes at the source site, or a number of the destination worker nodes at the destination site.

3. The method of claim 1 wherein a first pair of source and destination worker nodes processes a first task of the plurality of tasks, and a second pair of source and destination worker nodes, different from the first pair, processes a second task of the plurality of tasks while the first task is being processed.

4. A method comprising:

generating, by a controller at a source site, a plurality of tasks associated with a replication job between the source site and a destination site, the plurality of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing the plurality of tasks onto a message queue at the source site;

allowing source worker nodes at the source site to retrieve the tasks from the source site message queue for processing in conjunction with destination worker nodes at the destination site,

wherein a destination worker node, upon receiving a task from a source worker node, places the task onto a message queue at the destination site for retrieval by a backend worker node that handles writing to an object store at the destination site;

logging status update messages about the replication job into a persistent key-value store;

upon a failure to complete the replication job, accessing the key-value store to identify tasks of the plurality of tasks that were and were not completed before the failure; and

resending the tasks that were not completed to the source worker nodes and not resending the tasks that were completed.

5. A system comprising:

a processor, and memory storing one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:

generating, by a controller at a source site, a plurality of tasks associated with a replication job between the source site and a destination site, the plurality of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing the plurality of tasks onto a message queue at the source site;

allowing source worker nodes at the source site to retrieve the tasks from the source site message queue for processing in conjunction with destination worker nodes at the destination site,

wherein a destination worker node, upon receiving a task from a source worker node, places the task onto a message queue at the destination site for retrieval by a backend worker node that handles writing to an object store at the destination site;

monitoring the replication job;

based on the monitoring, determining that the replication job has failed;

upon the failure, determining that first portions of the object have been replicated from the source site to the destination site while second portions of the object have not yet been replicated from the source site to the destination site; and

generating new tasks to replicate only the second portions of the object that have not yet been replicated.

6. The system of claim 5 wherein the processor further carries out the steps of:

based on the monitoring, adjusting at least one of a number of the source worker nodes at the source site, or a number of the destination worker nodes at the destination site.

7. The system of claim 5 wherein a first pair of source and destination worker nodes processes a first task of the plurality of tasks, and a second pair of source and destination worker nodes, different from the first pair, processes a second task of the plurality of tasks while the first task is being processed.

8. A system comprising:

a processor, and memory storing one or more sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of:

generating, by a controller at a source site, a plurality of tasks associated with a replication job between the source site and a destination site, the plurality of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing the plurality of tasks onto a message queue at the source site;

allowing source worker nodes at the source site to retrieve the tasks from the source site message queue for processing in conjunction with destination worker nodes at the destination site,

wherein a destination worker node, upon receiving a task from a source worker node, places the task onto a message queue at the destination site for retrieval by a backend worker node that handles writing to an object store at the destination site;

logging status update messages about the replication job into a persistent key-value store;

upon a failure to complete the replication job, accessing the key-value store to identify tasks of the plurality of tasks that were and were not completed before the failure; and

resending the tasks that were not completed to the source worker nodes and not resending the tasks that were completed.

9. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:

generating, by a controller at a source site, a plurality of tasks associated with a replication job between the source site and a destination site, the plurality of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing the plurality of tasks onto a message queue at the source site;

allowing source worker nodes at the source site to retrieve the tasks from the source site message queue for processing in conjunction with destination worker nodes at the destination site,

wherein a destination worker node, upon receiving a task from a source worker node, places the task onto a message queue at the destination site for retrieval by a backend worker node that handles writing to an object store at the destination site;

monitoring the replication job;

based on the monitoring, determining that the replication job has failed;

upon the failure, determining that first portions of the object have been replicated from the source site to the destination site while second portions of the object have not yet been replicated from the source site to the destination site; and

generating new tasks to replicate only the second portions of the object that have not yet been replicated.

10. The computer program product of claim 9 wherein the method further comprises:

based on the monitoring, adjusting at least one of a number of the source worker nodes at the source site, or a number of the destination worker nodes at the destination site.

11. The computer program product of claim 9 wherein a first pair of source and destination worker nodes processes a first task of the plurality of tasks, and a second pair of source and destination worker nodes, different from the first pair, processes a second task of the plurality of tasks while the first task is being processed.

12. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:

generating, by a controller at a source site, a plurality of tasks associated with a replication job between the source site and a destination site, the plurality of tasks comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;

placing the plurality of tasks onto a message queue at the source site;

allowing source worker nodes at the source site to retrieve the tasks from the source site message queue for processing in conjunction with destination worker nodes at the destination site,

wherein a destination worker node, upon receiving a task from a source worker node, places the task onto a message queue at the destination site for retrieval by a backend worker node that handles writing to an object store at the destination site;

logging status update messages about the replication job into a persistent key-value store;

upon a failure to complete the replication job, accessing the key-value store to identify tasks of the plurality of tasks that were and were not completed before the failure; and

resending the tasks that were not completed to the source worker nodes and not resending the tasks that were completed.