Dynamic allocation of worker nodes for distributed replication
A set of tasks, associated with a replication job, are generated for replicating from a source to destination site. An initial number of worker nodes are allocated to process the tasks. Each task involves a first type of worker node at the source site, a second type of worker node at the destination site, and includes one or more of copying an object from the source site to the destination site, or deleting an object from the destination site. The replication job is monitored. Based on the monitoring, a number of worker nodes is adjusted from the initial number to a new number, different from the initial number.
1. A method comprising:
generating a set of tasks associated with a replication job, the replication job comprising replicating from a source site to a destination site;
allocating an initial number of worker nodes to process the set of tasks, each task involving a first type of worker node at the source site, a second type of worker node at the destination site, and comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;
monitoring the replication job;
based on the monitoring, adjusting at least one of a number of worker nodes of the first type or a number of worker nodes of the second type to process the set of tasks from the initial number to a new number, different from the initial number;
placing the set of tasks onto a message queue for worker nodes of the first type to retrieve and process in conjunction with worker nodes of the second type;
tracking a number of tasks remaining on the message queue; and
when the number of tasks remaining on the message queue exceeds a threshold, allocating at least one of additional worker nodes of the first type or additional worker nodes of the second type.
2. The method of claim 1 wherein the adjusting at least one of a number of worker nodes of the first type or a number of worker nodes of the second type further comprises at least one of allocating additional worker nodes of the first type or allocating additional worker nodes of the second type, the new number of worker nodes thereby being greater than the initial number of worker nodes.
3. The method of claim 1 wherein the adjusting at least one of a number of worker nodes of the first type or a number of worker nodes of the second type further comprises at least one of deallocating worker nodes of the first type or deallocating worker nodes of the second type, the new number of worker nodes thereby being less than the initial number of worker nodes.
4. A method comprising:
generating a set of tasks associated with a replication job, the replication job comprising replicating from a source site to a destination site;
allocating an initial number of worker nodes to process the set of tasks, each task involving a first type of worker node at the source site, a second type of worker node at the destination site, and comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;
monitoring the replication job;
based on the monitoring, adjusting at least one of a number of worker nodes of the first type or a number of worker nodes of the second type to process the set of tasks from the initial number to a new number, different from the initial number;
receiving a constraint indicating a duration of time permitted to complete the replication job;
calculating a number and a size of objects at the source site that are to be replicated to the destination site; and
based on the number and size of objects that are to be replicated, allocating a sufficient number of worker nodes of the first type at the source site and a sufficient number of worker nodes of the second type at the destination site to complete the replication job within the permitted duration of time.
5. A method comprising:
generating a set of tasks associated with a replication job, the replication job comprising replicating from a source site to a destination site;
allocating an initial number of worker nodes to process the set of tasks, each task involving a first type of worker node at the source site, a second type of worker node at the destination site, and comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;
monitoring the replication job;
based on the monitoring, adjusting at least one of a number of worker nodes of the first type or a number of worker nodes of the second type to process the set of tasks from the initial number to a new number, different from the initial number;
receiving a constraint indicating a maximum cost for the replication job; and
based on the maximum cost, allocating the initial number of worker nodes such that a total cost for the worker nodes does not exceed the maximum cost.
6. A method comprising:
generating a set of tasks associated with a replication job, the replication job comprising replicating from a source site to a destination site;
allocating an initial number of worker nodes to process the set of tasks, each task involving a first type of worker node at the source site, a second type of worker node at the destination site, and comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;
monitoring the replication job;
based on the monitoring, adjusting at least one of a number of worker nodes of the first type or a number of worker nodes of the second type to process the set of tasks from the initial number to a new number, different from the initial number;
assessing a priority level of a data set to be replicated;
if the priority level of the data set is of a low priority, allocating a first initial number of worker nodes; and
if the priority level of the data set is of a high priority, allocating a second initial number of worker nodes, greater than the first initial number of worker nodes.
7. The method of claim 1 further comprising:
obtaining a metric indicating CPU utilization associated with the worker nodes; and
scaling at least one of the number of worker nodes of the first type or the number of worker nodes of the second type based on the CPU utilization being above or below a threshold.
8. A system comprising:
a processor-based system executed on a computer system and configured to:
generate a set of tasks associated with a replication job, the replication job comprising replicating from a source site to a destination site;
allocate an initial number of worker nodes to process the set of tasks, each task involving a first type of worker node at the source site, a second type of worker node at the destination site, and comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;
monitor the replication job;
based on the monitoring, adjust at least one of a number of worker nodes of the first type or a number of worker nodes of the second type to process the set of tasks from the initial number to a new number, different from the initial number;
place the set of tasks onto a message queue for worker nodes of the first type to retrieve and process in conjunction with worker nodes of the second type;
track a number of tasks remaining on the message queue; and
when the number of tasks remaining on the message queue exceeds a threshold, allocate at least one of additional worker nodes of the first type or additional worker nodes of the second type.
9. A system comprising:
a processor-based system executed on a computer system and configured to:
generate a set of tasks associated with a replication job, the replication job comprising replicating from a source site to a destination site;
allocate an initial number of worker nodes to process the set of tasks, each task involving a first type of worker node at the source site, a second type of worker node at the destination site, and comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;
monitor the replication job;
based on the monitoring, adjust at least one of a number of worker nodes of the first type or a number of worker nodes of the second type to process the set of tasks from the initial number to a new number, different from the initial number;
receive a constraint indicating a duration of time permitted to complete the replication job;
calculate a number and a size of objects at the source site that are to be replicated to the destination site; and
based on the number and size of objects that are to be replicated, allocate a sufficient number of worker nodes of the first type at the source site and a sufficient number of worker nodes of the second type at the destination site to complete the replication job within the permitted duration of time.
10. A system comprising:
a processor-based system executed on a computer system and configured to:
generate a set of tasks associated with a replication job, the replication job comprising replicating from a source site to a destination site;
allocate an initial number of worker nodes to process the set of tasks, each task involving a first type of worker node at the source site, a second type of worker node at the destination site, and comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;
monitor the replication job;
based on the monitoring, adjust at least one of a number of worker nodes of the first type or a number of worker nodes of the second type to process the set of tasks from the initial number to a new number, different from the initial number;
receive a constraint indicating a maximum cost for the replication job; and
based on the maximum cost, allocate the initial number of worker nodes such that a total cost for the worker nodes does not exceed the maximum cost.
11. A system comprising:
a processor-based system executed on a computer system and configured to:
generate a set of tasks associated with a replication job, the replication job comprising replicating from a source site to a destination site;
allocate an initial number of worker nodes to process the set of tasks, each task involving a first type of worker node at the source site, a second type of worker node at the destination site, and comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;
monitor the replication job;
based on the monitoring, adjust at least one of a number of worker nodes of the first type or a number of worker nodes of the second type to process the set of tasks from the initial number to a new number, different from the initial number;
assess a priority level of a data set to be replicated;
if the priority level of the data set is of a low priority, allocate a first initial number of worker nodes; and
if the priority level of the data set is of a high priority, allocate a second initial number of worker nodes, greater than the first initial number of worker nodes.
12. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:
generating a set of tasks associated with a replication job, the replication job comprising replicating from a source site to a destination site;
allocating an initial number of worker nodes to process the set of tasks, each task involving a first type of worker node at the source site, a second type of worker node at the destination site, and comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;
monitoring the replication job;
based on the monitoring, adjusting at least one of a number of worker nodes of the first type or a number of worker nodes of the second type to process the set of tasks from the initial number to a new number, different from the initial number;
placing the set of tasks onto a message queue for worker nodes of the first type to retrieve and process in conjunction with worker nodes of the second type;
tracking a number of tasks remaining on the message queue; and
when the number of tasks remaining on the message queue exceeds a threshold, allocating at least one of additional worker nodes of the first type or additional worker nodes of the second type.
13. The computer program product of claim 12 wherein the adjusting at least one of a number of worker nodes of the first type or a number of worker nodes of the second type further comprises at least one of allocating additional worker nodes of the first type or allocating additional worker nodes of the second type, the new number of worker nodes thereby being greater than the initial number of worker nodes.
14. The computer program product of claim 12 wherein the adjusting at least one of a number of worker nodes of the first type or a number of worker nodes of the second type further comprises at least one of deallocating worker nodes of the first type or deallocating worker nodes of the second type, the new number of worker nodes thereby being less than the initial number of worker nodes.
15. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:
generating a set of tasks associated with a replication job, the replication job comprising replicating from a source site to a destination site;
allocating an initial number of worker nodes to process the set of tasks, each task involving a first type of worker node at the source site, a second type of worker node at the destination site, and comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;
monitoring the replication job;
based on the monitoring, adjusting at least one of a number of worker nodes of the first type or a number of worker nodes of the second type to process the set of tasks from the initial number to a new number, different from the initial number;
receiving a constraint indicating a duration of time permitted to complete the replication job;
calculating a number and a size of objects at the source site that are to be replicated to the destination site; and
based on the number and size of objects that are to be replicated, allocating a sufficient number of worker nodes of the first type at the source site and a sufficient number of worker nodes of the second type at the destination site to complete the replication job within the permitted duration of time.
16. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:
generating a set of tasks associated with a replication job, the replication job comprising replicating from a source site to a destination site;
allocating an initial number of worker nodes to process the set of tasks, each task involving a first type of worker node at the source site, a second type of worker node at the destination site, and comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;
monitoring the replication job;
based on the monitoring, adjusting at least one of a number of worker nodes of the first type or a number of worker nodes of the second type to process the set of tasks from the initial number to a new number, different from the initial number;
receiving a constraint indicating a maximum cost for the replication job; and
based on the maximum cost, allocating the initial number of worker nodes such that a total cost for the worker nodes does not exceed the maximum cost.
17. A computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement a method comprising:
generating a set of tasks associated with a replication job, the replication job comprising replicating from a source site to a destination site;
allocating an initial number of worker nodes to process the set of tasks, each task involving a first type of worker node at the source site, a second type of worker node at the destination site, and comprising one or more of copying an object from the source site to the destination site, or deleting an object from the destination site;
monitoring the replication job;
based on the monitoring, adjusting at least one of a number of worker nodes of the first type or a number of worker nodes of the second type to process the set of tasks from the initial number to a new number, different from the initial number;
assessing a priority level of a data set to be replicated;
if the priority level of the data set is of a low priority, allocating a first initial number of worker nodes; and
if the priority level of the data set is of a high priority, allocating a second initial number of worker nodes, greater than the first initial number of worker nodes.