Detached global scheduler
Systems, methods, and machine-readable media are disclosed for a global scheduler that manages jobs across nodes of one or more clusters. The global scheduler receives a request to replicate a volume from a first storage node to a second storage node, the global scheduler handling job scheduling for one or more storage operating systems across one or more clusters. The global scheduler stores a new job in a job queue based on the request to replicate the volume. The global scheduler determines that the new job has a higher priority than other jobs in the job queue. The global scheduler allocates one or more resources of the first storage node to the new job in response to determining that the new job has a higher priority than other jobs in the job queue. The global scheduler sends an instruction to run the new job.
1 . A method comprising:
periodically receiving, by a global scheduler, information about one or more resources of a first storage node and one or more resources of a second storage node,
wherein the global scheduler is external to, and handles job scheduling for, multiple storage operating systems running on multiple storage nodes across one or more clusters based on a view across at least one of clusters or regions including the periodically receiving the information about the one or more resources available on the first storage node and on the second storage node;
receiving, by the global scheduler, a request to replicate a volume from the first storage node to the second storage node;
storing, by the global scheduler, a new job in a global job queue external to the multiple storage operating systems running on the multiple storage nodes and based on the request to replicate the volume, the global job queue including at least one other job;
allocating, by the global scheduler, at least a first subset of resources of the first storage node from the one or more resources of the first storage node, and at least a second subset of resources of the second storage node from the one or more resources of the second storage node, to the new job based on the received information about the one or more resources of the first node and the second node, and in response to determining that the new job has a higher priority than the at least one other job in the global job queue; and
sending, by the global scheduler, an instruction to the first storage node and the second storage node for local schedulers that are part of storage operating systems at the first and second nodes, from among the multiple storage operating systems, to run the new job.
2 . The method of claim 1 , wherein the first storage node belongs to a first cluster of storage nodes and the second storage node belongs to a second cluster of storage nodes, wherein the new job and the at least one other job are configured to run in sessions hosted across different storage nodes.
3 . The method of claim 1 , further comprising:
identifying, by the global scheduler, available resources of the first storage node that are able to perform the new job, the available resources including one of volume replication sessions and bandwidth.
4 . The method of claim 1 , wherein:
the receiving the request is performed by a first service of the global scheduler,
the determining that the new job has the higher priority is performed by a second service of the global scheduler that is different than the first service, and
the periodically receiving the information is performed by a third service of the global scheduler that is different than the first and second services.
5 . The method of claim 1 , further comprising:
preempting, by the global scheduler, a second job being run based on a determination that the new job has a higher priority than the second job.
6 . The method of claim 1 , wherein:
the one or more resources of the first storage node include a first plurality of sessions, each session configured to run the new job to replicate the volume,
the one or more resources of the second storage node include a second plurality of sessions, and
the allocating further comprises decreasing a session availability count at both the first storage node and the second storage node.
7 . The method of claim 1 , further comprising:
determining, by the global scheduler, that a second job in the global job queue has not been run for a period of time; and
sending, by the global scheduler, a second instruction to run the second job before the new job in response to the determination that the second job has not run for a threshold period of time.
8 . A computing device comprising:
a memory containing machine readable medium comprising machine executable code having stored thereon instructions for performing a method of scheduling jobs across a plurality of storage nodes, the instructions being external to multiple storage operating systems running on the plurality of storage nodes; and
a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to:
periodically send a request for information about one or more resources of a first storage node of the plurality of storage nodes and one or more resources of a second storage node of the plurality of storage nodes;
periodically receive, in response to the request, the information about the one or more resources of the first storage node and the one or more resources of the second storage node;
receive a request to perform a task according to a schedule;
store a first job in a global job queue external to the multiple storage operating systems running on the plurality of storage nodes, the first job being based on the request to perform the task, and the global job queue including at least one other job;
pause a second job in response to a determination that there are insufficient resources to run the first job and in response to the received information about the one or more resources of the first storage node and the second storage node, the second job having a lower priority than the first job;
allocate a first subset of the one or more resources of the first storage node, and a second subset of the one or more resources of the second storage node, being used by the second job to the first job in response to the pausing the second job; and
send messages to each of the first storage node and the second storage node to initiate the first job using the one or more resources of the first and second storage nodes through local schedulers that are part of storage operating systems at the first and second storage nodes, from among the multiple storage operating systems, allocated to the first job.
9 . The computing device of claim 8 , wherein the first storage node belongs to a first storage cluster, the second storage node belongs to a second storage cluster, and the first job includes replicating a volume from the first storage node to the second storage node.
10 . The computing device of claim 8 , wherein the processor is further configured to:
send a resume message to each of the first storage node and the second storage node to resume the second job in response to the first job completing.
11 . The computing device of claim 8 , wherein the processor is further configured to:
identify, based on the periodically received information, available resources on the first and second storage nodes suitable to run the first job, the available resources including one of volume replication sessions and bandwidth.
12 . The computing device of claim 8 , wherein the processor is further configured to:
identify an available session on the first and second storage nodes of the plurality of storage nodes; and
allocate the available session to the first and second storage nodes to run the first job.
13 . The computing device of claim 8 , wherein the processor is further configured to:
identify a third job from the at least one other job in the global job queue that has exceeded a threshold for time passed since successfully completed according to a schedule of the third job; and
initiate the third job on a storage node of the plurality of storage nodes.
14 . The computing device of claim 8 , wherein the processor is further configured to:
adjust bandwidth allocated to a third job running on the plurality of storage nodes to enable the third job to complete sooner and free one or more resources for the first job.
15 . A non-transitory machine-readable medium having stored thereon instructions for performing a method of scheduling jobs across a plurality of storage nodes, the instructions for performing the method being external to multiple storage operating systems running on the plurality of storage nodes, which when executed by at least one machine, causes the at least one machine to:
periodically receive information about one or more resources of a first storage node and one or more resources of a second storage node from the plurality of storage nodes;
receive a request to run a first job according to a provided schedule;
publish the first job to a global job queue external to the multiple storage operating systems running on the plurality of storage nodes, the global job queue being one of a plurality of global job queues, the global job queue including a plurality of jobs of a similar type to the first job;
allocate, in response to determining the first job has a higher priority than a second job of the plurality of jobs and in response to the received information about the one or more resources of the first and second storage nodes, an additional amount of bandwidth to the first job of the plurality of jobs to complete the first job, the second job requiring a session on the first and second storage nodes of the plurality of storage nodes, the session comprising at least a part of the one or more resources of the first and second storage nodes; and
send an instruction to the first and second storage nodes for local schedulers of the first and second storage nodes of the plurality of storage nodes to run the second job after the first job has completed.
16 . The non-transitory machine-readable medium of claim 15 , further comprising machine executable code that causes the machine to:
determine, before allocating the additional amount of bandwidth to the first job, that a session is not available to run the first job; and
allocate the session run by the second job to the first job.
17 . The non-transitory machine-readable medium of claim 15 , further comprising machine executable code that causes the machine to:
identify available resources, based on the information about the one or more resources of the first storage node, to run the first job, the first job including replicating a volume from the first storage node to the second storage node;
identify, based on the information about the one or more resources of the second storage node, available resources on the second storage node to run the first job;
allocate resources on the first storage node to run the first job; and
allocate resources on the second storage node to run the first job.
18 . The non-transitory machine-readable medium of claim 15 , wherein the first job includes replicating a volume from the first storage node to the second storage node, the second storage node being in a different storage cluster than the first storage node.
19 . The non-transitory machine-readable medium of claim 15 , further comprising machine executable code that causes the machine to:
update a status of the first job in a first database;
poll the first database for a status update; and
update the status of the first job in a second database in response to receiving the status update from the first database.
20 . The non-transitory machine-readable medium of claim 15 , further comprising machine executable code that causes the machine to:
query, before publishing the first job to the global job queue, a database including the plurality of jobs;
determine a subset of the plurality of jobs that are eligible to be scheduled; and
publish the subset of the plurality of jobs to the global job queue in priority order, the subset including the first job.