Transient sharing of available SAN compute capability
Embodiments are described for a executing a processing job using one or more nodes of a storage area network using computing resources on the SAN that are predicted to be idle. A predictive model is generated by monitoring idle states of resources of nodes of the SAN and using machine learning to build the predictive model. A scheduler executes jobs on one or more nodes of the SAN with sufficient predicted idle resources to process the job, in accordance with resource requirements and job attributes in a manifest of the job. If a job cannot be completed during a window of time that the necessary resources are predicted to be idle, or if one or more resources become unavailable, the job can be paused and resumed, migrated to another node, or restarted at a later time when the required resources to complete the job are predicted to be idle.
1. A computer-implemented comprising:
receiving, by a scheduler, a processing job and a manifest for the processing job, the manifest including an estimate of one or more resources required to perform the processing job;
determining, by a scheduler on a first node in a multi-node network, at least one second node in the multi-node network, on which to execute the processing job using a plurality of computing resources predicted to be idle to execute the processing job on the at least one second node, wherein the determining is based at least in part upon whether a predictive model of idle times of computing resources in the multi-node network predicts that there are idle computing resources on the at least one second node having a magnitude that meets or exceeds an estimated requirements to perform the processing job; and
in response to determining that an actual state of the computing resources predicted to be idle on the at least one second node meets or exceeds a predicted state of the computing resources on the at least one second node, executing the processing job on the at least one second node.
2. The method of claim 1 , wherein the multi-node network comprises a storage area network (SAN) and nodes of the multi-node network comprise a plurality of host computers and at least one storage appliance, and the predictive model includes predictions for resources of the at least one storage appliance.
3. The method of claim 1 , further comprising updating the predictive model of idle times of computing resources on a plurality of nodes of the multi-node network by updating an estimate of one or more resources required to perform the processing job with actual execution resources used to perform the processing job.
4. The method of claim 1 , further comprising:
in response to determining that the actual state of computing resources predicted to be idle on the at least one second node does not meet the computing resources predicted to be idle for executing the processing job on the at least one second node:
determining at least one third node on which to execute the processing job;
executing the processing job on the at the least one third node.
5. The method of claim 1 , further comprising:
in response to determining, during the execution of the processing job on the at least one second node, that at least some of the computing resources predicted to be idle on the at least one second node are no longer idle or are no longer available to the processing job:
determining that the processing job is pausible;
pausing the processing job until the computing resources required to complete the processing job on the at least one second node are predicted to be idle for a predicted remaining execution time of the processing job;
resuming execution of the processing job on the at least one second node.
6. The method of claim 1 , further comprising:
in response to determining, during the execution of the processing job on the at least one second node, that at least some of the computing resources predicted to be idle on the at least one second node are no longer idle or are no longer available to the processing job:
determining that the processing job is restartable;
determining a second predicted time at which the resources for executing the processing job are predicted to be idle on the at least one second node;
restarting execution of the processing job on the at least one second node.
7. The method of claim 1 , further comprising:
in response to determining that at least some of the computing resources predicted to be idle on the at least one second node are not idle or are no longer available to the processing job:
migrating the processing job to a fourth node in the multi-node network that has the plurality of resources for executing the processing job predicted to be idle;
executing the processing job on the at least one fourth node.
8. A non-transitory computer-readable medium programmed with executable instructions that, when executed by a processing system having at least one hardware processor, perform operations comprising:
receiving a processing job and a manifest for the processing job, the manifest including an estimate of one or more resources required to perform the processing job;
determining, by a scheduler on a first node in a multi-node network, at least one second node the a multi-node network, on which to execute the processing job using a plurality of computing resources predicted to be idle to execute the processing job on the at least one second node, wherein the determining is based at least in part upon whether a predictive model of idle times of computing resources in the multi-node network predicts that there are idle computing resources on the at least one second node having a magnitude that meets or exceeds estimated requirements to perform the processing job; and
in response to determining that an actual state of the computing resources predicted to be idle on the at least one second node meets or exceeds a predicted state of the computing resources on the at least one second node, executing the processing job on the at least one second node.
9. The medium of claim 8 , wherein the multi-node network comprises a storage area network (SAN) and nodes of the multi-node network comprise a plurality of host computers and at least one storage appliance, and the predictive model includes predictions for resources of the at least one storage appliance.
10. The medium of claim 8 , wherein the operations further comprise updating the predictive model of idle times of computing resources on a plurality of nodes of the multi-node network by updating an estimate of one or more resources required to perform the processing job with actual execution resources used to perform the processing job.
11. The medium of claim 8 , wherein operations further comprise:
in response to determining that an actual state of computing resources predicted to be idle on the at least one second node does not meet the computing resources predicted to be idle for executing the processing job on the at least one second node:
determining at least one third node on which to execute the processing job;
executing the processing job on the at the least one third node.
12. The medium of claim 8 , the operations further comprising:
in response to determining, during the execution of the processing job on the at least one second node, that at least some of the computing resources predicted to be idle on the at least one second node are no longer idle or are no longer available to the processing job:
determining that the processing job is pausible;
pausing the processing job until the computing resources required to complete the processing job on the at least one second node are predicted to be idle for a predicted remaining execution time of the processing job;
resuming execution of the processing job on the at least one second node.
13. The medium of claim 8 , the operations further comprising:
in response to determining, during the execution of the processing job on the at least one second node, that at least some of the computing resources predicted to be idle on the at least one second node are no longer idle or are no longer available to the processing job:
determining that the processing job is restartable;
determining a second predicted time at which the resources for executing the processing job are predicted to be idle on the at least one second node;
restarting execution of the processing job on the at least one second node.
14. The medium of claim 8 , the operations further comprising:
in response to determining that at least some of the computing resources predicted to be idle on the at least one second node are not idle or are no longer available to the processing job:
migrating the processing job to a fourth node in the multi-node network that has the plurality of resources for executing the processing job predicted to be idle;
executing the processing job on the at least one fourth node.
15. A system comprising:
a processing system having at least one hardware processor, the processing system coupled to a memory programmed with executable instructions that, when executed by the processing system, perform operations comprising:
receiving a processing job and a manifest for the processing job, the manifest including an estimate of one or more resources required to perform the processing job;
determining, by a scheduler on a first node in a multi-node network, at least one second node in the multi-node network, on which to execute the processing job using a plurality of computing resources predicted to be idle to execute the processing job on the at least one second node, wherein the determining is based at least in part upon whether a predictive model of idle times of computing resources in the multi-node network predicts that there are idle computing resources on at least one second node having a magnitude that meets or exceeds estimated requirements to perform the processing job; and
in response to determining that an actual state of the computing resources predicted to be idle on the at least one second node meets or exceeds a predicted state of the computing resources on the at least one second node, executing the processing job on the at least one second node.
16. The system of claim 15 , wherein the multi-node network comprises a storage area network (SAN) and nodes of the multi-node network comprise a plurality of host computers and at least one storage appliance, and the predictive model includes predictions for resources of the at least one storage appliance.
17. The system of claim 15 , wherein the operations further comprise updating the predictive model of idle times of computing resources on a plurality of nodes of the multi-node network by updating an estimate of one or more resources required to perform the processing job with actual execution resources used to perform the processing job.
18. The system of claim 15 , wherein the operations further comprise:
in response to determining that an actual state of computing resources predicted to be idle on the at least one second node does not meet the computing resources predicted to be idle for executing the processing job on the at least one second node:
determining at least one third node on which to execute the processing job;
executing the processing job on the at the least one third node.
19. The system of claim 15 , the operations further comprising:
in response to determining, during the execution of the processing job on the at least one second node, that at least some of the computing resources predicted to be idle on the at least one second node are no longer idle or are no longer available to the processing job:
determining that the processing job is pausible;
pausing the processing job until the computing resources required to complete the processing job on the at least one second node are predicted to be idle for a predicted remaining execution time of the processing job;
resuming execution of the processing job on the at least one second node.
20. The system of claim 15 , the operations further comprising:
in response to determining, during the execution of the processing job on the at least one second node, that at least some of the computing resources predicted to be idle on the at least one second node are no longer idle or are no longer available to the processing job:
determining that the processing job is restartable;
determining a second predicted time at which the resources for executing the processing job are predicted to be idle on the at least one second node;
restarting execution of the processing job on the at least one second node.
21. The system of claim 15 , the operations further comprising:
in response to determining that at least some of the computing resources predicted to be idle on the at least one second node are not idle or are no longer available to the processing job:
migrating the processing job to a fourth node in the multi-node network that has the plurality of resources for executing the processing job predicted to be idle;
executing the processing job on the at least one fourth node.