Routing job submissions between disparate compute environments
A system and method are provided for directing a workload between distributed computing environments. Performance and use data from each of a plurality of computer clusters is monitored on a periodic or continuous basis. The plurality of computers can include a first subset being in a first region and a second subset being in a second region. Each region has known performance characteristics, zone of performance and zone of reliability which is used in distributing a workload or job. A job is received at the system, wherein the system determines a routing for the job to a distributed computing environment, wherein the routing is in response to the obtained performance and use data, and the region encompassing the given computer cluster.
1. A method, implemented at a computer system that includes one or more processors, for directing a workload between distributed computing environments, the method comprising:
identifying a plurality of computer clusters, the plurality of computer clusters including at least one local computer cluster and at least one remote computer cluster;
configuring each of the plurality of computer clusters to comprise a distinct region of computational power having known performance characteristics, including a zone of performance, and a zone of reliability;
continuously obtaining performance and use data from each of the plurality of computer clusters;
receiving a job configured to be routed to a single, particular computer cluster of the plurality of computer clusters for processing;
identifying metadata associated with processing the received job;
analyzing the metadata of the job; and
automatically routing at least a portion of the job to at least one computer cluster of the plurality of computer clusters that is different than the single, particular computer cluster, the routing automatically being performed in response to (i) the obtained performance and use data of the plurality of computer clusters and (ii) the analyzed metadata of the job.
2. The method of claim 1 , wherein the continuously obtaining performance and use data includes obtaining performance and use data on a periodic basis.
3. The method of claim 1 , wherein the routing mimics the behavior of a scheduler-based job submission.
4. The method of claim 1 , wherein the routing corresponds to operation of a job handler.
5. The method of claim 1 , wherein job routing is coupled to scheduler monitoring.
6. The method of claim 1 , wherein routing includes executing the job on multiple heterogeneous environments and transparently locating results in a consolidated, independent of performance related input from the user.
7. The method of claim 1 , wherein routing includes multi-pass routing of the job to a given subset of the plurality of clusters.
8. A computer system, comprising:
one or more processors; and
one or more hardware storage devices having stored thereon computer-executable instructions that are executable by the one or more processors to cause the computer system to direct a workload between distributed computing environments, the computer-executable instructions including instructions that are executable to cause the computer system to perform at least the following:
identify a plurality of computer clusters, the plurality of computer clusters including at least one local computer cluster and at least one remote computer cluster;
configure each of the plurality of computer clusters to comprise a distinct region of computational power having known performance characteristics, including a zone of performance, and a zone of reliability;
continuously obtain performance and use data from each of the plurality of computer clusters;
receive a job configured to be routed to a single, particular computer cluster of the plurality of computer clusters for processing;
identify metadata associated with processing the received job;
analyze the metadata of the job; and
automatically routing at least a portion of the job to at least one computer cluster of the plurality of computer clusters that is different than the single, particular computer cluster, the routing automatically being performed in response to (i) the obtained performance and use data of the plurality of computer clusters and (ii) the analyzed metadata of the job.
9. The computer system of claim 8 , wherein the continuously obtaining performance and use data includes obtaining performance and use data on a periodic basis.
10. The computer system of claim 8 , wherein the routing mimics the behavior of a scheduler-based job submission.
11. The computer system of claim 8 , wherein the routing corresponds to operation of a job handler.
12. The computer system of claim 8 , wherein job routing is coupled to scheduler monitoring.
13. The computer system of claim 8 , wherein routing includes executing the job on multiple heterogeneous environments and transparently locating results in a consolidated, independent of performance related input from the user.
14. The computer system of claim 8 , wherein routing includes multi-pass routing of the job to a given subset of the plurality of clusters.
15. A computer program product comprising one or more hardware storage devices having stored thereon computer-executable instructions that are executable by one or more processors of a computer system to cause the computer system to direct a workload between distributed computing environments, the computer-executable instructions including instructions that are executable to cause the computer system to perform at least the following:
identify a plurality of computer clusters, the plurality of computer clusters including at least one local computer cluster and at least one remote computer cluster;
configure each of the plurality of computer clusters to comprise a distinct region of computational power having known performance characteristics, including a zone of performance, and a zone of reliability;
continuously obtain performance and use data from each of the plurality of computer clusters;
receive a job configured to be routed to a single, particular computer cluster of the plurality of computer clusters for processing;
identify metadata associated with processing the received job;
analyze the metadata of the job; and
automatically routing at least a portion of the job to at least one computer cluster of the plurality of computer clusters that is different than the single, particular computer cluster, the routing automatically being performed in response to (i) the obtained performance and use data of the plurality of computer clusters and (ii) the analyzed metadata of the job.
16. The computer program product of claim 15 , wherein the continuously obtaining performance and use data includes obtaining performance and use data on a periodic basis.
17. The computer program product of claim 15 , wherein the routing mimics the behavior of a scheduler-based job submission.
18. The computer program product of claim 15 , wherein the routing corresponds to operation of a job handler.
19. The computer program product of claim 15 , wherein routing includes executing the job on multiple heterogeneous environments and transparently locating results in a consolidated, independent of performance related input from the user.
20. The computer program product of claim 15 , wherein routing includes multi-pass routing of the job to a given subset of the plurality of clusters.