Dynamic network of supercomputing resources with unified management interface
Systems, methods, and apparatuses are disclosed for implementation and management of a network of computing clusters and interfaces. In various embodiment, a dynamic supercomputing resource marketplace system can include a cluster network having one or more interconnected computing clusters. The dynamic supercomputing resource marketplace system also can include a user interface system or an application program interface system for enabling a user to access the computing clusters. Advantageously, the dynamic supercomputing resource marketplace system can be used to facilitate increased utilization of computing clusters.
1. A method for increasing computing cluster utilization in large scale computing environments, the method comprising:
aggregating a plurality of interconnected computing clusters into a cluster network, wherein different subsets of the plurality of interconnected computing clusters use different resource management software;
providing a user access to the aggregated computing clusters via a unified application programming interface (API) that provides (i) a single interface to the plurality of interconnected computing clusters and the different resource management software, and (ii) a user interface for providing one or more user interface services;
creating a plurality of portable containers for one or more applications and executing the plurality of portable containers via a plurality of selected computing clusters in the cluster network; and
wherein the aggregated computing clusters form a dynamic supercomputing resource marketplace system.
2. The method of claim 1 , wherein the plurality of interconnected computing clusters comprise multiple different public clouds and one or more private computing clusters connected via one or more virtual private networks (VPNs) that are automatically configured for the user.
3. The method of claim 2 , further comprising presenting the user with controls for selecting (i) an application to be executed and (ii) a set of nodes to be used to execute the application, wherein the set of nodes are selected from the aggregated computing clusters.
4. The method of claim 3 , wherein creating the plurality of containers includes one or more containers for the application to be executed and executing a container meta-scheduler configured to schedule containerized jobs across the selected set of nodes, wherein the container meta-scheduler communicates with the different resource management software for the selected set of nodes.
5. The method of claim 4 , further comprising calculating and presenting to the user an estimated cost and an estimated execution time for the selected application on the selected set of nodes.
6. The method of claim 4 , further comprising calculating an estimated cost to execute an application selected by the user.
7. The method of claim 1 , wherein the plurality of portable containers include master and worker containers configured for a selected application.
8. The method of claim 1 , wherein the plurality of portable containers are executed via each of the computing clusters in the cluster network.
9. The method of claim 1 , further comprising scheduling jobs for a selected application on nodes of the aggregated computing clusters.
10. The method of claim 9 , wherein said scheduling jobs for the selected application includes supporting submission of jobs to several of the plurality of selected computing clusters.
11. The method of claim 10 , wherein said supporting submission of jobs includes configuring each of the plurality of selected computing clusters to include a resource management system.
12. The method of claim 11 , wherein said configuring each of the plurality of selected computing clusters comprises configuring each of the plurality of selected computing clusters to include a different resource management system.
13. The method of claim 1 , further comprising enabling at least one of the computing clusters to support a pre-packaged container.
14. The method of claim 1 , further comprising enabling at least one of the computing clusters to support a proprietary container that is associated with one or more users.
15. The method of claim 14 , wherein the proprietary container complies with preselected containerization guidelines for ensuring portability across different computing clusters in the cluster network.
16. A computer program product for increasing computing cluster utilization in large scale computing environments, the computer program product comprising:
one or more non-transitory machine-readable storage media;
instruction for aggregating a plurality of interconnected computing clusters into a cluster network;
instruction for accessing the aggregated computing clusters, said instruction for accessing the aggregated computing clusters including at least one of (i) instruction for accessing the aggregated computing clusters via a unified application programming interface (API) for providing one or more API interface services and (ii) instruction for accessing the aggregated computing clusters via a user interface for providing one or more user interface services;
instruction for creating a plurality of portable containers for one or more applications and executing the plurality of portable containers via a plurality of selected computing clusters in the cluster network;
wherein the aggregated computing clusters form a dynamic supercomputing resource marketplace system; and
wherein at least one of (i) the one or more API interface services and (ii) the one or more user interface services are hosted publicly.
17. A method for increasing computing cluster utilization in large scale computing environments, the method comprising:
forming a dynamic marketplace with supercomputing resources acquirable by a user to execute a computing job via aggregating a plurality of interconnected computing clusters into a cluster network;
providing the user access to the aggregated computing clusters via a unified application programming interface (API) that provides (i) a single interface to the plurality of interconnected computing clusters and the different resource management software, and (ii) a user interface for providing one or more user interface services;
wherein different subsets of the plurality of interconnected computing clusters use different resource management software;
presenting the user with controls for selecting (i) an application to be executed and (ii) a set of nodes to be used to execute the application, wherein the set of nodes are selected from the aggregated computing clusters; and
calculating and presenting to the user an estimated cost and an estimated execution time to execute the selected application on the selected set of nodes.
18. The method of claim 17 , further comprising creating a portable container and executing the portable container via a selected computing cluster in the cluster network.
19. The method of claim 17 , further comprising creating a plurality of containers for the application to be executed and executing a container meta-scheduler configured to schedule containerized jobs across the selected set of nodes, wherein the container meta-scheduler communicates with the different resource management software for the selected set of nodes.
20. A method for increasing computing cluster utilization in large scale computing environments, the method comprising:
aggregating a plurality of interconnected computing clusters into a cluster network, wherein different subsets of the plurality of interconnected computing clusters use different resource management software;
providing a user access to the aggregated computing clusters via a unified application programming interface (API) that provides (i) a single interface to the plurality of interconnected computing clusters and the different resource management software, and (ii) a user interface for providing one or more user interface services;
wherein the aggregated computing clusters form a dynamic supercomputing resource marketplace system;
wherein the plurality of interconnected computing clusters comprise multiple different public clouds and one or more private computing clusters connected via one or more virtual private networks (VPNs) that are automatically configured for the user; and
further including presenting the user with controls for selecting (i) an application to be executed and (ii) a set of nodes to be used to execute the application, wherein the set of nodes are selected from the aggregated computing clusters.