Decoupling cluster data from cloud deployment
The present invention extends to methods, systems, and computer program products for decoupling cluster data from cloud deployment. The decoupling facilitates taking down and reattaching a cluster to persisted data as needed. Other embodiments of the invention include resuming a cluster's execution on a different topology and/or different system states based on a persisted data set.
1. One or more computer hardware storage device for use in a computing environment in which a computer system with one or more processors interoperates with one or more other computer systems in a distributed system, the one or more computer hardware storage device having stored computer-executable instructions which, when executed by the one or more processors, implement a method for resuming execution of a cluster from saved cluster data, wherein the method includes:
an act of deploying a cluster of compute resources within the distributed system, the deployed cluster of compute resources defining a current topology for executing a job, and the current topology indicating a current number of nodes in one or more node groups, a current number of cores at each node, a current memory size at each node, and a current naming patter at each node;
an act of executing a portion of the job on the deployed cluster, execution of the portion of the job utilizing cluster data corresponding to the job;
an act of persisting cluster data that represents the current cluster topology of the deployed cluster of compute resources in an external database that is separate from the deployed cluster of compute resources;
an act of taking down the deployed cluster of compute resources after persisting the cluster data for the current cluster topology but prior to completing execution of the job, taking down the deployed cluster leaving a remaining portion of the job uncompleted;
an act of receiving a command to resume execution of the deployed cluster of compute resources in order to complete the job;
in response to the command to resume execution, an act of identifying the persisted cluster data that represents the cluster topology in the external database, and then re-deploying the cluster of compute resources within the distributed system based on the cluster topology that was identified in the persisted cluster data; and
an act of resuming execution of the job on the re-deployed cluster of compute resources to complete the remaining portion of the job, completion of the remaining portion of the job utilizing the persisted cluster data.
2. The one or more computer hardware storage device recited in claim 1 , wherein the act of deploying a cluster of compute resources within the distributed system comprises an act of deploying a cluster of computer recourses on a cloud-based platform.
3. The one or more computer hardware storage device recited in claim 1 , wherein the current topology includes at least two different types of nodes, the at least two different types of nodes including nodes that rely on persisted cluster data and nodes that are driven by incoming requests.
4. The one or more computer hardware storage device recited in claim 1 , wherein the act of persisting the cluster data in the external database comprises an act of persisting one or more of: node information, resource information, job information, task information, cluster-wide settings, action history, monitoring information, cluster users, cluster passwords, and cluster permissions.
5. The one or more computer hardware storage device recited in claim 1 , the method further comprising:
an act of receiving account information prior to deploying the cluster of compute resources; and
wherein the act of re-deploying the cluster of compute resources comprises re-deploying based on permissions corresponding to the account information.
6. The one or more computer hardware storage device recited in claim 1 , wherein the act of re-deploying the cluster of compute resources within the distributed system comprises an act of re-deploying the cluster of compute resources by identifying a second, different cluster topology which provides equivalent compute resources to those of the cluster of compute resources for which the cluster data was persisted.
7. The one or more computer hardware storage device recited in claim 6 , wherein the act of re-deploying the cluster of compute resources on the second, different cluster topology comprises aligning the cluster topology as identified by the persisted cluster data with the second, different cluster topology.
8. The one or more computer hardware storage device recited in claim 7 , wherein the act of aligning the cluster topology as identified by the persisted cluster data with the second, different cluster topology comprises an act of adjusting one or more of: a number of cores at each node in the cluster topology to comply with the second different cluster topology, adjusting the memory size of at least at each node in the cluster topology as identified by the persisted cluster data to comply with the second, different cluster topology, and adjusting the naming pattern at each node in the cluster topology as identified by the persisted cluster data to comply with the second, different cluster topology.
9. The one or more computer hardware storage device recited in claim 7 , the method further comprising:
prior to aligning the cluster topology as identified by the persisted cluster data with the second, different cluster topology mapping one or more cloud-based roles to the second, different topology.
10. The one or more computer hardware storage device of claim 1 , wherein the one or more computer hardware storage device comprises the computer system with the one or more processors.
11. One or more computer hardware storage device for use in a computing environment in which a computer system with one or more processors interoperates with one or more other computer systems in a distributed system, the one or more computer hardware storage device having stored computer-executable instructions which, when executed by the one or more processors, implement a method for resuming execution of a cluster on a different topology, wherein the method includes:
an act of receiving a command to resume execution of a cluster based on a persisted data set, the persisted data set including a persisted cluster topology used at the time the persisted data set was persisted, the persisted cluster topology persisting a number of nodes in one or more node groups, the number of cores at each node, the memory size at each node, and the naming pattern at each node;
an act of mapping distributed system roles to a current cluster topology, the current cluster topology indicating a current number of nodes in the one or more node groups, a current number of cores at each node, a current memory size at each node, and a current naming pattern at each node;
an act of accessing the persisted data set from an external database, the external database separate from the deployed cluster;
an act of aligning the persisted cluster topology with the current cluster topology, including one or more of: (1) adjusting the number of nodes in the one or more node groups in the persisted cluster topology to fit the current cluster topology, (2) adjusting the number of cores at each node in the persisted cluster topology to comply with the current cluster topology, (3) adjusting the memory size at least at each node in the persisted cluster topology to comply with the current cluster topology, and (4) adjusting the naming pattern at each node in the persisted topology to comply with the current cluster topology;
an act of persisting the results of aligning the persisted cluster topology with the current cluster topology in the external database; and
an act of initiating the cluster to accept requests.
12. The one or more computer hardware storage device recited in claim 11 , the method further comprising:
an act of preventing the reception of new job requests in response to receiving the command to resume execution of a cluster based on a persisted data set and prior to aligning the persisted cluster topology with the current cluster topology.
13. The one or more computer hardware storage device recited in claim 11 , the method furthering comprising:
an act of storing the persisted data set in the external database prior to receiving the resume command.
14. The one or more computer hardware storage device recited in claim 13 , wherein the act of storing the persisted data set in the external database comprises storing one or more of: node information, resource information, job information, task information, cluster-wide settings, action history, monitoring information, cluster users, cluster passwords, and cluster permissions, for the persisted current topology.
15. The one or more computer hardware storage device recited in claim 14 , wherein the act of aligning the persisted cluster topology with the current cluster topology comprises:
an act of downloading cluster users, cluster passwords, and cluster permissions from the external data base; and
an act of creating the cluster users, cluster passwords, and cluster permissions on the current cluster topology.
16. The one or more computer hardware storage device recited in claim 11 , the method further comprising an act of stopping all active jobs on the currently cluster topology subsequent to aligning the persisted cluster topology with the current cluster topology.
17. The one or more computer hardware storage device of claim 11 , wherein the one or more computer hardware storage device comprises the computer system with the one or more processors.
18. One or more computer hardware storage device for use in a computing environment in which a computer system with one or more processors interoperates with one or more other computer systems in a cloud-based platform, the one or more computer hardware storage device having stored computer-executable instructions which, when executed by the one or more processors, implement a method for resuming execution of a cluster on a different topology, wherein the method includes:
an act of receiving a command to resume execution of a cluster based on a persisted data set, the persisted data set identifying a persisted cluster topology used at the time the persisted data set was persisted, the persisted cluster topology persisting a number of nodes in one or more node groups, the number of cores at each node, the memory size at each node, and the naming pattern at each node;
an act of preventing the reception of new job requests in response to receiving the command to resume execution of a cluster based on a persisted data set;
an act of accessing distributed system roles being utilized in a current cluster topology;
an act of mapping the distributed system roles to the current cluster topology components, including for each distributed system role:
an act of mapping a role name for the distributed system role to a node group;
an act of mapping a virtual memory size for the distributed system role to a number of cores per node and memory per node; and
an act of mapping an instance count for the distributed system role to a number of nodes;
an act of aligning the persisted cluster topology with the current cluster topology into an aligned topology, including one or more of: (1) adjusting the number of nodes in the one or more node groups in the persisted cluster topology to fit the current cluster topology, (2) adjusting the number of cores at each node in the persisted cluster topology to comply with the current cluster topology, (3) adjusting the memory size at least at each node in the persisted cluster topology to comply with the current cluster topology, and (4) adjusting the naming pattern at each node in the persisted topology to comply with the current cluster topology;
an act of stopping all active jobs on the current topology;
an act of persisting the aligned topology in the external database; and
an act of initiating the cluster to accept requests in accordance with the aligned topology.
19. The one or more computer hardware storage device recited in claim 18 , wherein the external database is a distributed database and wherein the act of accessing distributed system roles being utilized in a current cluster topology comprises an act of accessing the distributes system roles from a configuration file.
20. The one or more computer hardware storage device of claim 18 , wherein the one or more computer hardware storage device comprises the computer system with the one or more processors.