Data processing in a hybrid cluster environment
A hybrid cluster environment with a public cloud cluster having nodes storing data and a plurality of private clusters is provided, wherein each of the plurality of private clusters has nodes storing data. Registration data that indicates a customer identifier, a new private cluster, and a file transfer server is received. The new private cluster is added to the plurality of private clusters in the hybrid cluster environment. Input to design a job to process data in the hybrid cluster environment is received. It is determined that the job is to be deployed to the new private cluster. The job is deployed to the new private cluster using the file transfer server, wherein the job is executed at the new private cluster. Job status information and one or more job logs are received with the file transfer server.
1. A computer program product, the computer program product comprising a computer readable storage medium having program code embodied therewith, the program code executable by at least one processor to perform operations comprising:
providing a hybrid cluster environment with a public cloud cluster having nodes storing data and a plurality of private clusters, wherein each of the plurality of private clusters has nodes storing data;
receiving registration data that indicates a customer identifier, a new private cluster, and a file transfer server;
adding the new private cluster to the plurality of private clusters in the hybrid cluster environment;
receiving input to design a job to process data in the hybrid cluster environment;
determining that the job is to be deployed to the new private cluster;
deploying the job to the new private cluster using the file transfer server, wherein the job is executed at the new private cluster; and
receiving job status information and one or more job logs with the file transfer server.
2. The computer program product of claim 1 , wherein the job is deployed at any combination of: the public cloud cluster and any cluster from the plurality of private clusters.
3. The computer program product of claim 1 , wherein the registration data identifies a master node of the new private cluster.
4. The computer program product of claim 3 , wherein the program code is executable by the at least one processor to perform further operations comprising:
selecting a node of the nodes of the new private cluster using a load balancing proxy server of the master node of the new private cluster;
executing the job to process the data at the selected node of the new private cluster while monitoring the execution to generate the job status information and the one or more job logs without moving the data out of the selected node; and
returning the job status information and the one or more job logs using the file transfer server of the new private cluster.
5. The computer program product of claim 1 , wherein a cluster management service of the hybrid cluster environment creates the job based on the input to design the job.
6. The computer program product of claim 1 , wherein the new private cluster is one of an on-premise cluster at a customer location and a private cloud cluster.
7. The computer program product of claim 1 , wherein a Software as a Service (SaaS) is configured to perform the operations of the computer program product.
8. A computer system, comprising:
one or more processors, one or more computer-readable memories and one or more computer-readable, tangible storage devices; and
program instructions, stored on at least one of the one or more computer-readable, tangible storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to perform operations comprising:
providing a hybrid cluster environment with a public cloud cluster having nodes storing data and a plurality of private clusters, wherein each of the plurality of private clusters has nodes storing data;
receiving registration data that indicates a customer identifier, a new private cluster, and a file transfer server;
adding the new private cluster to the plurality of private clusters in the hybrid cluster environment;
receiving input to design a job to process data in the hybrid cluster environment;
determining that the job is to be deployed to the new private cluster;
deploying the job to the new private cluster using the file transfer server, wherein the job is executed at the new private cluster; and
receiving job status information and one or more job logs with the file transfer server.
9. The computer system of claim 8 , wherein the job is deployed at any combination of: the public cloud cluster and any cluster from the plurality of private clusters.
10. The computer system of claim 8 , wherein the registration data identifies a master node of the new private cluster.
11. The computer system of claim 10 , wherein the operations further comprise:
selecting a node of the nodes of the new private cluster using a load balancing proxy server of the master node of the new private cluster;
executing the job to process the data at the selected node of the new private cluster while monitoring the execution to generate the job status information and the one or more job logs without moving the data out of the selected node; and
returning the job status information and the one or more job logs using the file transfer server of the new private cluster.
12. The computer system of claim 8 , wherein a cluster management service of the hybrid cluster environment creates the job based on the input to design the job.
13. The computer system of claim 8 , wherein a Software as a Service (SaaS) is configured to perform operations of the computer system.