Data processing in a hybrid cluster environment
A hybrid cluster environment with a public cloud cluster having nodes storing data and a plurality of private clusters is provided, wherein each of the plurality of private clusters has nodes storing data. Registration data that indicates a customer identifier, a new private cluster, and a file transfer server is received. The new private cluster is added to the plurality of private clusters in the hybrid cluster environment. Input to design a job to process data in the hybrid cluster environment is received. It is determined that the job is to be deployed to the new private cluster. The job is deployed to the new private cluster using the file transfer server, wherein the job is executed at the new private cluster. Job status information and one or more job logs are received with the file transfer server.
1. A computer-implemented method, comprising operations for:
providing a hybrid cluster environment with a public cloud cluster having nodes storing data and a plurality of private clusters, wherein each of the plurality of private clusters has nodes storing data;
receiving registration data that indicates a customer identifier, a new private cluster, and a file transfer server;
adding the new private cluster to the plurality of private clusters in the hybrid cluster environment;
receiving input to design a job to process data in the hybrid cluster environment;
determining that the job is to be deployed to the new private cluster;
deploying the job to the new private cluster using the file transfer server, wherein the job is executed at the new private cluster; and
receiving job status information and one or more job logs with the file transfer server.
2. The computer-implemented method of claim 1 , wherein the job is deployed at any combination of: the public cloud cluster and any cluster from the plurality of private clusters.
3. The computer-implemented method of claim 1 , wherein the registration data identifies a master node of the new private cluster.
4. The computer-implemented method of claim 3 , further comprising operations for:
selecting a node of the nodes of the new private cluster using a load balancing proxy server of the master node of the new private cluster;
executing the job to process the data at the selected node of the new private cluster while monitoring the execution to generate the job status information and the one or more job logs without moving the data out of the selected node; and
returning the job status information and the one or more job logs using the file transfer server of the new private cluster.
5. The computer-implemented method of claim 1 , wherein a cluster management service of the hybrid cluster environment creates the job based on the input to design the job.
6. The computer-implemented method of claim 1 , wherein the new private cluster is one of an on-premise cluster at a customer location and a private cloud cluster.
7. The computer-implemented method of claim 1 , wherein a Software as a Service (SaaS) is configured to perform the operations of the method.