Cluster job submission

1. A method comprising:

receiving a request at a proxy service from a first client, via a first network communication, to submit a first job to a cluster; and

managing the first job externally to the first client, including

sending, by the proxy service, a request to an orchestration system to launch an orchestration system job in a container to start a cluster job running on the cluster via a job client running on a job client server separate from the cluster and the proxy service, the container encapsulating the first job that the client requested the proxy service to submit and including the job client and a launcher wrapper that launches the job client on the job client server,

storing, in a database under control of the proxy service, job identification information to identify the first job and a mapping of the job identification information to a cluster identifier associated with the cluster job, and

receiving state information regarding the orchestration system job sent by the launcher wrapper to the proxy service for storage in the database with the job identification information and the mapping of the job identification information to the cluster identifier associated with the cluster;

receiving, by the proxy service, a kill request from the first client to stop execution of the first job; and

signaling, by the proxy service, the cluster job directly without proceeding through the orchestration system using the cluster identifier stored in memory to kill execution of the cluster job.

2. The method defined in claim 1 further comprising:

storing, under control of the proxy service, job identification information to identify the first job and one or both of status and log information regarding the first job produced by the job client in the container; and

enabling access, to the first client, to status and/or log information regarding the first job produced by the job client in the container.

3. The method defined in claim 1 further comprising:

receiving a call back from the container with information indicating the orchestration system job is running;

receiving a heartbeat messages from the container indicating the orchestration system job is running; and

receiving one or more logs emitted by the job client.

4. The method defined in claim 1 wherein sending the request to the orchestration system to launch the orchestration system job comprises launching a container that encapsulates the first job.

5. The method defined in claim 4 wherein the encapsulated job includes hooks for one or more operations for the cluster to perform.

6. The method defined in claim 1 wherein the first job is a batch job.

7. The method defined in claim 1 wherein the request is an HTTP request.

8. The method defined in claim 1 wherein the cluster is a Hadoop cluster, and wherein the orchestration system is Kubernetes.

9. The method of claim 1 further comprising:

deleting, by the proxy service, the orchestration system job in response to the kill request.

10. A system comprising:

a network interface to receive a request at a proxy service from a first client, via a first network communication, to submit a first job to a cluster;

a memory to store instructions;

one or more processors coupled to the memory and the network interface to execute the stored instructions to:

manage the first job externally to the first client, by

sending, by the proxy service, a request to an orchestration system to launch an orchestration system job in a container to start a cluster job running on the cluster via a job client running on a job client server separate from the cluster and the proxy service, the container encapsulating the first job that the client requested the proxy service to submit and including the job client and a launcher wrapper that launches the job client on the job client server,

storing, in a database under control of the proxy service, job identification information to identify job the first job and a mapping of the job identification information to a cluster identifier associated with the cluster job,

receiving state information regarding the orchestration system job sent by the launcher wrapper to the proxy service for storage in the database with the job identification information and the mapping of the job identification information to the cluster identifier associated with the cluster,

receiving, by the proxy service, a kill request from the first client to stop execution of the first job, and

signaling, by the proxy service, the cluster job directly without proceeding through the orchestration system using the cluster identifier stored in memory to kill execution of the cluster job.

11. The system defined in claim 10 wherein the one or more processors are configured to:

store, under control of the proxy service, job identification information to identify the first job and one or both of status and log information regarding the first job produced by the job client in the container; and

enable access, to the first client, to status and/or log information regarding the first job produced by the job client in the container.

12. The system defined in claim 10 wherein the network interface is operable to:

receive a call back from the container with information indicating the orchestration system job is running;

receive a heartbeat messages from the container indicating the orchestration system job is running; and

receive one or more logs emitted by the job client.

13. The system defined in claim 10 wherein the one or more processors send the request to the orchestration system to launch the orchestration system job by launching a container that encapsulates the first job.

14. The system defined in claim 13 wherein the encapsulated job includes hooks for one or more operations for the cluster to perform.

15. The system defined in claim 10 wherein the request is an HTTP request.

16. One or more non-transitory computer readable storage media having instructions stored thereupon which, when executed by a system having at least a processor and a memory therein, cause the system to perform operations comprising:

receiving a request at a proxy service from a first client, via a first network communication, to submit a first job to a cluster; and

managing the first job externally to the first client, including

sending, by the proxy service, a request to an orchestration system to launch an orchestration system job in a container to start a cluster job running on the cluster via a job client running on a job client server separate from the cluster and the proxy service, the container encapsulating the first job that the client requested the proxy service to submit and including the job client and a launcher wrapper that launches the job client on the job client server,

storing, in a database under control of the proxy service, job identification information to identify job the first job and a mapping of the job identification information to a cluster identifier associated with the cluster job,

receiving state information regarding the orchestration system job sent by the launcher wrapper to the proxy service for storage in the database with the job identification information and the mapping of the job identification information to the cluster identifier associated with the cluster,

receiving, by the proxy service, a kill request from the first client to stop execution of the first job, and

signaling, by the proxy service, the cluster job directly without proceeding through the orchestration system using the cluster identifier stored in memory to kill execution of the cluster job.

17. The computer readable storage media defined in claim 16 , wherein the method further comprises:

storing, under control of the proxy service, job identification information to identify the first job and one or both of status and log information regarding the first job produced by the job client in the container; and

enabling access, to the first client, to status and/or log information regarding the first job produced by the job client in the container.

18. The computer readable storage media defined in claim 16 wherein the method further comprises:

receiving a call back from the container with information indicating the orchestration system job is running;

receiving a heartbeat messages from the container indicating the orchestration system job is running; and

receiving one or more logs emitted by the job client.

19. The computer readable storage media defined in claim 16 wherein sending the request to the orchestration system to launch the orchestration system job comprises launching a container that encapsulates the first job.

20. The computer readable storage media defined in claim 19 wherein the encapsulated job includes hooks for one or more operations for the cluster to perform.

21. The computer readable storage media defined in claim 16 wherein the first job is a batch job, the request is an HTTP request, the cluster is a Hadoop cluster, and the orchestration system is Kubernetes.