Cluster job submission
A method and apparatus for job submission are described. In one embodiment, the jobs are submitted by a job submission service or gateway that schedules large-scale data processing jobs on remote infrastructure. In one embodiment, the method comprises: receiving a request at a proxy service from a client, via a network communication, to submit a job to a cluster; and managing the job externally to the client, including sending a request to an orchestration system to launch an orchestration system job in a container to start the job running on the cluster via a client process run on a job client in the container and provide state information back to the proxy service regarding the orchestration system job.
1. A method comprising:
receiving, at a proxy service, a request to submit a job to a cluster
sending, by the proxy service, the request to an orchestration system;
causing the orchestration system to launch an orchestration system job in a container on a job client server, wherein the job client server is separate from the cluster and the proxy service, the orchestration system job to start the job on the cluster, the orchestration system job comprising:
a job client; and
a launcher wrapper to launch the job client on the job client server;
storing, in a database under control of the proxy service, a job identifier associated with the job and a cluster identifier associated with the cluster;
receiving state information regarding the orchestration system job sent by the launcher wrapper to the proxy service for storage in the database with the job identifier associated with the job and the cluster identifier associated with the cluster;
receiving, by the proxy service, a kill request to terminate the job; and
terminating, by the proxy service, the job using the job identifier stored in the database without proceeding through the orchestration system.
2. The method defined in claim 1 , wherein the request includes a set of modification hooks, wherein the set of modification hooks are provided via a submit job endpoint for modifying, by the proxy service, the request based on the modification hooks, the modifying comprising at least one of:
a set of arguments associated with the job; or
an execution environment associated with the job.
3. The method defined in claim 2 wherein one of the set of modification hooks is an access control hook, wherein the access control hook:
obtains a name of a user placing the request, a set of inputs, and a set of outputs associated with the job; and
calls a library to confirm that the user has access to a set of data the job will access.
4. The method defined in claim 2 wherein the set of modification hooks comprises a first modification hook and a second modification hook and the second modification hook is applied to an output of the first modification hook.
5. The method of claim 1 , further comprising:
receiving a call back from the launcher wrapper with information indicating the orchestration system job is running;
receiving a heartbeat message from the launcher wrapper indicating the orchestration system job is running; and
receiving one or more logs emitted by the job client.
6. The method defined in claim 1 wherein the job is an HTTP request.
7. The method defined in claim 1 wherein the cluster is a Hadoop cluster.
8. A system comprising:
a network interface to receive a request for a proxy service;
a memory to store instructions;
one or more processors coupled to the memory and the network interface to execute the stored instructions to:
receive, at the proxy service, a request to submit a job to a cluster, the request including a set of modification hooks;
modify, by the proxy service, the request based on the modification hooks, the modifying comprising at least one of:
a set of arguments associated with the job; or
an execution environment associated with the job;
send, by the proxy service, the request to an orchestration system;
cause the orchestration system to launch an orchestration system job in a container on a job client server, wherein the job client server is separate from the cluster and the proxy service, the orchestration system job to start the job on the cluster, the orchestration system job comprising:
a job client; and
a launcher wrapper to launch the job client on the job client server;
store, in a database under control of the proxy service, a job identifier associated with the job and a cluster identifier associated with the cluster;
receive state information regarding the orchestration system job sent by the launcher wrapper to the proxy service for storage in the database with the job identifier associated with the job and the cluster identifier associated with the cluster;
receive, by the proxy service, a kill request to terminate the job; and
terminate, by the proxy service, the job using the job identifier stored in the database without proceeding through the orchestration system.
9. The system defined in claim 8 wherein the one or more processors are configured to:
store, under control of the proxy service, job identification information to identify the job and at least one of status or log information regarding the job produced by the job client in the container; and
enable access to at least one of status or log information regarding the job produced by the job client in the container.
10. The system defined in claim 8 wherein the network interface is operable to:
receive a call back from the launcher wrapper with information indicating the orchestration system job is running;
receive a heartbeat messages from the launcher wrapper indicating the orchestration system job is running; and
receive one or more logs emitted by the job client.
11. The system defined in claim 8 wherein the one or more processors send the request to the orchestration system to launch the job by launching a container that encapsulates the job.
12. The system defined in claim 11 wherein the encapsulated job includes hooks for one or more operations for the cluster to perform.
13. The system defined in claim 8 wherein the request is an HTTP request.
14. The system defined in claim 8 wherein the cluster is a Hadoop cluster.
15. A non-transitory computer readable storage media comprising instructions stored thereupon which, when executed by a system having at least a processor and a memory therein, cause the system to perform operations comprising:
receiving, at a proxy service, a request to submit a job to a cluster, the request including a set of modification hooks;
modifying, by the proxy service, the request based on the modification hooks, the modifying comprising at least one of:
a set of arguments associated with the job; or
an execution environment associated with the job;
sending, by the proxy service, the request to an orchestration system;
causing the orchestration system to launch an orchestration system job in a container on a job client server, wherein the job client server is separate from the cluster and the proxy service, the orchestration system job to start the job on the cluster, the orchestration system job comprising:
a job client; and
a launcher wrapper to launch the job client on the job client server;
storing, in a database under control of the proxy service, a job identifier associated with the job and a cluster identifier associated with the cluster;
receiving state information regarding the orchestration system job sent by the launcher wrapper to the proxy service for storage in the database with the job identifier associated with the job and the cluster identifier associated with the cluster;
receiving, by the proxy service, a kill request to terminate the job; and
terminating, by the proxy service, the job using the job identifier stored in the database without proceeding through the orchestration system.
16. The non-transitory computer readable storage media defined in claim 15 wherein the operations further comprise:
storing, under control of the proxy service, job identification information to identify the job and at least one of status and log information regarding the job produced by the job client in the container; and
enabling access to at least one of status or log information regarding the job produced by the job client in the container.
17. The non-transitory computer readable storage media defined in claim 15 wherein the operations further comprise:
receiving a call back from the launcher wrapper with information indicating the orchestration system job is running;
receiving a heartbeat messages from the launcher wrapper indicating the orchestration system job is running; and
receiving one or more logs emitted by the job client.
18. The non-transitory computer readable storage media defined in claim 15 wherein sending the request to the orchestration system to launch the orchestration system job comprises launching a container that encapsulates the orchestration system job.
19. The non-transitory computer readable storage media defined in claim 15 wherein the set of modification hooks comprises a first modification hook and a second modification hook and the second modification hook is applied to an output of the first modification hook.
20. The non-transitory computer readable storage media defined in claim 15 wherein one of the set of modification hooks is an access control hook, wherein the access control hook:
obtains a name of a user placing the request, a set of inputs, and a set of outputs associated with a job; and
calls a library to confirm that the user has access to a set of data the job will access.